Implement cleanup controller

[lkysow]

• Runs by default
• Watches for pod delete events and ensures there are no stale Consul
  service instances
• Runs a Reconcile loop on a timer that checks all Consul services
• Adds new connect-inject flags: -enable-cleanup-controller,
  -cleanup-controller-reconcile-period.
• Adds new k8s-namespace meta key to connect service instances
• Adds new server-acl-init flag: -enable-cleanup-controller (default
  true) and modifies the ACL templates.

How I've tested this PR:

• manually and with acceptance tests: Cleanup controller consul-helm#806
• reconcile can be tested by registering a service directly with a consul client:
  • port-forward to consul client on 8500
  • use svc.json:

    {
      "ID": "redis1",
      "Name": "redis",
      "Tags": ["primary", "v1"],
      "Address": "127.0.0.1",
      "Port": 8000,
      "Meta": {
        "pod-name": "dead",
        "k8s-namespace": "default"
      }
    }
    

  • curl -X PUT -v --data @svc.json localhost:8500/v1/agent/service/register
  • wait for reconcile to run (can bump default time with the helm value)

  2021-02-10T00:50:32.329Z [DEBUG] cleanupResource: starting reconcile
  2021-02-10T00:50:32.342Z [INFO]  cleanupResource: found service instance from terminated pod still registered: pod=dead id=redis1
  2021-02-10T00:50:32.346Z [INFO]  cleanupResource: service instance deregistered: id=redis1
  2021-02-10T00:50:32.346Z [DEBUG] cleanupResource: finished reconcile
  

How I expect reviewers to test this PR:

• try it out with ghcr.io/lkysow/consul-k8s-dev:feb09 and see if you can break it

Checklist:

• Tests added
• CHANGELOG entry added (HashiCorp engineers only, community PRs should not add a changelog entry)

[lkysow]

This is a slight change to the Run function from the health check resource (https://github.com/hashicorp/consul-k8s/blob/master/connect-inject/health_check_resource.go#L54-L76).

1. I refactored the loop so we don't have h.Reconcile() called twice. I believe they're logically equivalent but please double check.
2. reconcile() here doesn't return an error. I did this because we're already logging all the places there are errors inside reconcile() with a lot of good context as to why the error is occurring and so logging it again here just results in duplicate error logs.
3. I've also passed in ConsulScheme and ConsulPort as separate fields into the resource struct up above rather than ConsulURL. I did this because we are only using the scheme and port from the URL so it felt weird to pass in the full URL when we're discarding the hostname.

I plan to contribute these changes back to HealthCheckResource in a separate PR.

[kschoche]

Great idea on point 3!

[lkysow]

Sorry in advance. I implemented these tests via a truth table that covers all permutations.

[thisisnotashwin]

I tried hard to look for something to pick on, but the best i could find was deregister to de-register 😫
Great work @lkysow

[lkysow]

I tried hard to look for something to pick on, but the best i could find was deregister to de-register 😫
Great work @lkysow

lol I went back and forth on that because my editor was complaining. https://www.dictionary.com/browse/deregister?s=t says it's allowed and it's one less character which is why I went with deregister but I'm easy.

[thisisnotashwin]

pff..good chance i would have suggested deregister to save a character had you chosen de-register 😉

I'm easy

❤️

[ishustava]

Looks really good!!! I really appreciate all comments and notes you left along the way - felt like you read my mind in a lot of places!

I've added some comments, but they are non-blocking.