connect: register the service before the proxy

[ishustava]

Because the proxy service registers an alias health check that points to the service ID of the main service and we're registering the proxy service before the main service, the alias check starts out as red
because at that time the service doesn't yet exist. Consul runs this check every minute, and so this will become green only after one minute.

This is particularly bad in a case when a service is restarted either due to scheduled or unscheduled maintenance. For example, when you have deployment and you trigger a re-deploy (kubectl rollout restart), kubernetes by default will do a rolling deploy, where it won't terminate the old instance until the new one comes up and is healthy. But Consul will take an additional minute or so to mark this service as healthy, causing downtime, where either no downtime or minimal downtime should be experienced.

Changes proposed in this PR:

Switch the order of how services are registered with Consul, with the main service registered first and the proxy service after.

Steps to reproduce and test

1. Deploy the latest helm chart with connect enabled

    helm install consul --set global.name=consul --set connectInject.enabled=true --set server.replicas=1 --set server.bootstrapExpect=1 hashicorp/consul

2. Create static-server and static-client deployments.

    kubectl apply -f https://raw.githubusercontent.com/hashicorp/consul-helm/acceptance-tests-base/test/acceptance/tests/connect/fixtures/static-server.yaml
    kubectl apply -f https://raw.githubusercontent.com/hashicorp/consul-helm/acceptance-tests-base/test/acceptance/tests/connect/fixtures/static-client.yaml

3. Exec into the static client pod and run the following loop (it should print "hello world" every second):

    $ kubectl exec -it static-client-758b47746d-r5rll /bin/sh
    kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl kubectl exec [POD] -- [COMMAND] instead.
    Defaulting container name to static-client.
    Use 'kubectl describe pod/static-client-758b47746d-r5rll -n default' to see all of the containers in this pod.
    # while true; do echo "$(date) $(curl -sS http://localhost:1234)"; sleep 1; done
   

4. Restart the static-server deployment and observe about 1 minute of downtime in the logs from the while loop above.

    kubectl rollout restart deploy/static-server
   

To fix, upgrade to the image built from this PR and run helm upgrade:

helm upgrade consul --set global.name=consul --set global.imageK8S=hashicorpdev/consul-k8s:f817303 --set connectInject.enabled=true --set server.replicas=1 --set server.bootstrapExpect=1 hashicorp/consul

Once the connect injector becomes healthy, restart the static-server deployment again (step 4 above). You should either see no or 1-2 errors (i.e. 1-2 seconds of downtime) printed from the while loop running in the static-client container.

Checklist:

• Tests added
• CHANGELOG entry added (HashiCorp engineers only, community PRs should not add a changelog entry)

[thisisnotashwin]

This looks great. Was able to successfully reproduce the error and the fix. 🎉

[adilyse]

The code changes look fine, but the changelog needs to include a warning about the implications of this change.

[adilyse]

Per our conversation earlier, this should include the caveat that while this fix reverts to the previous behavior, that previous behavior includes Consul routing to services that may not be ready yet.