ISSUE-779 - check for agent status before deregistration

[TomasKohout]

Changes proposed in this PR:
Potential fix for ISSUE-779

How I've tested this PR:
Deployed to our DEV cluster and ungracefully killed a node.

system-consul/consul-consul-connect-injector-webhook-deployment-755d55bdzb245[sidecar-injector]: 2021-10-19T12:29:11.955Z	INFO	controller.endpoints	retrieved	{"name": "csi-rbdplugin-metrics", "ns": "system-storage"}
system-consul/consul-consul-connect-injector-webhook-deployment-755d55bdzb245[sidecar-injector]: 2021-10-19T12:29:11.959Z	INFO	controller.endpoints	**Consul agent is not ready, skipping sync**	{"consul-agent": "consul-consul-8kw6s"}

How I expect reviewers to test this PR:

On cluster with consul agent and connected pods, ungracefully restart one of a node and see if new consul connected pods are able to start.

Checklist:

• Tests added
• CHANGELOG entry added

  HashiCorp engineers only, community PRs should not add a changelog entry.
  Entries should use present tense (e.g. Add support for...)

[hashicorp-cla]

All committers have signed the CLA.

[david-yu]

Hi @TomasKohout thank you for the PR! We saw the issue and it looks like we may be able to repro based on the description you provided in the PR.

[ishustava]

This looks great! thanks for working on a fix for it!

I think it'd be good to add a test for this, perhaps a case in the TestReconcileDeleteEndpoint test that checks that if the consul client pod is not ready, the service is not deregistered. Otherwise, this is good to merge.

[ishustava]

This change looks like it's not intentional. I think we want to keep this and the the corresponding test case param above.

[ishustava]

Thanks so much for the contribution @TomasKohout !!

I'm going to close this PR as I've included your change in #991

[TomasKohout]

Thank you so much! 🙂 I'm sorry that I couldn't get this done in a reasonable amount of time though. Shame on me. 😬😁