Consul: add preflight checks for Envoy bootstrap

[tgross]

Nomad creates Consul ACL tokens and service registrations to support Consul service mesh workloads, before bootstrapping the Envoy proxy. Nomad always talks to the local Consul agent and never directly to the Consul servers. But the local Consul agent talks to the Consul servers in stale consistency mode to reduce load on the servers. This can result in the Nomad client making the Envoy bootstrap request with a tokens or services that have not yet replicated to the follower that the local client is connected to. This request gets a 404 on the ACL token and that negative entry gets cached, preventing any retries from succeeding.

To workaround this, we'll use a method described by our friends over on consul-k8s where after creating the objects in Consul we try to read them from the local agent in stale consistency mode (which prevents a failed read from being cached). This cannot completely eliminate this source of error because it's possible that Consul cluster replication is unhealthy at the time we need it, but this should make Envoy bootstrap significantly more robust.

This changset adds preflight checks for the objects we create in Consul:

• We add a preflight check for ACL tokens after we login via via Workload Identity and in the function we use to derive tokens in the legacy workflow. We do this check early because we also want to use this token for registering group services in the allocrunner hooks.
• We add a preflight check for services right before we bootstrap Envoy in the taskrunner hook, so that we have time for our service client to batch updates to the local Consul agent in addition to the local agent sync.

We've added the timeouts to be configurable via node metadata rather than the usual static configuration because for most cases, users should not need to touch or even know these values are configurable; the configuration is mostly available for testing.

Fixes: #9307
Fixes: #10451
Fixes: #20516

Ref: hashicorp/consul-k8s#887
Ref: https://hashicorp.atlassian.net/browse/NET-10051
Ref: https://hashicorp.atlassian.net/browse/NET-9273
Follow-up: https://hashicorp.atlassian.net/browse/NET-10138

---

Notes for reviewers:

• This is an unfortunately large PR as I had to touch a lot of test code and there are separate code paths for supporting the legacy workflow. To help make this a little more reviewable, I've split this into 2 commits with the code changes and their specific tests, a third commit with a test helper fix, and a final commit with docs and changelog.
• I've got a follow-up described in NET-10138 to reduce the risk of this kind of bug in the future by expanding our Consul cluster in E2E to be a 3-node cluster.
• The backport to 1.6.x+ent is not going to be clean, as all this code was heavily reworked for WI and multi-cluster support in 1.7.0. So I'm going to manually backport the relevant sections and open a new PR for that in the ENT repo.

[schmichael]

I'm very excited to see envoy bootstrapping get more reliable! Thanks for following this through.

[schmichael]

Hm, I'm not super familiar with this code, but it appears NewConsulClientFactory will be called from NewClient and therefore the timeout and interval node meta will be captured only during agent startup. Don't the durationFromMeta calls need to happen inside the closure and use client.Node() instead of a captured Node?

[tgross]

That's a good catch. For partitions, we don't expect that to change because it's part of the fingerprint we get from the agent. But metadata can be changed via the node meta command so yeah we'll need to move this into the closure.

[tgross]

Fixed! (And rebased on main) Also:

I'm not super familiar with this code
The various dependency injection tricks we're doing here to avoid circular references doesn't help. This code all got really janky during the big WI + Consul/Vault refresh. Once we remove the legacy workflow we should be able to refactor this in the process to be a little more comprehensible.

[shoenig]

LGTM!