Add a sync status endpoint / metrics to agents

[ShimmerGlass]

Background

The Consul agents sync to servers model is eventually consistent. It will not return errors when an action is performed but rather retry periodically :

• A service register call will always succeed even if the sync fails (we saw that with ACLs pre 0.8 : [Fix] Services sometimes not being synced with acl_enforce_version_8 = false #4771 and agent wipe : Allow service deregistration with node write permission #5217)
• A node id conflict can prevent a node from registering itself in the catalog

Issue

Whether the sync failing is due to a bug, a bad setup, or temporary failure it is hard do detect these desyncs at scale :

• When such issues occur the only traces are in the logs
• There is no way to have an overview of an agent sync state

We would like to detect these issues to :

• Monitor the sync status of agents and their services
• Fail early in the case of conflicts

Proposal

• Add an endpoint such as /v1/agent/status returning the state of the agent sync for its node, service , checks. Additionally other things such as the proposed async persistent state (Make agent local state async #6101) could find there place there too.
• Add metrics for these async elements to enable global monitoring