Decouple xds capacity controller and raft-autopilot

[hashi-derek]

This prevents a potential bug where raft-autopilot could deadlock while attempting to execute AutopilotDelegate.NotifyState() on an xdscapacity controller that has stopped / delayed consuming messages.

The following line of logic appears that it could be a potential problem:

1. The call to countProxies() can wait for up to 1 minute before retrying to load info. It also never resets its counter, which means a significant wait is more likely. https://github.com/hashicorp/consul/blob/v1.17.2/agent/consul/xdscapacity/capacity.go#L194

2. countProxies() shares a loop with the consumption from the serverCh that has counts published to it. https://github.com/hashicorp/consul/blob/v1.17.2/agent/consul/xdscapacity/capacity.go#L93-L97

3. The serverCh channel is published to by the SetServerCount() function. https://github.com/hashicorp/consul/blob/v1.17.2/agent/consul/xdscapacity/capacity.go#L113

4. The autopilot state delegate calls SetServerCount() on every change. https://github.com/hashicorp/consul/blob/main/agent/consul/autopilot.go#L68

5. Autopilot acquires an exclusive lock and waits for the delegate to finish execution. https://github.com/hashicorp/raft-autopilot/blob/v0.1.6/state.go#L390-L393

[mkeeler]

This seems fine but the better solution is to probably not directly call SetServerCount from NotifyState. Instead, the xds capacity controller should use the event publisher to listen for server events and update itself.

[mkeeler]

As the goal here is to backport pretty far this limited change makes sense over rethinking the interaction completely.

[hashi-derek]

This seems fine but the better solution is to probably not directly call SetServerCount from NotifyState. Instead, the xds capacity controller should use the event publisher to listen for server events and update itself.

Agreed. I will take that into consideration when we implement the corresponding changes for the new catalog in a future release.

[hc-github-team-consul-core]

@hashi-derek, a backport is missing for this PR [20511] for versions [1.15,1.16,1.17] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>