Add support for quicker Fleet check-in upon component status

[moukoublen]

What does this PR do?

At the moment, the fleet server "holds" a long poll of 5 minutes on every elastic agent check-in. As a result, when an input has an issue and is in an error or degraded state, information is not relayed to Fleet until at most 5 minutes.

In this PR, a "state watch" was introduced into the fleet gateway component to cancel ongoing check-in requests if a state change occurs while the elasti-agent waits on check-in.

The flow is this:

• During checking start, (using a mutex) the current state is fetched and a ctx with cancel is created. The cancel function is stored for later usage.
• On the happy path (check-in completes successfully), the cancel function is being cleaned up.
• If a state update is received in the state channel while waiting for a check-in response (long poll), the context is canceled due to a specific cause. This leads to the check-in re-start with the new updated state

Worst case scenario: During the agent's start-up, potentially many state changes might occur in the inputs till the "healthy" state is achieved. At that moment, a lot of cancellations might happen. The already implemented backoff mechanism covers this case as well.

The "state watch and cancel ongoing checkin" functionality works only if the fleet agent config checkin.mode is set to on_state_changed (will be set by agentless controller on agentless). Otherwise, the flow works without any change.

New configuration checkin added under the FleetAgentConfig

fleet:
  # ...
  mode: "standard" # choose between `standard` or `on_state_change`
  request_backoff_init: 5s # default 60 seconds
  request_backoff_max: 10m # default 10 mins

On agentless we can set (through the agentless controller) the above config to enable fast checkin.

Why is it important?

On agentless environments where the agent logs are not available to the customer, the only way to communicate operational issues (e.g., misconfigured credentials) is through the degraded state status message of the input/component. But creating an agentless agent with wrong credentials will take up to 5 minutes to display the degraded status, and meanwhile, the input will appear as healthy. This provides a bad experience to the customer.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Related issues

• Closes Add support for quicker Fleet check-in upon component status #9348

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[cmacknz]

Worst case scenario: During the agent's start-up, potentially many state changes might occur in the inputs till the "healthy" state is achieved. At that moment, a lot of cancellations might happen. The already implemented backoff mechanism covers this case as well.

Are we sure about this? The limiting factor for the number of agents Fleet can manage is the frequency of authentication and API key operations in general usually, this has the potential to drastically increase those. The backoff mechanism is still faster than 5 minutes.

The safest way to introduce this is to put it behind a configuration flag, that is initially only turned on in agentless.

Then we can look at enabling it in the Horde based scale tests, which will need this change ported into the Horde drones and require us to add the ability for the Horde drone health status to change.

This is a very high consequence change as bugs or interactions here can potentially DDoS all existing Fleet Servers accidentally.

We are absolutely going to need more than a unit test to validate this before enabling it by default. As mentioned already, I'm OK introducing this behind a feature toggle as long as the change has no impact when it is disabled.

Probably the fleet configuration section is the place to put a configuration flag for this.

[blakerouse]

Overall this looks good. I am going to wait for a green CI before approval, but this looks good.

LGTM, tomorrow is a holiday for me so delegated approval to @blakerouse and removing my request for changes.

[elastic-sonarqube]

Quality Gate passed

Issues
5 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
63.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0ff1cce

History

• 💚 Build #27719 succeeded dc21d56
• 💔 Build #27690 failed 3332130
• 💔 Build #27671 failed b4257d9
• 💔 Build #27662 failed 51fe3cb
• 💔 Build #27658 failed a1ff973
• 💚 Build #27565 succeeded fcc1752

cc @moukoublen

[moukoublen]

@blakerouse CI seems ok now

[blakerouse]

Looks good. Thanks for working through this with me and getting it to be mergable.