Fix indefinite memory and CPU consumption when waiting fleet to be ready

[AndersonQ]

What does this PR do?

Fixes the wait for Fleet Server to be ready

Why is it important?

When waiting for Fleet Server to start, the Elastic Agent does not account for the timeout when waiting for Fleet Server to be ready.

Currently, when the timeout is reached, the operation isn't interrupted and the goroutine waiting for Fleet Server to be ready gets stuck in an infinite loop without any delay between iterations. It continually prints a log like:

{"log.level":"info","@timestamp":"2024-07-02T13:18:59.354Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":812},"message":"Waiting for Elastic Agent to start: rpc error: code = Canceled desc = context canceled","ecs.version":"1.6.0"} .

This causes a spike in memory and CPU consumption until the agent is killed by the OS, potentially jeopardising the normal operation of the host.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in ./changelog/fragments using the changelog tool
• [ ] I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Try to reproduce #5033, the issue should not be reproducible with this fix.

Related issues

• Closes High memory and CPU consumption when fleet-server fails to start during enroll #5033

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[mergify]

This pull request does not have a backport label. Could you fix it @AndersonQ? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[blakerouse]

Looks good and the test covers the fix. Nice!

[elastic-sonarqube]

Quality Gate passed

Issues
1 New issue
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube