Reuse error channel

[michel-laterman]

What is the problem this PR solves?

SLES instances running a fleet-server instance under the elastic-agent are unhealthy after a restart (fleet-server does not reconfigure properly)

How does this PR solve the problem?

The error channel used by the server struct is being created on each iteration of the run loop: https://github.com/elastic/fleet-server/blob/main/internal/pkg/server/fleet.go#L136

However, runServer call actually occurs in a different goroutine: https://github.com/elastic/fleet-server/blob/main/internal/pkg/server/fleet.go#L117.

Which can lead to a config change being read before the runServer method is ran and leaking the channel; the error passed to the channel is lost and thus the fleet-server is stuck as it should signal that it's failed but has not.

How to test this PR locally

Install an agent with the fleet-server integration on a SLES machine and restart the machine.

Without the changes the logs should stop with:

{"log.level":"info","@timestamp":"2023-06-12T22:16:42.131Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":985},"message":"Unit state changed fleet-server-default-fleet-server-fleet_server-9baba0cd-d2de-47cc-8a9b-d1d2f8c12d66 (STARTING->CONFIGURING): Re-configuring","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server-fleet_server-9baba0cd-d2de-47cc-8a9b-d1d2f8c12d66","type":"input","state":"CONFIGURING","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-12T22:16:42.131Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":985},"message":"Unit state changed fleet-server-default (STARTING->CONFIGURING): Re-configuring","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"CONFIGURING","old_state":"STARTING"},"ecs.version":"1.6.0"}

# This error is lost
{"log.level":"error","@timestamp":"2023-06-12T22:16:42.132Z","message":"fail elasticsearch info","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"cluster.maxConnsPersHost":128,"error.message":"dial tcp: lookup 200dd16c33174ccb94de816bb6fd5c18.us-central1.gcp.qa.cld.elstc.co on [::1]:53: read udp [::1]:38166->[::1]:53: read: connection refused","ecs.version":"1.6.0","service.name":"fleet-server","cluster.addr":["200dd16c33174ccb94de816bb6fd5c18.us-central1.gcp.qa.cld.elstc.co:443"],"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-12T22:16:42.132Z","message":"Stats endpoint (/opt/Elastic/Agent/data/tmp/fleet-server-default.sock) finished: accept unix /opt/Elastic/Agent/data/tmp/fleet-server-default.sock: use of closed network connection","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","log.logger":"fleet-metrics.api","ecs.version":"1.6.0"}

With the changes the logs will have the Fleet Server failed message after.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Closes [SLES15]: Fleet-server Agent gets into offline state on machine reboot. #2431

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-06-20T17:00:09.033+0000

• Duration: 40 min 23 sec

Test stats 🧪

Test	Results
Failed	0
Passed	718
Skipped	1
Total	719

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[jsoriano]

Would it be possible to add a test case for this?

[juliaElastic]

Good catch! The fix looks good, we should add an integration test if possible to simulate the restart scenario.

[michel-laterman]

I was unable to add an explicit test for this bug
We test a config reload with the agent integration tests.

EDIT: Managed to add another reload test at the fleet-level.
also stopped the error that occurs when a fleet-server restarts on a config change from changing the status to failed

[michel-laterman]

/test

[juliaElastic]

can we remove commented out code?