Fix/change the initialization of management layer

[ph]

This fix an issue on Filebeat that makes the start sequence of Filebeat
non-deterministic. It was possible that not all the hooks were
configured correctly before the managed was receiving a configuration
from the Elastic Agent.

This causes inconsistency between the expected configuration state coming from Agent
and the actual running state. This situation can have one or many of the following symptoms:

• Having Filebeat runnings and not sending any data to Elasticsearch
• Having Filebeat partially configured, when only some inputs were
  sending data.
• Missing log from the Filebeat collector
• Having only Metricbeats running and sending logs.
• A problematic process is restarted by the agent.

This solves the issues by moving the Start and stop Stop of the
manager into the beats initialization process, each beat need to be
adjusted to support this new sequence.

This is indeed a breaking change for beats author,
but the bootstrap process of beats and libbeat cannot easily be
extended to make the change into a unique place.

Every beats has a different code path.

How it was detected

This was detected on a log where log events were actually missing from the log.

Working endpoint.

{"log.level":"info","@timestamp":"2022-03-03T15:21:46.739+0100","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":109},"message":"Starting fleet management service","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-03T15:21:47.354+0100","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":150},"message":"Status change to Configuring: Updating configuration","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-03T15:21:47.354+0100","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":271},"message":"Applying settings for filebeat.inputs","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-03T15:21:47.354+0100","log.logger":"centralmgmt","log.origin":{"file.name":"cfgfile/list.go","file.line":63},"message":"Starting reload procedure, current runners: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-03T15:21:47.355+0100","log.logger":"centralmgmt","log.origin":{"file.name":"cfgfile/list.go","file.line":81},"message":"Start list: 2, Stop list: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-03T15:21:47.356+0100","log.logger":"centralmgmt","log.origin":{"file.name":"cfgfile/list.go","file.line":105},"message":"Starting runner: input [type=log]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-03T15:21:47.358+0100","log.logger":"centralmgmt","log.origin":{"file.name":"cfgfile/list.go","file.line":105},"message":"Starting runner: input [type=log]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-03T15:21:47.358+0100","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":271},"message":"Applying settings for output","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-03T15:21:47.358+0100","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":102},"message":"elasticsearch url: https://<redacted>:443","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-03T15:21:47.358+0100","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":271},"message":"Applying settings for filebeat.modules","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-03T15:21:47.358+0100","log.logger":"centralmgmt","log.origin":{"file.name":"cfgfile/list.go","file.line":63},"message":"Starting reload procedure, current runners: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-03-03T15:21:47.358+0100","log.logger":"centralmgmt","log.origin":{"file.name":"cfgfile/list.go","file.line":81},"message":"Start list: 0, Stop list: 0","service.name":"filebeat","ecs.version":"1.6.0"}

Problematic endpoint

{"log.level":"info","@timestamp":"2022-03-03T11:20:41.207+0100","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":109},"message":"Starting fleet management service","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-03T11:20:41.732+0100","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":150},"message":"Status change to Configuring: Updating configuration","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-03T11:20:41.733+0100","log.logger":"centralmgmt.fleet","log.origin":{"file.name":"management/manager.go","file.line":271},"message":"Applying settings for output","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-03-03T11:20:41.733+0100","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":102},"message":"elasticsearch url: https://<redacted>:443","service.name":"filebeat","ecs.version":"1.6.0"}

The later log extract only contains information about the outputs (`Applying settings...) nothing about the inputs.

What does this PR do?

Why is it important?

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
  - [ ] I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

Since the problem is non-deterministic reproducing this issue really hard, I was able to reproduce a few times by having simulated load on agent virtual machine.

Related issues

• Closes Filebeat running under Elastic-Agent not harvesting logs after restart #30533

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mergify]

This pull request does not have a backport label. Could you fix it @ph? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
[](https://ci-stats.elastic.co/app/apm/services/beats-ci/transactions/view?rangeFrom=2022-03-09T16:39:31.895Z&rangeTo=2022-03-09T16:59:31.895Z&transactionName=BUILD Beats/beats/PR-{number}&transactionType=job&latencyAggregationType=avg&traceId=4bc4c0542f43ada17ec92de9fd81a8cf&transactionId=b5a59d7b4c3a3b35)

Expand to view the summary

Build stats

• Start Time: 2022-03-09T16:49:31.895+0000

• Duration: 120 min 48 sec

Test stats 🧪

Test	Results
Failed	0
Passed	43016
Skipped	3846
Total	46862

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[ph]

/package

[ph]

The failure looks valid to me, I will take look.

[ph]

I've looked at the issues, I am going to rebase this PR and have another go with the test, I've added a changelog too.

[ph]

I've tested this PR using one of our vagrant machines (vagrant up ubuntu2004), I've installed the stress utility and put a really large CPU load on the machine, the machine was really slow, and I've done a few restarts of the service. Every time Filebeat and Metricbeat were back online and sending events to Elasticsearch.

[ph]

@aleksmaus can you take a look at the osquerybeat part?

[ph]

@simitt Thanks, I've fixed the typo add added more information for the SetStopCallback see https://github.com/elastic/beats/pull/30694/files#diff-fcf0ac1927a6e4a560125bca3691cb6c27006277664735a58500420535fedc27R94

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix/change-the-initialization-manager upstream/fix/change-the-initialization-manager
git merge upstream/main
git push upstream fix/change-the-initialization-manager

[ph]

It seems I've missed the alert of mergify concerning the conflict. I will make a followup PR.

[ph]

it seems, I've fixed that last week , time change are hard. :(