[Heartbeat] Add publish pipeline timeout to run_once

[emilioalvap]

What does this PR do?

Closes #35706.

Decouples run_once from sync pipeline and adds a timeout before exiting for emitting pending events.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Note
To simplify testing, it's recommended disable exiting on run_once state loader error fmt.Errorf("run_once mode fatal error: %w", err)

1. Build heartbeat locally.
2. Set up a few monitors and run_once mode.
3. Point output to a non-existing ES, like https://localhost:9200/ without it running locally.
4. Run HB, notice it never exits.
5. Stop it and add heartbeat.publish_timeout: 15s to heartbeat.yml
6. Run HB again and notice it exits after 15s, with events still pending:

{"log.level":"info","@timestamp":"2023-06-09T17:52:44.677+0200","log.origin":{"file.name":"beater/signalwait.go","file.line":88},"message":"Ending run_once run.","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-09T17:52:44.677+0200","log.origin":{"file.name":"beater/heartbeat.go","file.line":219},"message":"Shutting down, waiting for output to complete","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-09T17:52:44.677+0200","log.origin":{"file.name":"beater/heartbeat.go","file.line":230},"message":"shutdown: output timer started. Waiting for max 15s.","service.name":"heartbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-06-09T17:52:59.681+0200","log.origin":{"file.name":"beater/signalwait.go","file.line":88},"message":"shutdown: time out waiting for pipeline to publish events.","service.name":"heartbeat","ecs.version":"1.6.0"}

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @emilioalvap? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-07-28T09:56:36.339+0000

• Duration: 56 min 26 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2031
Skipped	25
Total	2056

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

Pinging @elastic/uptime (Team:Uptime)

[emilioalvap]

/test

[vigneshshanmugam]

Just did a first pass, Code changes LGTM. However I did not have a change to run all the test cases yet.

[vigneshshanmugam]

LGTM, Great work 🎉

Nothing blocking though, Added few comments. Also, I tested different modes with HTTP, Browser monitors. Looking good so far.

[vigneshshanmugam]

I don't remember if we ever waited for Kill signal from the OS or any interrupts? Not blocking the PR, just want to understand if we are handling anything differently here.

[emilioalvap]

Yes, any interrupt is handled by libbeat which in turn triggers heartbeat.Stop() and finally <-bt.done. This is just replicating the same condition using the wait signal approach and listening both for interrupts OR run once finished event, if in run once mode.