add pid awareness to file locking

[fearful-symmetry]

What does this PR do?

This PR seeks to add pid-awareness to the lockfile system that's initialized when a beat starts up. When beats attempts to create the lockfile, it checks for a pre-existing file, and if it exists, it checks to see if the process listed in the file exists or not. If the process does not exist, (in the case of a bad shutdown where the lockfile never got removed) it will attempt to recover, and create a new lockfile.

Why is it important?

In automated agent systems, the beat can sometimes take to long to shutdown, causing agent to hard-kill the process. In this instance, the beat won't restart, as the lockfile is still hanging around. This fixes that.

• Closes beats locker should be pid aware #31670

Checklist

• Test while running under agent
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-10-11T23:10:15.842+0000

• Duration: 132 min 34 sec

Test stats 🧪

Test	Results
Failed	0
Passed	23656
Skipped	1950
Total	25606

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[cmacknz]

Were you able to reproduce the problem? Did we test this manually using the entire beat process in addition to the unit tests?

[cmacknz]

Why can this be an arbitrary PID?

[cmacknz]

Ah right PID 1 on Linux is the init process that is guaranteed to exist.

Does this still work on Windows, or in general as a cross platform test?

I would also recommend making the 1 a constant for those that either don't know or are slow to remember the significance of PID 1.

[fearful-symmetry]

Yah, that was the simplest way I could think of to get a reproducible pid. Might have to change it for windows...

[cmacknz]

Do we need to handle zombie processes? It seems the agent is able to leave some of them around elastic/elastic-agent#1215

[fearful-symmetry]

Ah, didn't even think of testing that.

[leehinman]

Could we have at least one retry if the os.Remove() fails? The reason is that on Windows if a file was locked, and the process dies the OS will release the lock. But it isn't immediate and is subject to resources, so often waiting 1 sec and trying again it will work.

[fearful-symmetry]

ahhh, interesting, alright.

[fearful-symmetry]

@cmacknz

Were you able to reproduce the problem? Did we test this manually using the entire beat process in addition to the unit tests?

Yup, tested with metricbeat under the usual cases (remaining lockfile for dead process, two beats trying to start at once)

[fearful-symmetry]

Okay, so, only issue that isn't dealt with is how to deal with zombie beats processes. It's definitely a problem, and I'm currently trying to find the most elegant way to handle this cross-platform.

[fearful-symmetry]

This PR now depends on: elastic/elastic-agent-system-metrics#53

[fearful-symmetry]

Alright. This now can handle zombie states for processes. Also managed to test most of the major use cases manually (sudden stop & start without removing the lockfile, starting two beats at once, and re-starting while a zombie process remains), which was a bit of a pain. We should be good for the zombie issue now, at least.

[fearful-symmetry]

Alright, so the typecheck errors in CI are not related, but kinda baffling. I assume the issue (based on what I've hit before) is the fact that the github CI thing has cgo disabled, and the build constraints in elastic-agent-system-metrics require darwin&cgo:

//go:build (darwin && cgo) || freebsd || linux || windows || aix
// +build darwin,cgo freebsd linux windows aix

We'll probably need a separate fix for this, as I assume cgo is disabled for a reason.

[fearful-symmetry]

So, made a few changes based on @andrewkroh 's feedback.

Two things to address still:

• As @andrewkroh mentioned, we should look into fiddling with the shutdown_timeout settings under agent to make this problem less likely in the first place, although this falls outside the scope of this PR.
• My hack to deal with github CI not liking cgo still really bugs me, and I'd like to find another way to handle it.

[fearful-symmetry]

Ah, E2E tests are again being weird:

fatal: [18.218.130.4]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: kex_exchange_identification: Connection closed by remote host", "unreachable": true}

[joshdover]

• As @andrewkroh mentioned, we should look into fiddling with the shutdown_timeout settings under agent to make this problem less likely in the first place, although this falls outside the scope of this PR.

Yeah this makes sense. We should try testing this shortly after getting this in. I think this change is still valuable since we do not know that the queue flush is the only reason the beat is failing to clean this up.

• My hack to deal with github CI not liking cgo still really bugs me, and I'd like to find another way to handle it.

IMO we could solve this as a follow up after getting the PR in.

[fearful-symmetry]

IMO we could solve this as a follow up after getting the PR in.

Agreed, this has been in a PR long enough. Gonna give the E2E tests one last chance to work, then merge.

[cmacknz]

As @andrewkroh mentioned, we should look into fiddling with the shutdown_timeout settings under agent to make this problem less likely in the first place, although this falls outside the scope of this PR.

As far as I can tell the Beats don't actually wait for the queue to empty by default (and this is what the documentation says as well). At least the ones I looked at do not obviously wait for the queue to drain before shutting down, since ShutdownTimeout defaults to 0.

beats/filebeat/beater/filebeat.go

Lines 409 to 412 in 27807aa

	waitPublished := fb.config.ShutdownTimeout > 0 || *once
	if waitPublished {
	// Wait for registrar to finish writing registry
	waitEvents.Add(withLog(wgEvents.Wait,

beats/winlogbeat/beater/winlogbeat.go

Lines 170 to 176 in 27807aa

	if eb.config.ShutdownTimeout > 0 {
	eb.log.Infof("Shutdown will wait max %v for the remaining %v events to publish.",
	eb.config.ShutdownTimeout, acker.Active())
	ctx, cancel := context.WithTimeout(context.Background(), eb.config.ShutdownTimeout)
	defer cancel()
	acker.Wait(ctx)
	}

[cmacknz]

I think at this point in the 8.5 release cycle with only a few days before the release we should target 8.5.1 instead of 8.5.0 with this. This change needs some soak time before release considering the complexity and the chance to create some new unintended failure mode where the beats refuse to start.

This means we should defer merging the backport to the 8.5 branch until after 8.5.0 is released to be safe.

[fearful-symmetry]

This means we should defer merging the backport to the 8.5 branch until after 8.5.0 is released to be safe.

Agreed, this is kind of a scary change, would prefer not to backport anything last-minute.

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @fearful-symmetry? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[lucabelluccini]

Can we confirm if this will only make to 8.6.0? From what I see it will not be backported to 8.5

[amitkanfer]

@lucabelluccini i'm pretty sure this goes into 8.5.1. @jlind23 keep me honest

[cmacknz]

Yes, I had planned to put this in 8.5.1. We are waiting for the 8.5.0 release to occur before back porting, as we are still generating 8.5.0 build candidates due to blockers.

[cmacknz]

@fearful-symmetry I've added the label to create the 8.5.0 backport, we are good to backport for inclusion in 8.5.1 now.