Fix race condition in multiline reader shutdown and other tests

[belimawr]

Proposed commit message

The following tests had race conditions in the test code:
- TestFileWatcher
- TestProspectorHarvesterUpdateIgnoredFiles

TestParsersRabbitMQMultilineLog was failing with the race detector due
to an race condition during shutdown. The race condition is fixed by
adding a mutex and a benchmark (BenchmarkPatternReaderRabbitMQLike) is
added to ensure there is no performance regression.

Benchmark results from BenchmarkPatternReaderRabbitMQLike
Benchstat output:
goos: linux
goarch: amd64
pkg: github.com/elastic/beats/v7/libbeat/reader/multiline
cpu: Intel(R) Core(TM) Ultra 9 285H
                             │ before-10.txt │            after-10.txt            │
                             │    sec/op     │   sec/op     vs base               │
PatternReaderRabbitMQLike-16     750.5µ ± 7%   714.8µ ± 5%  -4.75% (p=0.035 n=10)

                             │ before-10.txt │            after-10.txt             │
                             │      B/s      │     B/s       vs base               │
PatternReaderRabbitMQLike-16    63.11Mi ± 7%   66.26Mi ± 5%  +4.99% (p=0.035 n=10)

                             │ before-10.txt │          after-10.txt          │
                             │     B/op      │     B/op      vs base          │
PatternReaderRabbitMQLike-16    557.6Ki ± 0%   557.8Ki ± 0%  ~ (p=0.579 n=10)

                             │ before-10.txt │          after-10.txt           │
                             │   allocs/op   │  allocs/op   vs base            │
PatternReaderRabbitMQLike-16     5.680k ± 0%   5.680k ± 0%  ~ (p=1.000 n=10) ¹
¹ all samples are equal

Before the changes:
goos: linux
goarch: amd64
pkg: github.com/elastic/beats/v7/libbeat/reader/multiline
cpu: Intel(R) Core(TM) Ultra 9 285H
BenchmarkPatternReaderRabbitMQLike-16    	    1590	    756970 ns/op	  65.61 MB/s	  570284 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    2132	    726478 ns/op	  68.36 MB/s	  570434 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    1486	    738715 ns/op	  67.23 MB/s	  570981 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    1982	    642365 ns/op	  77.31 MB/s	  571042 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    1344	    753941 ns/op	  65.87 MB/s	  570748 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    1504	    774597 ns/op	  64.12 MB/s	  571365 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    1844	    805264 ns/op	  61.67 MB/s	  571732 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    1579	    729805 ns/op	  68.05 MB/s	  570959 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    1574	    818257 ns/op	  60.69 MB/s	  571389 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    1419	    747003 ns/op	  66.48 MB/s	  571037 B/op	    5680 allocs/op
PASS
ok  	github.com/elastic/beats/v7/libbeat/reader/multiline	12.369s

After the changes:
goos: linux
goarch: amd64
pkg: github.com/elastic/beats/v7/libbeat/reader/multiline
cpu: Intel(R) Core(TM) Ultra 9 285H
BenchmarkPatternReaderRabbitMQLike-16    	    2268	    742420 ns/op	  66.89 MB/s	  571058 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    1586	    730727 ns/op	  67.97 MB/s	  571797 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    2067	    699545 ns/op	  70.99 MB/s	  570589 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    1742	    717458 ns/op	  69.22 MB/s	  570487 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    1569	    786419 ns/op	  63.15 MB/s	  571493 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    1429	    703584 ns/op	  70.59 MB/s	  570798 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    1802	    680524 ns/op	  72.98 MB/s	  571433 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    1714	    712191 ns/op	  69.73 MB/s	  571443 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    1783	    722538 ns/op	  68.74 MB/s	  570243 B/op	    5680 allocs/op
BenchmarkPatternReaderRabbitMQLike-16    	    1624	    679507 ns/op	  73.09 MB/s	  571222 B/op	    5680 allocs/op
PASS
ok  	github.com/elastic/beats/v7/libbeat/reader/multiline	12.648s

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works. Where relevant, I have used the stresstest.sh script to run them under stress conditions and race detector to verify their stability.
• I have added an entry in ./changelog/fragments using the changelog tool.

## Disruptive User Impact

How to test this PR locally

Runt the tests multiple times with the race detector

go test -race -count=5 -tags=integration -run="TestFileWatcher|TestProspectorHarvesterUpdateIgnoredFiles|TestParsersRabbitMQMultilineLog" .

## Related issues
## Use cases
## Screenshots
## Logs

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[Copilot]

Pull request overview

Fixes race conditions observed under the Go race detector in Filebeat filestream tests and in the multiline reader shutdown path, and adds a benchmark to validate the mutex change doesn’t regress performance.

Changes:

• Add mutex protection around multiline reader state transitions (pattern/while/counter readers).
• Stabilize filestream tests by removing racy logging assertions and synchronizing shared test metadata storage.
• Add BenchmarkPatternReaderRabbitMQLike and a changelog fragment for the multiline shutdown race fix.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
libbeat/reader/multiline/while.go	Protects state field access with a mutex during Next()/Close() transitions.
libbeat/reader/multiline/pattern.go	Same mutex protection for the pattern-based multiline reader state machine.
libbeat/reader/multiline/counter.go	Same mutex protection for the count-based multiline reader state machine.
libbeat/reader/multiline/benchmark_test.go	Adds benchmark covering a RabbitMQ-like multiline workload to measure mutex overhead.
filebeat/input/filestream/prospector_test.go	Adds a mutex to the mock metadata updater used by tests to avoid map races.
filebeat/input/filestream/fswatch_test.go	Switches warning detection to file-backed test logger to avoid races on in-memory log buffer.
changelog/fragments/1775602464-fix-race-condition-during-multiline-shutdown.yaml	Adds changelog entry for the race-condition fix.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[belimawr]

The failures are unrelated infrastructure issues.

[belimawr]

I agree with linter about switch to errors.Is().

Thanks, I'll update it.

The mutex looks ok. I am wondering if the loadState function is required. Could we just do the lock/unlock in Next? I can live with it either way.

I looked into that, but because there is already setState and resetState, so I believe it makes more sense to keep loadState for consistency: all access to state are made through methods and those methods handle the mutex.

[belimawr]

The CI failure in x-pack/auditbeat/abreceiver TestLeak is unrelated and is already being worked on, see #50386

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix-race-conditions-in-tests upstream/fix-race-conditions-in-tests
git merge upstream/main
git push upstream fix-race-conditions-in-tests

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@x-pack/auditbeat/abreceiver/receiver_leak_test.go`:
- Around line 27-29: The TestLeak function currently unconditionally skips the
entire test; remove the blanket t.Skip and instead add a narrowly-scoped
conditional skip (e.g., based on runtime.GOOS, runtime.GOARCH, or an env var) so
the test body still runs on unaffected platforms; update TestLeak to call
genSocketPath, run the receiver startup/shutdown path, and invoke
oteltest.VerifyNoLeaks except when the specific known flake condition is
detected, and include a comment referencing the issue number for traceability.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: 8b4e1169-715c-4375-aef0-dfda4725b1d9

📥 Commits

Reviewing files that changed from the base of the PR and between cc918f3 and c15312d.

📒 Files selected for processing (2)

• libbeat/reader/multiline/while.go
• x-pack/auditbeat/abreceiver/receiver_leak_test.go

🚧 Files skipped from review as they are similar to previous changes (1)

• libbeat/reader/multiline/while.go

[github-actions]

TL;DR

This Buildkite failure is a merge-conflict failure during pipeline upload, not a test/runtime failure. All failed jobs stop in the repository post-checkout hook when trying to merge main into the PR branch, and the immediate fix is to rebase/merge main and resolve conflicts in two filestream test files.

Remediation

• Rebase belimawr:fix-race-conditions-in-tests on latest main (or merge main) and resolve conflicts in:
  • filebeat/input/filestream/fswatch_test.go
  • filebeat/input/filestream/prospector_test.go
• Push the updated branch and re-run Buildkite for this PR.

Investigation details

Root Cause

The failing step is buildkite-agent pipeline upload ..., but it fails earlier in .buildkite/hooks/post-checkout while creating pr_merge_49980 and auto-merging main into the PR checkout.

The same conflict appears in all 16 failed jobs, so this is a single branch-state/configuration issue (not independent beat failures).

Relevant PR-touched files also include the conflict targets:

• filebeat/input/filestream/fswatch_test.go (PR changes around :286-303 and :391-448)
• filebeat/input/filestream/prospector_test.go (PR changes around :700-769 and :905-888 in the diff context)

Evidence

• Build: https://buildkite.com/elastic/beats/builds/45243
• Example failed job log: /tmp/gh-aw/buildkite-logs/beats-libbeat-pipelinearrow_up-upload-pipeline-buildkitelibbeatpipelinelib.txt:137-149
• Key log excerpt:

Auto-merging filebeat/input/filestream/fswatch_test.go
CONFLICT (content): Merge conflict in filebeat/input/filestream/fswatch_test.go
Auto-merging filebeat/input/filestream/prospector_test.go
CONFLICT (content): Merge conflict in filebeat/input/filestream/prospector_test.go
Automatic merge failed; fix conflicts and then commit the result.
Error: running "repository post-checkout" shell hook: The repository post-checkout hook exited with status 1

Verification

• Not run (read-only detective workflow): diagnosis is based on failed-step logs and conflict signatures across all listed failed jobs.

Follow-up

After rebasing/resolving, if CI still fails, the next failure (if any) should be a real test/build error and can be investigated separately.

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

• Fix race condition in multiline reader shutdown and other tests #49980 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #49980 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[khushijain21]

LGTM

[github-actions]

@Mergifyio backport 8.19 9.3 9.4

[mergify]

backport 8.19 9.3 9.4

✅ Backports have been created

Details

• #50524 [8.19](backport #49980) Fix race condition in multiline reader shutdown and other tests has been created for branch 8.19
• #50525 [9.3](backport #49980) Fix race condition in multiline reader shutdown and other tests has been created for branch 9.3
• #50526 [9.4](backport #49980) Fix race condition in multiline reader shutdown and other tests has been created for branch 9.4