[9.4](backport #50020) perf: optimize FileMetaReader hot path and EncoderReader map sizing

[mergify]

~20% faster filestream reader pipeline, 29% fewer allocations per file, benefits all inputs using EncoderReader.

BenchmarkFilestream exercises the full filestream reader chain: file I/O → line buffering → UTF-8 encoding → newline stripping → file metadata enrichment → parsers → line limiting → ToEvent() → cursor tracking → publish to pipeline. This measures the complete input-side cost of reading and enriching log lines (no output serialization or network I/O).

Commit 1: FileMetaReader — avoid DeepUpdate, cache metadata

Replace DeepUpdate (recursive map merge) and dotted-key Put with direct map assignment. Lazy-cache per-file metadata (path, device_id, inode, fingerprint) on first Next() call since these values are constant for a file's lifetime. The cached interface{} values are copied via maps.Copy without re-boxing, which is where the alloc savings come from.

                                              │    main        │       commit 1 only        │
  sec/op                                        11.840m ± 2%     9.655m ± 6%  -18.5% (p=0.002)
  allocs/op                                      170.5k ± 0%     120.5k ± 0%  -29.3% (p=0.002)
  B/op                                           15.84Mi ± 0%    15.16Mi ± 0%  -4.3% (p=0.002)

Commit 2: EncoderReader — pre-size Fields map

Change mapstr.M{} (capacity 0) to make(mapstr.M, 1) so the first downstream write doesn't trigger a map grow. Benefits all inputs using EncoderReader: filestream, legacy log, and S3.

                                              │    main        │      both commits          │
  sec/op                                        11.840m ± 2%     9.541m ± 5%  -19.4% (p=0.002)
  allocs/op                                      170.5k ± 0%     120.5k ± 0%  -29.3% (p=0.002)
  B/op                                           15.84Mi ± 0%    15.16Mi ± 0%  -4.3% (p=0.002)

Safety analysis

• Thread safety: FileMetaReader runs in a single harvester goroutine — no concurrent access to cachedMeta.
• No reference escapes: Each event gets its own fileMap via make + maps.Copy. Downstream code (LimitReader, parsers, ToEvent, publishers) can freely mutate the per-event map without corrupting the cache.
• Immutable values: All cached values are Go strings (immutable). Copying the interface{} wrappers avoids re-boxing.
• Values are truly constant: fi is captured at file-open time; GetOSState() returns a stored struct. path and fingerprint are constructor args.
• Error path: cachedMeta is built into a local variable and only assigned on success. A failed setFileSystemMetadata leaves cachedMeta nil so the next call retries — identical behavior to the original per-call code.
• Output schema unchanged: The nested structure log.file.device_id is identical — only the internal construction path changed from dotted-key traversal to direct map assignment.
• make(mapstr.M, 1) is a drop-in replacement for mapstr.M{} — both produce an empty, non-nil map. The only difference is the capacity hint. len(), == nil, and all standard operations behave identically.

Raw benchmark data (benchstat, 6 runs each, Apple M2 Pro)

BenchmarkFilestream — full pipeline, 10k lines

                                           │     main       │       commit 1 only        │        both commits         │
                                           │     sec/op     │   sec/op     vs base       │   sec/op     vs base        │
Filestream/single_file/inode_throughput-10    11.840m ± 2%    9.655m ± 6%  -18.45%         9.541m ± 5%  -19.42%
                                           │     B/op       │                             │                             │
                                             15.84Mi ± 0%   15.16Mi ± 0%  -4.34%         15.16Mi ± 0%  -4.34%
                                           │   allocs/op    │                             │                             │
                                              170.5k ± 0%    120.5k ± 0%  -29.32%         120.5k ± 0%  -29.32%

All comparisons p=0.002 n=6.

BenchmarkFileMetaReaderNext — isolated FileMetaReader

                                       │   main         │          this PR              │
FileMetaReaderNext/no-fingerprint-10      587.0n ± 1%     370.5n ± 21%  -36.9% (p=0.002)
FileMetaReaderNext/with-fingerprint-10    652.6n ± 1%     384.0n ±  8%  -41.2% (p=0.002)
geomean                                   619.0n          377.2n        -39.1%

allocs/op: 11 → 7 (-36%). B/op: 1072 → 1016 (-5%).

BenchmarkEncoderReader — commit 2 improvement (benefits all inputs)

                                    │   main     │   make(mapstr.M, 1)        │
EncoderReader/long_lines-10           867.8µ ± 8%  815.7µ ± 3%  -6.0% (p=0.002)
EncoderReader/skip_lines-10           101.3µ ± 2%   93.7µ ± 2%  -7.5% (p=0.002)
EncoderReader/short_lines-10          10.91µ ± 4%  10.89µ ± 4%       ~ (p=0.589)
EncoderReader/buffer-sized_lines-10   57.10µ ± 3%  56.19µ ± 2%       ~ (p=0.394)
geomean                               86.01µ       82.69µ       -3.9%

Test plan

• Existing TestMetaFields / TestMetaFieldsOwnerAndGroup pass (updated for new map structure)
• New BenchmarkFileMetaReaderNext added for regression detection
• Windows cross-compilation verified (GOOS=windows GOARCH=amd64 go build)
• CI: Linux + Windows unit tests
• CI: integration tests

🤖 Generated with Claude Code

---

This is an automatic backport of pull request #50020 done by Mergify.

[github-actions]

🤖 GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /test : Run the Buildkite pipeline.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mauri870]

/test

[github-actions]

TL;DR

Libbeat: Run check/update fails due to a compile-time type reference error in libbeat/publisher/pipeline/client_test.go:220: the test closure uses queue.Entry, but that type does not exist in libbeat/publisher/queue/queue.go. Update that argument type to publisher.Event.

Remediation

• Change libbeat/publisher/pipeline/client_test.go:220 from:
  publish: func(_ bool, event queue.Entry) (queue.EntryID, bool) {
  to:
  publish: func(_ bool, event publisher.Event) (queue.EntryID, bool) {
• Re-run the failing Buildkite step (make -C libbeat check update && make check-no-changes) and confirm go vet ./... passes.

Investigation details

Root Cause

go vet fails in libbeat/publisher/pipeline because queue.Entry is referenced in a test, but queue defines EntryID and generic interfaces (Batch[T].Entry(i int) T) and has no Entry type.

• Failing symbol: libbeat/publisher/pipeline/client_test.go:220
• Missing type definition: libbeat/publisher/queue/queue.go (no type Entry ...)

This is a code bug (type mismatch/invalid symbol), not infrastructure.

Evidence

• Build: https://buildkite.com/elastic/beats/builds/46325
• Job/step: Libbeat: Run check/update (beats-libbeat build 29824)
• Key log excerpt (/tmp/gh-aw/buildkite-logs/beats-libbeat-libbeat-run-checkupdate.txt):

# github.com/elastic/beats/v7/libbeat/publisher/pipeline
vet: publisher/pipeline/client_test.go:220:38: undefined: queue.Entry
Error: failed running go vet, please fix the issues reported: running "go vet ./..." failed with exit code 1
make: *** [scripts/Makefile:141: check] Error 1

Verification

• Not run locally in this workflow; diagnosis is based on the Buildkite failure log plus source inspection of the PR head revision.

Follow-up

If there are other backported test hunks around pipeline client race fixes, quickly grep for additional queue.Entry references and convert them to publisher.Event for consistency.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.