Skip to content

pipeline: remove redundant Paths from Settings and Supporter - #49841

Merged
orestisfl merged 11 commits into
elastic:mainfrom
orestisfl:pipeline-use-info-paths
May 12, 2026
Merged

pipeline: remove redundant Paths from Settings and Supporter#49841
orestisfl merged 11 commits into
elastic:mainfrom
orestisfl:pipeline-use-info-paths

Conversation

@orestisfl

@orestisfl orestisfl commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

Proposed commit message

pipeline: remove redundant Paths from Settings and Supporter

Pipeline stored both beatInfo and a separate *paths.Path, and
threaded it through the processing.Supporter.Create interface just
to call SetPaths on processors. Now that Info carries Paths, remove
the field from Pipeline and Settings, drop the parameter from
Supporter.Create, and let the builder read paths from info.Paths.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works.
  • I have added an entry in ./changelog/fragments using the changelog tool.

Disruptive User Impact

None. Internal API change only.

How to test this PR locally

Related issues

@orestisfl orestisfl added refactoring Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team backport-active-all Automated backport with mergify to all the active branches skip-changelog labels Apr 1, 2026
@botelastic botelastic Bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 1, 2026
@github-actions

github-actions Bot commented Apr 1, 2026

Copy link
Copy Markdown
Contributor

🤖 GitHub comments

Just comment with:

  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

The Pipeline already has beatInfo which contains Paths. Remove the
separate paths field from Pipeline and Settings, and remove the paths
parameter from the processing.Supporter.Create interface. The builder
now uses info.Paths directly when calling SetPaths on processors.
@orestisfl
orestisfl force-pushed the pipeline-use-info-paths branch from cc35ae1 to b662742 Compare April 17, 2026 10:48
@orestisfl
orestisfl marked this pull request as ready for review April 17, 2026 10:48
@orestisfl
orestisfl requested a review from a team as a code owner April 17, 2026 10:48
@elasticmachine

Copy link
Copy Markdown
Contributor

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@github-actions

This comment has been minimized.

@coderabbitai

coderabbitai Bot commented Apr 17, 2026

Copy link
Copy Markdown
Contributor

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: 569b5eff-cad7-43de-9306-51e31f8e6c31

📥 Commits

Reviewing files that changed from the base of the PR and between 2b476b7 and 4b057df.

📒 Files selected for processing (1)
  • x-pack/dockerlogbeat/pipelinemanager/libbeattools_test.go

📝 Walkthrough

Walkthrough

The change removes the Paths field from pipeline.Settings and the paths *paths.Path parameter from the Supporter.Create/processor creation API. Pipeline construction, processor creation, and queue factory initialization no longer pass explicit path arguments; path configuration is provided via beat.Info.Paths instead. Call sites and tests were updated to set beat.Info.Paths (including a new test ensuring getBeatInfo initializes Paths).

🚥 Pre-merge checks | ✅ 2
✅ Passed checks (2 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • 🛠️ Update Documentation: Commit on current branch
  • 🛠️ Update Documentation: Create PR

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

  • Generate code and open pull requests
  • Plan features and break down work
  • Investigate incidents and troubleshoot customer tickets together
  • Automate recurring tasks and respond to alerts with triggers
  • Summarize progress and report instantly

Built for teams:

  • Shared memory across your entire org—no repeating context
  • Per-thread sandboxes to safely plan and execute work
  • Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

This comment has been minimized.

After the merge with main the local beatPaths variables in
dockerlogbeat/libbeattools.go and stress/run.go were left dangling
once Settings.Paths was removed (causing a go vet break in
dockerlogbeat). Address the review feedback by setting info.Paths
on beat.Info instead, so paths flow through Info as the rest of
this PR does.
@orestisfl
orestisfl requested a review from AndersonQ April 30, 2026 10:05
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

Copy link
Copy Markdown
Contributor

TL;DR

All three failing Buildkite jobs are failing in cross-build go build before tests/package logic runs, with the same error obtaining VCS status: exit status 128 error. Immediate unblock is to make Git metadata resolvable inside the crossbuild container (preferred) or run cross-build with -buildvcs=false.

Remediation

  • Ensure the Docker cross-build container can resolve the repo Git metadata after the pipeline checkout change (the container only mounts the workspace path; if .git now points outside it, go build VCS stamping fails).
  • As a short-term unblock, append -buildvcs=false to cross-build go build args/flags for these Buildkite jobs.
  • Re-run:
    • x-pack/dockerlogbeat unit-test build step (mage build unitTest)
    • packetbeat packaging steps (linux and linux arm64)
Investigation details

Root Cause

This is a configuration/infrastructure failure, not a beat-specific code regression:

  • x-pack/dockerlogbeat and both packetbeat packaging jobs fail with the same VCS stamping error from go build.
  • The failure happens inside golang-crossbuild Docker runs before unit tests/assertions or packaging content checks.

Relevant build path in repo:

  • dev-tools/mage/crossbuild.go:362-364 mounts only repoInfo.RootDir into the container (-v <root>:/go/src/..., -w <workdir>).
  • dev-tools/mage/build.go:267 invokes go build without -buildvcs=false.

If the checkout change in this PR alters where Git metadata lives (e.g. .git indirection/info paths outside the mounted workspace), go build cannot collect VCS info and exits with status 128.

Evidence

Verification

Not run locally in this environment; diagnosis is based on the failing job logs and the cross-build invocation path.

Follow-up

After fixing Git metadata visibility (or applying temporary -buildvcs=false), these jobs should progress past the current early build failure, which will confirm this root cause.

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Guards against the nil-Paths regression that broke disk-queue
output configuration on dockerlogbeat after the outputs refactor.

Closes elastic#50450
@orestisfl
orestisfl enabled auto-merge (squash) May 11, 2026 11:20
@orestisfl
orestisfl merged commit 3712f9b into elastic:main May 12, 2026
198 of 201 checks passed
@orestisfl
orestisfl deleted the pipeline-use-info-paths branch May 12, 2026 12:43
@github-actions

Copy link
Copy Markdown
Contributor

@Mergifyio backport 8.19 9.3 9.4

@mergify

mergify Bot commented May 12, 2026

Copy link
Copy Markdown
Contributor

backport 8.19 9.3 9.4

✅ Backports have been created

Details

Cherry-pick of 3712f9b has failed:

On branch mergify/bp/8.19/pr-49841
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 3712f9bc7.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   libbeat/cmd/instance/beat.go
	modified:   libbeat/publisher/pipeline/client_test.go
	modified:   libbeat/publisher/pipeline/stress/run.go
	modified:   libbeat/publisher/processing/default.go
	modified:   libbeat/publisher/processing/default_test.go
	modified:   libbeat/publisher/processing/processing.go
	modified:   x-pack/dockerlogbeat/pipelinemanager/libbeattools.go
	new file:   x-pack/dockerlogbeat/pipelinemanager/libbeattools_test.go
	modified:   x-pack/libbeat/cmd/instance/beat.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   libbeat/publisher/pipeline/pipeline.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

Cherry-pick of 3712f9b has failed:

On branch mergify/bp/9.3/pr-49841
Your branch is up to date with 'origin/9.3'.

You are currently cherry-picking commit 3712f9bc7.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   libbeat/cmd/instance/beat.go
	modified:   libbeat/publisher/pipeline/client_test.go
	modified:   libbeat/publisher/pipeline/stress/run.go
	modified:   libbeat/publisher/processing/default.go
	modified:   libbeat/publisher/processing/default_test.go
	modified:   libbeat/publisher/processing/processing.go
	modified:   x-pack/dockerlogbeat/pipelinemanager/libbeattools.go
	new file:   x-pack/dockerlogbeat/pipelinemanager/libbeattools_test.go
	modified:   x-pack/libbeat/cmd/instance/beat.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   libbeat/publisher/pipeline/pipeline.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

Cherry-pick of 3712f9b has failed:

On branch mergify/bp/9.4/pr-49841
Your branch is up to date with 'origin/9.4'.

You are currently cherry-picking commit 3712f9bc7.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   libbeat/cmd/instance/beat.go
	modified:   libbeat/publisher/pipeline/client_test.go
	modified:   libbeat/publisher/pipeline/stress/run.go
	modified:   libbeat/publisher/processing/default.go
	modified:   libbeat/publisher/processing/default_test.go
	modified:   libbeat/publisher/processing/processing.go
	modified:   x-pack/dockerlogbeat/pipelinemanager/libbeattools.go
	new file:   x-pack/dockerlogbeat/pipelinemanager/libbeattools_test.go
	modified:   x-pack/libbeat/cmd/instance/beat.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   libbeat/publisher/pipeline/pipeline.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

@github-actions github-actions Bot mentioned this pull request May 12, 2026
orestisfl added a commit that referenced this pull request May 13, 2026
…s and Supporter (#50615)

* pipeline: remove redundant Paths from Settings and Supporter (#49841)

pipeline: remove redundant Paths from Settings and Supporter

Pipeline stored both beatInfo and a separate *paths.Path, and
threaded it through the processing.Supporter.Create interface just
to call SetPaths on processors. Now that Info carries Paths, remove
the field from Pipeline and Settings, drop the parameter from
Supporter.Create, and let the builder read paths from info.Paths.

(cherry picked from commit 3712f9b)

# Conflicts:
#	libbeat/publisher/pipeline/pipeline.go

* fix backport: drop NewForReceiver and otel controller leftovers

NewForReceiver and newOTelOutputController/newProcessOutputController
do not exist on this branch; they were introduced on main by
#50075/#50143 which have not been backported. Mergify left
unresolved conflict markers around an attempt to bring them in.
Resolve by keeping the HEAD side (existing newOutputController flow)
while preserving the intended PR change of using beat.Paths in
queueFactoryForUserConfig.

---------

Co-authored-by: Orestis Floros <orestis.floros@elastic.co>
orestisfl added a commit that referenced this pull request May 13, 2026
… and Supporter (#50616)

* pipeline: remove redundant Paths from Settings and Supporter (#49841)

pipeline: remove redundant Paths from Settings and Supporter

Pipeline stored both beatInfo and a separate *paths.Path, and
threaded it through the processing.Supporter.Create interface just
to call SetPaths on processors. Now that Info carries Paths, remove
the field from Pipeline and Settings, drop the parameter from
Supporter.Create, and let the builder read paths from info.Paths.

(cherry picked from commit 3712f9b)

# Conflicts:
#	libbeat/publisher/pipeline/pipeline.go

* fix backport: drop NewForReceiver and otel controller leftovers

NewForReceiver and newOTelOutputController/newProcessOutputController
do not exist on this branch; they were introduced on main by
#50075/#50143 which have not been backported. Mergify left
unresolved conflict markers around an attempt to bring them in.
Resolve by keeping the HEAD side (existing newOutputController flow)
while preserving the intended PR change of using beat.Paths in
queueFactoryForUserConfig.

---------

Co-authored-by: Orestis Floros <orestis.floros@elastic.co>
orestisfl added a commit that referenced this pull request May 13, 2026
… and Supporter (#50617)

* pipeline: remove redundant Paths from Settings and Supporter (#49841)

pipeline: remove redundant Paths from Settings and Supporter

Pipeline stored both beatInfo and a separate *paths.Path, and
threaded it through the processing.Supporter.Create interface just
to call SetPaths on processors. Now that Info carries Paths, remove
the field from Pipeline and Settings, drop the parameter from
Supporter.Create, and let the builder read paths from info.Paths.

(cherry picked from commit 3712f9b)

# Conflicts:
#	libbeat/publisher/pipeline/pipeline.go

* fix backport: drop NewForReceiver leftover from conflict resolution

NewForReceiver does not exist on this branch; it was introduced on main
by #50075/#50143 which have not been backported. Mergify left
unresolved conflict markers around an attempt to bring it in. Resolve
by keeping the HEAD side and removing the function and the markers so
the package compiles.

---------

Co-authored-by: Orestis Floros <orestis.floros@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-active-all Automated backport with mergify to all the active branches refactoring skip-changelog Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug-hunter] dockerlogbeat leaves beat.Info.Paths nil, breaking queue.disk configuration [beatreceiver] move paths object into Beat.Info

3 participants