Improve package backport tooling for routine multi-version support

[kpollich]

The current backport process (docs) is a 5-step manual procedure built for one-off hotfixes. As more packages adopt format_version 3.5/3.6 and need to maintain fixes on 8.19 (under maintenance until Jan 2027), it's not viable for routine use.

Gaps:

1. Branch creation - manual Buildkite pipeline today; needs a CLI command or API surface
2. Cherry-pick + version bump + changelog entry - no automation; needs a single operation from a commit SHA + target version
3. Main-branch changelog update PR - un-automated and easy to skip; needed to keep the integrations changelog site current
4. Backport branch inventory - no record of which branches exist and which packages they cover; needed to auto-create backport PRs when changes land in main

Proposal:

Build a script or elastic-package command that accepts a commit SHA and target version and handles gaps 1-3 as a single operation. Define a backport branch manifest (in-repo file or derivable from branch naming) to power future auto-backport PR creation (gap 4).

Context: Near-term unblock for security service integrations adopting 9.4 features while keeping 8.19 support. See elastic/package-spec#1148 for policy context and elastic/package-spec#1163 for long-term options.

AC:

• Branch creation scriptable without manual Buildkite inputs
• Cherry-pick + version bump + changelog entry runnable as a single command
• Main-branch changelog update PR automated or explicitly documented
• Backport branch inventory defined and populated for existing branches

[infra-vault-gh-plugin-prod]

Pinging @elastic/ecosystem (Team:Ecosystem)

[kpollich]

cc @mrodm Hope this makes sense - happy to work through this for a quick grooming/definition pass before we start any implementation. Let me know if you'd like some help getting a plan together for this.

[github-actions]

tl;dr: #19016 is a valid next-phase gap (not a duplicate of #17403/#18577) — branch creation exists, but routine backport execution is still mostly manual (especially cherry-pick + version bump + changelog flow), so I recommend implementing a single non-interactive backport command and a derivable inventory primitive.

Recommendation

Implement this issue as phase 2 of backport tooling:

1. Add a non-interactive command that takes commit SHA + target version (or target series) and performs cherry-pick, package version bump, and changelog entry in one operation.
2. Keep existing branch-creation logic, but expose it without UI-only Buildkite input coupling.
3. Add an explicit inventory output (derivable JSON) for backport branches/packages to enable future auto-backport PR generation.

Findings

What is already implemented

• Backport branch creation exists in Buildkite and is parameterized:
  • .buildkite/pipeline.backport.yml:20-50 (input fields)
  • .buildkite/pipeline.backport.yml:52-54 (runs .buildkite/scripts/backport_branch.sh)
• Backport script resolves package path, validates version/commit, creates/pushes branch:
  • .buildkite/scripts/backport_branch.sh:48 (branch naming backport-${PACKAGE_NAME}-${TRIMMED_PACKAGE_VERSION})
  • .buildkite/scripts/backport_branch.sh:92-106 (get_package_path)
  • .buildkite/scripts/backport_branch.sh:300-314 (manifest version checks + branch creation)

What remains manual (matches issue gaps)

• Docs still describe manual multi-step process:
  • docs/extend/developer-workflow-support-old-package.md:10 (manual 5-step procedure)
  • docs/extend/developer-workflow-support-old-package.md:80-84 (manual bugfix PR flow)
  • docs/extend/developer-workflow-support-old-package.md:90-94 (manual main-branch changelog PR)
• No single operation currently performs cherry-pick + version bump + changelog entry from SHA + target version.
• No explicit in-repo/derived branch inventory artifact currently exists; current discovery primitives return package paths only:
  • .buildkite/scripts/common.sh:904-906 (list_all_directories)
  • magefile.go:221-231 (ListPackages prints package paths)
  • dev/citools/packages.go:19-50 (returns sorted package paths)

Related history indicates this is not duplicate work

• #17403 closed as completed for nested package directory support and CI compatibility, including backport script compatibility updates.
• #18577 merged those compatibility updates (including edits to backport pipeline/script), but did not implement end-to-end routine backport automation.

Verification

I attempted to run local Go-based tooling/tests to validate execution paths, but this runner’s Go toolchain is below repo minimum:

$ go test ./dev/citools ./dev/codeowners ./dev/packagenames
go: go.mod requires go >= 1.26.0 (running go 1.25.10; GOTOOLCHAIN=local)

$ go run github.com/magefile/mage -d . listPackages | head -n 20
go: go.mod requires go >= 1.26.0 (running go 1.25.10; GOTOOLCHAIN=local)

Code-path findings above are from direct repository inspection.

Detailed Action Plan

1. Create a non-interactive backport apply command

   • Add a script under .buildkite/scripts/ (or dev/) that accepts: --base-sha, --package-name, --from-version, --to-version (or target series).
   • Reuse logic from .buildkite/scripts/backport_branch.sh:92-106 (package resolution) and .buildkite/scripts/backport_branch.sh:300-314 (version integrity checks).

2. Automate fix application in one operation

   • In that script, perform:
     1. checkout/create target backport-* branch,
     2. cherry-pick SHA,
     3. patch version bump in package manifest.yml,
     4. append package changelog.yml entry.
   • Keep hard-fail behavior on conflicts/invalid version transitions.

3. Define branch/package inventory output (gap 4)

   • Add a citools primitive that outputs JSON rows like {package_name, package_path, current_version, backport_branch} derived from:
     • package discovery (dev/citools/packages.go:19-50)
     • branch naming convention (.buildkite/scripts/backport_branch.sh:48)
   • Expose via new Mage target near ListPackages (magefile.go:221-231).

4. Integrate with backport pipeline + docs

   • Extend .buildkite/pipeline.backport.yml with a non-UI execution path (script args/env) to support automation/repeatability.
   • Update docs/extend/developer-workflow-support-old-package.md to replace manual steps with command-driven flow and explicitly define how main-branch changelog PR is handled (automated vs required manual command).

5. Follow-up (optional but recommended)

   • Use inventory JSON as input to future auto-backport PR generation when relevant package changes land on main.

Related Items

Type	Link	Relevance
Issue	#19016	Current request
Issue	#17403	Completed nested package-dir support; prerequisite context
PR	#18577	Updated CI/backport scripts for nested dirs; did not implement routine backport automation
File	.buildkite/pipeline.backport.yml:20-54	Current backport entrypoint and inputs
File	.buildkite/scripts/backport_branch.sh:48,92-106,300-314	Existing branch creation/validation logic to reuse
File	docs/extend/developer-workflow-support-old-package.md:10,80-94	Current manual workflow and changelog step
File	magefile.go:221-231	Existing package listing target
File	dev/citools/packages.go:19-50	Package discovery primitive

Note

🔒 Integrity filter blocked 8 items

The following items were blocked because they don't meet the GitHub integrity level.

• #14421 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #14419 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #14420 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #13874 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #14064 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #6237 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #18495 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• [CI] Support package discovery in nested directories - WIP #18495 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[mrodm]

cc @mrodm Hope this makes sense - happy to work through this for a quick grooming/definition pass before we start any implementation. Let me know if you'd like some help getting a plan together for this.

Thanks @kpollich! That all makes sense. I'd like to take some time to review the plan and draft a few ideas on my end first before we jump into implementation. I'll get back to you once I have something to share and we can align from there.

[mrodm]

I've been looking at this issue and wanted to share my current understanding of the problem and a rough proposal to get some early feedback to make sure we're aligned before writing up a detailed breakdown of the work. @kpollich @elastic/ecosystem

Current state

The documented backport workflow is a 5-step manual procedure that was designed for one-off hotfixes. As more packages move to format_version 3.5/3.6 and need to maintain fixes on 8.19 (under maintenance until January 2027), it is becoming a recurring pain point.

The biggest friction today: there is no way to trigger a backport branch creation from a laptop. Two things block it:

1. The Buildkite pipeline enforces UI-only triggering — there is an explicit guard that exits with an error if the build is not started from the Buildkite UI.
2. backport_branch.sh depends on the buildkite-agent binary, which only exists inside a running Buildkite agent environment. The env var fallbacks in the script don't help here.

Beyond branch creation, the remaining steps — cherry-pick, version bump in manifest.yml, changelog.yml entry, and opening a PR — are all manual and need to be repeated for every active backport branch the touched package has.

What I'm thinking

The goal would be to cover the four Acceptance Criterias in this issue through a small set of scripts and two GHA workflows, all living in elastic/integrations.

Developer applying a fix manually would go from a 5-step procedure to a single command:

dev/scripts/backport_apply.sh \
  --sha abc123 \
  --package aws \
  --target 6.14 \
  --open-pr

The script handles cherry-pick, version bump, changelog.yml entry via elastic-package changelog add --next patch, and opens a PR ready for review. If the backport branch doesn't exist yet, --create-if-missing triggers the Buildkite pipeline via its REST API and waits until the branch is ready before continuing. Conflict? The script aborts cleanly and prints the exact files in conflict and the command to run manually.

Automated backport on every merge to main: a GHA detects which packages were touched, looks up active backport branches from a .backports.yml inventory file at the repo root — for example:

backports:
  - package: aws
    branch: backport-aws-6.14
    maintained_until: "2027-01-15"   # 8.19 EOL
    archived: false
  - package: aws
    branch: backport-aws-6.15
    maintained_until: null           # supporting 9.4 users
    archived: false

The GHA then calls the script for each active branch. On success it opens a PR ready for review and comments on the original with a link. On conflict it comments on the original PR mentioning the author with the list of conflicting files and the exact manual command — no one needs to remember to check.

The default would be opt-out: backport attempted for every merged PR. Two labels give authors control:

Label	Effect
(none)	Backport attempted for all active branches matching the touched packages
backport:skip	Skip backport for this PR
backport	Force backport (useful if we later add type-based filtering)

The backport label is not currently used by Mergify so there's no conflict. If the default-all behaviour turns out to be too noisy we can narrow it to bugfix entries only in a follow-up — the label infrastructure would already be in place.

Changelog kept in sync automatically: a separate GHA on push to backport-<package>-* branches keeps main's changelog.yml in sync by opening a PR to insert each new version entry at the right semver position. No manual step needed after a backport lands.

Explicit decisions worth calling out:

1. Conflict resolution stays manual. The tooling surfaces everything needed but doesn't attempt to resolve conflicts.
2. Backport branch creation still runs inside the Buildkite pipeline. The script triggers it via the REST API rather than running the logic locally, which preserves the existing guarantee that only CI infrastructure files are written to the branch and no package changes can sneak in during creation.
3. Auto-generated PRs are opened as ready for review.

A few things I'd like your input on before going further

1. Script location — I'm leaning towards dev/scripts/ as a new subfolder under dev/ (which currently has only Go modules). Does that feel right, or would you prefer a top-level scripts/ folder or somewhere else?

2. Opt-out default — auto-backporting every merged PR touching a package with active backport branches: does that feel too aggressive? If so, what would be a better default — only bugfix changelog types, only PRs with the backport label, something else?

3. Packages to exclude — are there packages or scenarios where we know upfront that auto-backport should never run, regardless of labels?

4. PR state — auto-generated backport PRs would be opened as ready for review. Should they be drafts instead, to avoid pressuring anyone until they've been manually checked?

Happy to share the full design doc if that would be useful context.

[mrodm]

Opened for review a PR with a script to help developers to find the right commit: #19104

This PR also updates the documentation about how to backport a fix to older versions.

[mrodm]

As part of this process, a new validation to check links set in the changelog entries is going to be added via #19171

As this automation will be updating packages in different branches (including changelog and versions) as well as the step to sync the changelog back to main, this new validation will be helpful to avoid any error in the automation process.

[mrodm]

@elastic/ecosystem Following up on my earlier comment with a more concrete breakdown of the approach after incorporating the feedback received. The goal is still the same — cover all four acceptance criteria — but several things have evolved based on the discussion.

What's changed since the earlier comment

Branch creation is now PR-driven. Rather than running a local script, a developer (or agent) opens a PR adding a new entry to a .backports.yml inventory file at the repo root. On merge, create-backport-branch.yml detects the new entry, triggers the existing Buildkite pipeline via its REST API, and posts a comment on the PR when the branch is ready. Developers are not going to trigger the Buildkite pipeline anymore — anyone who can open a PR can request a new backport branch. The Buildkite pipeline still runs the actual branch creation, so the existing guarantee (only CI infrastructure files written, no package changes) is preserved.

Note: dev/scripts/get_release_commit.sh from #19104 fits in naturally here — create-backport-branch.yml will call it internally to derive the base commit from the package version. In any case, it would be interesting to include in the .backports.yml entry both the version number (base_version) and the commit SHA (base_commit).

.backports.yml is the single source of truth for the whole process. Every GitHub Actions workflow reads this file before acting — a branch is considered inactive if archived: true OR its maintained_until date has passed, and dead branches are never targeted. The same file drives both branch creation and fix propagation. A one-time generation script using git ls-remote will bootstrap it from existing branches, defaulting those with no commit in the past year to archived: true.

backports:
  - package: aws
    branch: backport-aws-6.14
    base_version: 6.14.2             # used to find the creation commit automatically
    base_commit: "cd8d67e56e"
    maintained_until: "2027-01-15"   # 8.19 EOL
    archived: false
  - package: aws
    branch: backport-aws-3.17
    base_version: 3.17.0
    base_commit: "5b593f6681"
    maintained_until: null
    archived: false

To be considered: It could be added more information like spec version or kibana constraint set in this package.

Checklist instead of a label. When a PR targeting main is opened or updated, backport-packages-detect.yml detects which packages are touched and saves the result as an artifact. post-backport-checklist.yml picks up the artifact and posts (or updates) a checklist comment if any of those packages have active backport branches — no label needed. The author ticks the branches they want to target; nothing is backported until a box is checked.

## Backport checklist

Check the branches to backport this change to. PRs will be created automatically
on merge, or when you update this checklist after merge.

**aws**
- [ ] `backport-aws-6.14` (maintained until 2027-01-15)
- [ ] `backport-aws-3.17`

This is implemented as a two-workflow pair because PRs from forks cannot use write-permission workflows directly: backport-packages-detect.yml runs in the restricted pull_request context (read-only, no secrets), while post-backport-checklist.yml runs in the base repository context via workflow_run and posts the comment with write permissions.

If the PR scope changes (new commits touch more packages), the comment is updated while preserving any already-checked boxes. Checking boxes after merge is also supported — auto-backport.yml re-triggers via issue_comment and creates backport PRs for the newly checked branches.

This replaces the opt-out label approach described in the earlier comment (backport:skip / backport). The checklist is opt-in by default — nothing is backported until a box is checked, so there is no noise on PRs that don't need backporting.

Remaining components

• backport_apply.sh — cherry-pick, version bump, changelog.yml entry, and PR creation as a single command. Uses git cherry-pick -n so the original changelog entry is not imported; calls elastic-package changelog add --next patch fresh instead. Can also be run directly by a developer to manually apply a fix to a specific backport branch.
• post-backport-branch.yml (new) — fires after the backport PR is opened and replaces the placeholder link in changelog.yml with the actual backport PR URL. Mirrors the post-dependabot.yml pattern in elastic-agent.
• auto-backport.yml — triggers on push to main; reads the checked boxes in the checklist comment and calls backport_apply.sh for each selected branch. On success it updates the checklist line with the PR link. On conflict it updates the line mentioning the author with the list of conflicting files and the exact manual command. This workflow is also triggered with the issue_comment event to create more backport Pull Requests, if any checkbox in the message is checked once the Pull Request is merged.
• sync-backport-changelog.yml — triggers on push to any backport-<package>-* branch and opens a PR against main to insert the new version entry at the right semver position. These PRs are labelled automation so they don't re-trigger auto-backport.yml.
• Conflict handling — script aborts cleanly, prints which files conflict, and surfaces the exact manual command. No auto-resolution.

GitHub Actions workflows summary

Workflow	Trigger	Purpose
create-backport-branch.yml	push to main (.backports.yml changed)	Triggers Buildkite to create the branch; comments on the originating PR when ready
backport-packages-detect.yml	pull_request opened/synchronize → main	Detects touched packages; saves artifact (read-only, fork-safe)
post-backport-checklist.yml	workflow_run of backport-packages-detect	Posts or updates the backport checklist comment
auto-backport.yml	push to main; issue_comment edited	Creates backport PRs for checked branches; updates checklist with results
post-backport-branch.yml	pull_request opened → backport-* (auto-backport/ prefix)	Updates changelog.yml link to the backport PR URL
sync-backport-changelog.yml	push to backport-*	Syncs new version entry into main's changelog.yml

Manually created backport Pull Requests

Developers can still open backport PRs by hand — for example, by running backport_apply.sh directly or by creating the branch and PR themselves. In that case post-backport-branch.yml will not fire if the PR head branch does not follow the auto-backport/<package>-<major.minor>-<sha> naming convention, so the changelog.yml entry will keep the original PR link as a placeholder rather than being updated to the backport PR URL. Fork-based backport PRs created outside the automated flow have the same limitation.

One thing that will still work regardless: once the PR merges into a backport-<package>-* branch, sync-backport-changelog.yml triggers on the push event and opens a PR against main to sync the changelog entry. The branch naming convention is the only requirement.

Assumptions

1. Script location — dev/scripts/ is the working assumption, and PR Add backport release commit helper script and improve hotfix workflow docs #19104 already put get_release_commit.sh there, which makes it the natural home.

2. Initial .backports.yml population — the automated bootstrap handles the archived flag, but someone needs to do a manual pass to fill in known maintained_until dates (e.g. 8.19 → 2027-01-15).

3. Packages to exclude — Currently, there is no package to exclude from this automation.

4. How to manage the Pull Requests that include more than one package — Show in the comment all available branches for each package updated in the Pull Request. Create as many backport Pull Requests as packages involved in the given Pull Request.

5. New validation for links set in changelog entries - There is going to be a new validation check introduced to avoid possible errors when updating changelog [CI] Add Buildkite step to validate changelog PR links #19171.

6. Avoid the usage of pull_request_target event in GitHub actions.

7. Auto-generated backport PR state — PRs are opened as ready for review, not as drafts. They should be reviewed before merging like any other PR, so opening them as drafts would just add an extra step with no benefit.

---

I'll create the required issues to track the implementation of each component @elastic/ecosystem