Beats DRA pipeline

[pazone]

Proposed commit message

Beats Packaging pipeline

Migrates packaging.groovy to buildkite.
Buildkite packaging pipeline is used to build and publish the packages for the Beats. The pipeline is triggered by a commit to the main or release branches.
The pipeline definition is located in the .buildkite/packaging.pipeline.yml

Triggers

Staging packaging DRA is triggered for the main and release branches.
Snapshot can be triggered for any branch by the /package comment in the PR. The release-manager dry-run will be used for PR builds.

Pipeline steps

Beats dashboards

Generates build/distributions/dependencies.csv and adds it to the beats-dashboards artifact. The dependencies.csv is required by the release-manager configuration

Packaging snapshot/staging

• Builds the Beats packages for all supported platforms and architectures (mage package, mage ironbank)
• Copies artifacts build/distributions/<beat>/ directory and adds it as an artifact. Where <beat> is the name of the beat
• x-pack artifacts a also copied to build/distributions/<beat>/ directory, where <beat> is the name of the beat. For example, auditbeat, not x-pack/auditbeat

DRA publish

Downloads the artifacts from the packaging snapshot/staging step and publishes them to the Elastic DRA registry.

Snapshot publishing example (Dry run)

Beats-tester and e2e tests are not triggered

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• All Snapshot and staging artifacts are built, present, and properly transferred to the corresponding DRA steps
• DRA dry run passes
• All artifacts have the same size and permissions as on Jenkins. (Need to check one more time)

How to test this PR locally

Related issues

• https://github.com/elastic/ingest-dev/issues/3072
• https://github.com/elastic/ingest-dev/issues/3095

Use cases

Screenshots

Logs

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @pazone? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Duration: 14 min 28 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[dliappis]

I recommend we change it to

if [[ "$DRY_RUN" == "false" ]]; then echo "--- Running in publish mode"; DRY_RUN=""; else echo "--- Running in dry-run mode"; DRY_RUN="--dry-run"; fi

to facilitate easy testing.

IMPORTANT because DRY_RUN maybe unset, we should also move set -euo pipefail from line 2 right after the suggested conditional.

[dliappis]

LGTM!