ci: support arn automation within release pipeline #2453

v1v · 2022-02-07T15:51:39Z

What does this PR do?

Enable automation for the AWS lambda publishing.
Create ARN table with the details in each region.
Add two parameters:
- whether to publish the AWS lambda, to support the old releases without lambda support.
- build number for the release job where the AWS zip file is stored.

Follow ups

This PR is the very first attempt to support the automated publishing. We will merge it and test and if it works then:
- re-enable the disabled steps
- test a full release
Upload the aws-lambda zip file in the GiHub release, so far disabled but we can revisit
Publish the ARN details somewhere else rather than in the GitHub releases

Issues

Superseedes #2452

Test

ghost · 2022-02-07T15:57:18Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-03-21T18:39:23.766+0000
Duration: 46 min 38 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2490
Skipped	16
Total	2506

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
run benchmark tests : Run the benchmark tests.
run jdk compatibility tests : Run the JDK Compatibility tests.
run integration tests : Run the Agent Integration tests.
run end-to-end tests : Run the APM-ITs.
run windows tests : Build & tests on windows.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

v1v · 2022-02-07T16:52:54Z

.ci/release/Jenkinsfile

+          }
+          steps {
+            setEnvVar('ELASTIC_LAYER_NAME', "elastic-apm-java${env.RELEASE_AWS_LAMBDA_VERSION}")
+            withAWSEnv(secret: 'secret/observability-team/ci/service-account/apm-aws-lambda', forceInstallation: true, version: '2.4.10') {


probably this requires the role_vault and so on....

:

v1v · 2022-02-10T12:46:07Z

.ci/release/Jenkinsfile

                env.RELEASE_ID = ret['id']
                env.RELEASE_NOTES_URL = finalUrl
              }
+              sh(label: 'make upload-lambda-asset', script: 'make -C .ci upload-lambda-asset')


this required to exclude if publish_aws_lambda is false

We don't need this for this iteration

v1v · 2022-02-10T12:54:19Z

.ci/release/Jenkinsfile

+                  fingerprintArtifacts: true,
+                  flatten: true,
+                  projectName: 'elastic+apm-agent-java+release',
+                  selector: lastSuccessful()


Enable the UI parameter to use in this selector.

v1v · 2022-02-10T12:54:47Z

.ci/release/Jenkinsfile

+                  selector: lastSuccessful()
+                )
+                sh(label: 'make publish-in-all-aws-regions', script: 'make -C .ci publish-in-all-aws-regions')
+                sh(label: 'make create-arn-file', script: 'make -C .ci create-arn-file')


Archive the file to see the look%feel

github-actions · 2022-02-10T18:32:19Z

/test

.ci/create-arn-table.sh

.ci/Makefile

Co-authored-by: Alexander Wert <AlexanderWert@users.noreply.github.com>

v1v · 2022-02-11T12:58:03Z

.ci/release/Jenkinsfile

+                // NOTE: copy the file from the last successful build. Therefore, this is a hard requirement
+                copyArtifacts(
+                  filter: "src/github.com/elastic/apm-agent-java/elastic-apm-agent/target/${SOURCE_AWS_FILE}",
+                  target: 'elastic-apm-agent/target',
+                  fingerprintArtifacts: true,
+                  flatten: true,
+                  projectName: 'elastic+apm-agent-java+release',
+                  selector: specific(params.release_build)
+                )


v1v · 2022-02-11T13:47:44Z

.ci/release/Jenkinsfile

+                dir ('elastic-apm-agent/target') {
+                  // TODO: copy file from a google bucket
+                  sh(label: 'fetch AWS lambda file', script: "wget https://github.com/elastic/apm-agent-java/releases/download/v${RELEASE_VERSION}/${SOURCE_AWS_FILE} -O ${SOURCE_AWS_FILE}")
+                }


This should be changed!

We agreed to manually provide the build number, but this can also work, no objection 🙂

We agreed to manually provide the build number

It does not work, unfortunately, the internal controller does not allow to copy artifacts between different jobs.

I think we are now forced to merge the release pipeline with the post-release one. I'll poke you in the internal issue that contains further detials

#2453 (comment)

eyalkoren · 2022-02-13T08:44:34Z

.ci/release/Jenkinsfile

+    RELEASE_AWS_LAMBDA_VERSION = '-ver-1-29-0'
+    RELEASE_VERSION = '1.29.0'


Reminder - don't forget to remove when restoring all release steps

eyalkoren · 2022-02-13T08:47:04Z

.ci/release/Jenkinsfile

+                dir ('elastic-apm-agent/target') {
+                  // TODO: copy file from a google bucket
+                  sh(label: 'fetch AWS lambda file', script: "wget https://github.com/elastic/apm-agent-java/releases/download/v${RELEASE_VERSION}/${SOURCE_AWS_FILE} -O ${SOURCE_AWS_FILE}")
+                }


We agreed to manually provide the build number, but this can also work, no objection 🙂

Co-authored-by: Alexander Wert <AlexanderWert@users.noreply.github.com>

cachedout · 2022-03-15T11:15:31Z

.ci/create-arn-table.sh

+	echo "|${f}|${LAYER_VERSION_ARN}|" >> "${ARN_FILE}"
+done
+
+echo '' >> "${ARN_FILE}"


A trap here so this cleanup always runs, perhaps?

.ci/release/Jenkinsfile

v1v · 2022-03-22T10:09:24Z

In order to test this we need to merge it, and these changes are not validated in the apm-ci as they are not executed as part of the main pipeline. I'll proceed and merge to be able to move forward with the release

v1v added 2 commits February 7, 2022 15:18

ci: support AWS ARN automation on release tag basis

0317992

ci: use the release pipeline instead

cac2dca

github-actions bot added the agent-java label Feb 7, 2022

ident fix

69020a9

v1v changed the title ~~Feature/support arn automation within release pipeline~~ ci: support arn automation within release pipeline Feb 7, 2022

ci: copy artifacts from the release job

1417ea0

v1v commented Feb 7, 2022

View reviewed changes

v1v added 3 commits February 7, 2022 16:54

remove unrequired entry in the changelog

31e2608

:

ci: ignore generated folder

66cf47c

use the right vault configuration

b38220d

v1v commented Feb 10, 2022

View reviewed changes

v1v added 5 commits February 10, 2022 18:15

ci: customise what build to be used

5ba1d7c

ci: upload assets won't be happening in the first iteration

b4c6891

ci: archive ARN table for debugging purposes

bb9e4f8

chore: disable other stages for testing the release for aws lambdas

a978a84

aws-publish: use release-version

39c392b

v1v marked this pull request as ready for review February 10, 2022 18:32

v1v requested review from a team February 10, 2022 18:32

v1v self-assigned this Feb 10, 2022

AlexanderWert reviewed Feb 11, 2022

View reviewed changes

.ci/create-arn-table.sh Outdated Show resolved Hide resolved

AlexanderWert reviewed Feb 11, 2022

View reviewed changes

.ci/Makefile Outdated Show resolved Hide resolved

Update .ci/Makefile

c364651

Co-authored-by: Alexander Wert <AlexanderWert@users.noreply.github.com>

v1v commented Feb 11, 2022

View reviewed changes

v1v added 2 commits February 11, 2022 13:11

ci: copy files between builds don't work

4c75ab0

make: fix dist target

5b63a64

make: use the right folder

179fa1f

v1v commented Feb 11, 2022

View reviewed changes

eyalkoren approved these changes Feb 13, 2022

View reviewed changes

Update .ci/create-arn-table.sh

8e238cf

Co-authored-by: Alexander Wert <AlexanderWert@users.noreply.github.com>

SylvainJuge added the await-release Mark issues that depend on next release, or PRs that are planned to be included label Feb 21, 2022

cachedout reviewed Mar 15, 2022

View reviewed changes

v1v commented Mar 22, 2022

View reviewed changes

Apply suggestions from code review

5f2c980

v1v merged commit e09b6a8 into elastic:main Mar 22, 2022

v1v deleted the feature/support-arn-automation-within-release-pipeline branch March 22, 2022 10:09

SylvainJuge removed the await-release Mark issues that depend on next release, or PRs that are planned to be included label May 5, 2022

		RELEASE_AWS_LAMBDA_VERSION = '-ver-1-29-0'
		RELEASE_VERSION = '1.29.0'

ci: support arn automation within release pipeline #2453

ci: support arn automation within release pipeline #2453

Uh oh!

Conversation

v1v commented Feb 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Follow ups

Issues

Test

Uh oh!

ghost commented Feb 7, 2022 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 10, 2022

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

v1v commented Mar 22, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

v1v commented Feb 7, 2022 •

edited

Loading

ghost commented Feb 7, 2022 •

edited by ghost

Loading