ci: support AWS lambda when a tag release is created #2585

v1v · 2022-02-23T19:38:44Z

What

Monitor the repository for a new release tag version (ex. v3.x.x)
When a tag is detected, a build process will run such that

A new GitHub release will be generated for the tag
A new layer will be published to AWS for every AWS region
The Layer ARN and version will be published in a table to the just created GitHub release
These should be publicly accessible layers

Notify when the AWS lambda is published

Tests

Create zip file

ELASTIC_LAYER_NAME=v1v-elastic-apm-node-ver-0-0-0 \
BRANCH_NAME=v3.29.0 \
SUFFIX_ARN_FILE=arn \
make -C .ci dist

produced:

rm -rf ../aws || true
mkdir -p ../aws
cd ../aws; \
		npm init -y; \
		npm install --global-style https://github.com/elastic/apm-agent-nodejs#v3.29.0; \
		mkdir nodejs; \
		mv node_modules nodejs
Wrote to /Users/vmartinez/work/src/github.com/elastic/apm-agent-nodejs/aws/package.json:

{
  "name": "aws",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}



added 157 packages, and audited 158 packages in 5s

36 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
rm -f v3.29.0.zip || true
cd ../aws ; \
		zip -r v3.29.0.zip nodejs
  adding: nodejs/ (stored 0%)
  adding: nodejs/node_modules/ (stored 0%)
  adding: nodejs/node_modules/.package-lock.json (deflated 74%)
  adding: nodejs/node_modules/elastic-apm-node/ (stored 0%)
  adding: nodejs/node_modules/elastic-apm-node/start.js (stored 0%)
  adding: nodejs/node_modules/elastic-apm-node/types/ (stored 0%)
  adding: nodejs/node_modules/elastic-apm-node/types/connect.d.ts (deflated 42%)

List regions:

AWS_ACCESS_KEY_ID=**** \                       
AWS_SECRET_ACCESS_KEY=**** \
ELASTIC_LAYER_NAME=v1v-elastic-apm-node-ver-0-0-0 \
BRANCH_NAME=v3.29.0 \
SUFFIX_ARN_FILE=arn \
make -C .ci get-all-aws-regions
cat .ci/.regions

produces:

af-south-1
eu-north-1
ap-south-1
eu-west-3

Publish

AWS_ACCESS_KEY_ID=**** \
AWS_SECRET_ACCESS_KEY=**** \
ELASTIC_LAYER_NAME=v1v-elastic-apm-node-ver-0-0-0 \
BRANCH_NAME=v3.29.0 \
SUFFIX_ARN_FILE=arn \
make -C .ci publish-in-all-aws-regions

produces:

publish 'v1v-elastic-apm-node-ver-0-0-0' in af-south-1
[debug] v1v-elastic-apm-node-ver-0-0-0 with version: 1
publish 'v1v-elastic-apm-node-ver-0-0-0' in eu-north-1
[debug] v1v-elastic-apm-node-ver-0-0-0 with version: 1
...

Create ARN markdown

ELASTIC_LAYER_NAME=v1v-elastic-apm-node-ver-0-0-0 \
BRANCH_NAME=v3.29.0 \
SUFFIX_ARN_FILE=arn \
make -C .ci create-arn-file

produces:

INFO: create-arn-table ARN(arn:aws:lambda:af-south-1:267093732750:layer:v1v-elastic-apm-node-ver-0-0-0:1):region(af-south-1))
INFO: create-arn-table ARN(arn:aws:lambda:ap-east-1:267093732750:layer:v1v-elastic-apm-node-ver-0-0-0:1):region(ap-east-1))
INFO: create-arn-table ARN(arn:aws:lambda:ap-northeast-1:267093732750:layer:v1v-elastic-apm-node-ver-0-0-0:1):region(ap-northeast-1))

Create Release Notes

ELASTIC_LAYER_NAME=v1v-elastic-apm-node-ver-0-0-0 \
BRANCH_NAME=v3.29.0 \
SUFFIX_ARN_FILE=arn \
make -C .ci release-notes

produces:

https://github.com/v1v/apm-agent-nodejs/releases/tag/untagged-5bcf4f0a0e61061a3f8f

ghost · 2022-02-23T20:00:23Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-03-01T19:33:48.772+0000
Duration: 21 min 33 sec

Test stats 🧪

Test	Results
Failed	0
Passed	241793
Skipped	0
Total	241793

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
run module tests for <modules> : Run TAV tests for one or more modules, where <modules> can be either a comma separated list of modules (e.g. memcached,redis) or the string literal ALL to test all modules
run benchmark tests : Run the benchmark test only.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

similar to https://github.com/elastic/apm-aws-lambda/blob/main/apm-lambda-extension/Makefile

v1v · 2022-02-28T16:35:13Z

Test

See https://github.com/elastic/apm-agent-nodejs/releases/tag/untagged-9e61fd3bcfbae26b66a5

v1v · 2022-02-28T16:38:43Z

.ci/Makefile

+		release \
+		create $(BRANCH_NAME) \
+		--title '$(BRANCH_NAME)' \
+		--generate-notes \


Probably we don't want this, otherwise

Suggested change

--generate-notes \

Agreed. I think it would be better to point to the published release notes, e.g. https://www.elastic.co/guide/en/apm/agent/nodejs/current/release-notes-3.x.html#release-notes-3.29.0, at the top of the release section, if possible.

https://github.com/elastic/apm-agent-nodejs/pull/2585/files#diff-bf80150258607c1e440ef8c2f2b828b033f7d11998cfd62e1f0716b9020acfbdR15 is the implementation and the URL is calculated on the fly based on the TAG version.

https://github.com/elastic/apm-agent-nodejs/releases/tag/untagged-67f14d7ca86fb8fa5e49

.ci/Jenkinsfile

.ci/Makefile

v1v · 2022-03-01T11:17:24Z

I just added a few more changes to:

consolidate the AWS_FOLDER in each Make goal.
Remove the generate notes in the Release Notes
Calculate the changelog

I tested it locally by just running the make goals and https://github.com/v1v/apm-agent-nodejs/releases/tag/v3.29.0 has been just created.

How does it look? If it's fine, I'd like to give a go with the test branch to verify everything goes smooth as expected, for such, I'll rebase the test branch with this PR and then the CI will do the rest. Please let me know your thoughts, as a consequence the ARN will be pointing to the version 2, I don't think that's problematic at all

astorm · 2022-03-01T15:54:53Z

.ci/Jenkinsfile

    OPBEANS_REPO = 'opbeans-node'
    GITHUB_CHECK = 'true'
+    RELEASE_URL_MESSAGE = "(<https://github.com/elastic/${env.REPO}/releases/tag/${env.TAG_NAME}|${env.TAG_NAME}>)"
+    SLACK_CHANNEL = '#apm-agent-nodejs'


Just a heads up -- for the test builds we didn't see a message sent to our slack channel.

That's right, I didn't enable notifications from the PullRequest but from Tags, I'm gonna proceed with a new release simulation that should also send notifications, if no issues from your side.

As stated in #2585 (comment), I'd like to give another go, therefore the new ARNs will be ending in 2.

(bear in mind, the existing null references are expected since it uses a specific env variable only available when running a build from a Git tag)

astorm · 2022-03-01T15:58:51Z

as a consequence the ARN will be pointing to the version 2, I don't think that's problematic at all

Agreed, not a problem at all.

How does it look? If it's fine, I'd like to give a go with the test branch to verify everything goes smooth as expected, for such, I'll rebase the test branch with this PR and then the CI will do the rest. Please let me know your thoughts,

Sounds good to me -- test away!

v1v · 2022-03-01T16:01:24Z

#2588 is now running again in the CI, (disabled all the stages but the AWS publishing) and enabled the slack/email notifications with the outcome for the AWS automation. Let's see how it goes in the CI

v1v · 2022-03-01T16:27:02Z

Status update

https://github.com/elastic/apm-agent-nodejs/releases/tag/untagged-67f14d7ca86fb8fa5e49 was generated (caused by #2588)

.ci/scripts/create-arn-table.sh

Co-authored-by: Trent Mick <trentm@gmail.com>

.ci/Jenkinsfile

.ci/Makefile

.gitignore

v1v · 2022-03-01T19:20:15Z

.ci/Makefile

+dist: validate-branch-name build
+	rm -f $(BRANCH_NAME).zip || true
+	cd $(AWS_FOLDER) ; \
+		zip -r $(BRANCH_NAME).zip nodejs


v1v · 2022-03-01T19:35:34Z

Status update

https://github.com/v1v/apm-agent-nodejs/releases/tag/v3.29.0 has been created by running the make goals locally.

@trentm , you can now move on with the code review 🙏

v1v added 2 commits February 23, 2022 19:35

ci: support AWS lambda when a tag release

4f7d913

we play safe in a subfolder

440a4a8

github-actions bot added the agent-nodejs Make available for APM Agents project planning. label Feb 23, 2022

v1v added 5 commits February 24, 2022 14:21

Use makefile to drive the AWS publish lambda

50c6a5b

similar to https://github.com/elastic/apm-aws-lambda/blob/main/apm-lambda-extension/Makefile

Use a no dot folder otherwise npm does not work

446b82d

support release notes and ARN

42a8188

remove the need for draft

b89c3b1

use -C instead

e66dd91

v1v mentioned this pull request Feb 28, 2022

test: aws lambda publishing #2588

Closed

v1v marked this pull request as ready for review February 28, 2022 16:35

v1v requested review from a team February 28, 2022 16:35

v1v self-assigned this Feb 28, 2022

v1v added the automation label Feb 28, 2022

v1v commented Feb 28, 2022

View reviewed changes

mdelapenya previously approved these changes Feb 28, 2022

View reviewed changes

trentm reviewed Feb 28, 2022

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

.ci/Makefile Outdated Show resolved Hide resolved

.ci/Makefile Show resolved Hide resolved

v1v added 9 commits March 1, 2022 09:29

publish: use top-level build-aws folder

a9b0dc3

don't use quotes

9ed5b38

ci: store the build-aws folder only

3536616

publish: use build-aws for all the files

9c09f42

publish: mkdir

6bfeeb6

publish: use ext to filter

4e5485c

publish: remove auto-generate release notes

7ee7170

publish: add release notes URL

edf6bff

chore: use e.g rather than f.i

24a7a48

v1v dismissed mdelapenya’s stale review via 24a7a48 March 1, 2022 11:17

astorm previously approved these changes Mar 1, 2022

View reviewed changes

v1v added 4 commits March 1, 2022 16:13

publish: fix the filename

91b554a

ci: use the right slack

c655ffc

ci: move post within the release meta-stage

4de1525

ci: report if a failure

c8f562e

v1v dismissed astorm’s stale review via c8f562e March 1, 2022 16:26

trentm reviewed Mar 1, 2022

View reviewed changes

.ci/scripts/create-arn-table.sh Outdated Show resolved Hide resolved

Update .ci/scripts/create-arn-table.sh

ea6d9cf

Co-authored-by: Trent Mick <trentm@gmail.com>

v1v commented Mar 1, 2022

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

.ci/Makefile Outdated Show resolved Hide resolved

.gitignore Outdated Show resolved Hide resolved

use build/aws folder

887962b

v1v commented Mar 1, 2022

View reviewed changes

release: f is a relative path

87ab753

trentm approved these changes Mar 2, 2022

View reviewed changes

v1v changed the title ~~ci: support AWS lambda when a tag release~~ ci: support AWS lambda when a tag release is created Mar 3, 2022

v1v merged commit 062f2e2 into elastic:main Mar 3, 2022

v1v deleted the feature/add-aws-lambda branch March 3, 2022 09:28

trentm mentioned this pull request Mar 10, 2022

3.30.0 #2607

Merged

trentm mentioned this pull request Mar 28, 2022

automate releases #2625

Closed

ci: support AWS lambda when a tag release is created #2585

ci: support AWS lambda when a tag release is created #2585

Uh oh!

Conversation

v1v commented Feb 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Tests

Create zip file

List regions:

Publish

Create ARN markdown

Create Release Notes

Uh oh!

ghost commented Feb 23, 2022 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

v1v commented Feb 28, 2022

Test

Uh oh!

v1v Feb 28, 2022

Choose a reason for hiding this comment

Uh oh!

trentm Feb 28, 2022

Choose a reason for hiding this comment

Uh oh!

v1v Mar 1, 2022

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

v1v commented Mar 1, 2022

Uh oh!

astorm Mar 1, 2022

Choose a reason for hiding this comment

Uh oh!

v1v Mar 1, 2022

Choose a reason for hiding this comment

Uh oh!

v1v Mar 1, 2022

Choose a reason for hiding this comment

Uh oh!

astorm commented Mar 1, 2022

Uh oh!

v1v commented Mar 1, 2022

Uh oh!

v1v commented Mar 1, 2022

Status update

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

v1v Mar 1, 2022

Choose a reason for hiding this comment

Uh oh!

v1v commented Mar 1, 2022

Status update

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

v1v commented Feb 23, 2022 •

edited

Loading

ghost commented Feb 23, 2022 •

edited by ghost

Loading