Add Windows and ARM support to prerelease pipeline (NR-52415) #147

kang-makes · 2023-01-03T08:20:51Z

There are lots of things that I had to tweak to make this pipeline work.

It has been tested in a fork repo and the pipeline worked up to upload everything to the S3 bucket.

According to the CAOS team, that action test that the repo signing is correct, and the pipeline cannot test it as the fork had a fake GPG key made for this PR.

I am creating this PR without that part working correctly as it is hard to test in the repo as it requires the secrets for package signing in order to test the repo signing.

Further effort to make this work is still needed at this point.

sigilioso

Thanks for all the effort!
I simply left a couple of nits.

.github/workflows/prerelease.yml

scripts/pkg/.goreleaser.yml.template

sigilioso · 2023-01-04T09:20:50Z

The pre-release action was tested using setting up the workflow file to trigger the workflow when we push on this branch and using s3 testing environment (check details in 9aa18c4).

The pre-release was created and the assets where sent to testing s3, as it shown here: https://github.com/newrelic/newrelic-prometheus-exporters-packages/actions/runs/3836093769

The installability test failled because it checks if the version corresponding to the tag was installed. However, that latests version is not in the staging s3 because the testing one was used instead.

Both the complete pre-release workflow and the release one (which was modified to include the s3-publish-schema generation) will be completely tested when we release one of the integrations, in a follow-up PR.

kang-makes

I am not approving this PR as I was the one who worked on it for a couple of weeks, but I am leaving comments that I can see now with a fresh view.

Thanks for taking care!

scripts/package.sh

.github/workflows/prerelease.yml

.github/workflows/release.yml

scripts/package.sh

.github/workflows/prerelease.yml

alvarocabanas

LGTM I just left a small NIT

.github/workflows/prerelease.yml

paologallinaharbur · 2023-01-04T14:20:05Z

@sigilioso Since we have all the context now, can you check if we are already addressing #104 here?

.github/workflows/prerelease.yml

sigilioso · 2023-01-04T14:53:11Z

@sigilioso Since we have all the context now, can you check if we are already addressing #104 here?

It was not completely addressed, but after 8fa0e82 it is.

.github/workflows/prerelease.yml

paologallinaharbur · 2023-01-04T15:54:34Z

.github/workflows/prerelease.yml

        with:
-          go-version: ${{env.GO_VERSION}}
+          version: v1.13.1


should we replace this as well?

Sure, I missed that one and also missed this comment before merging, I'll replace it in a follow-up PR. Thanks!

paologallinaharbur · 2023-01-04T15:56:45Z

.github/workflows/prerelease.yml

+    name: Create the publish schema
+    runs-on: ubuntu-latest
+    if: ${{ needs.check_exporter_preconditions.outputs.CREATE_RELEASE == 'true' }}
+    needs: [ check_exporter_preconditions ]
+    steps:
+      - uses: actions/checkout@v3
+      - run: make create-publish-schema-${{ needs.check_exporter_preconditions.outputs.INTEGRATION_NAME }}
+      - uses: actions/upload-artifact@v3
        with:
-          upload_url: ${{ needs.create_prerelease.outputs.UPLOAD_URL }}
-          asset_path: ${{ github.workspace }}\src\repo\exporters\${{ steps.vars.outputs.NAME }}\target\packages\${{ steps.vars.outputs.PACKAGE_NAME }}-${{ matrix.goarch }}.${{ steps.vars.outputs.VERSION }}.msi
-          asset_name: ${{ steps.vars.outputs.PACKAGE_NAME }}-${{ matrix.goarch }}.${{ steps.vars.outputs.VERSION }}.msi
-          asset_content_type: msi package
+          name: publish-schema
+          retention-days: 1
+          path: |
+            scripts/pkg/s3-publish-schema-tmp.yml


why creating the publish schema in a separate job, uploading it and then downloading it?
isn't easier just to call later on directly:

make create-publish-schema-${{ needs.check_exporter_preconditions.outputs.INTEGRATION_NAME }}

?

Another long story...

The publish action did not support local-custom when I worked in the pipeline. In fact, I had to create the feature: newrelic/infrastructure-publish-action#142

The first iteration was to upload the artifact using the action and use the API to get the URL of the artifact and use it with the publish action. A nasty quick and dirty hack but that should do the trick.

I learned that we cannot do that. The artifact that you upload are not immediately accessible via API. If you list the artifacts before and after uploading and artifact using the action there are no changes. Artifacts are only available AFTER the workflow has finished.

Link to the docs but I'll save you a click:

Downloading or deleting artifacts

During a workflow run, you can use the download-artifact action to download artifacts that were previously uploaded in the same workflow run.
After a workflow run has been completed, you can download or delete artifacts on GitHub or using the REST API.

Note that "After a workflow run has been completed".

How does the action to download fetch the artifacts? After reverse engineer the action calls a URL like ${config_variables_1.getRuntimeUrl()}_apis/pipelines/workflows/${config_variables_1.getWorkFlowRunId()}/artifacts?api-version=${getApiVersion()}. The function getRuntimeUrl() simply looks for the environment variable ACTIONS_RUNTIME_URL.

The environment variable ACTIONS_RUNTIME_URL is only available INSIDE an action but not in a step that runs a script. If I run a step with set that variable does not exist. These internal environment variables are undocumented: nektos/act#329

Someone created an action that dumps the environment variables from inside an action and you can see that environment variable there: https://gist.github.com/sonots/070f1059b71dd2dec8e43ec513c3f8f0

Then you'll ask, why did it survive after this failure?

It was really useful to have it uploaded for debugging purposes as you can fetch it and see how it templates just in case the yq that you have locally is not the same yq that GitHub is using (yes, there are two yq called yq in Python and Go).

It parallelizes (a bit) the jobs and removes the need to check out the code (again) in the last job when the artifacts are uploaded making that job half a minute shorter.

paologallinaharbur

thanks for taking care of it!

…rter file

kang-makes requested a review from a team as a code owner January 3, 2023 08:20

sigilioso reviewed Jan 3, 2023

View reviewed changes

.github/workflows/prerelease.yml Outdated Show resolved Hide resolved

.github/workflows/prerelease.yml Outdated Show resolved Hide resolved

scripts/pkg/.goreleaser.yml.template Outdated Show resolved Hide resolved

sigilioso requested review from paologallinaharbur and a team January 4, 2023 09:21

kang-makes commented Jan 4, 2023

View reviewed changes

paologallinaharbur reviewed Jan 4, 2023

View reviewed changes

.github/workflows/prerelease.yml Show resolved Hide resolved

alvarocabanas approved these changes Jan 4, 2023

View reviewed changes

.github/workflows/prerelease.yml Show resolved Hide resolved

.github/workflows/prerelease.yml Outdated Show resolved Hide resolved

paologallinaharbur reviewed Jan 4, 2023

View reviewed changes

.github/workflows/prerelease.yml Show resolved Hide resolved

paologallinaharbur reviewed Jan 4, 2023

View reviewed changes

.github/workflows/prerelease.yml Outdated Show resolved Hide resolved

paologallinaharbur mentioned this pull request Jan 4, 2023

add access_point_host for publish action #104

Closed

paologallinaharbur reviewed Jan 4, 2023

View reviewed changes

paologallinaharbur approved these changes Jan 4, 2023

View reviewed changes

kang-makes added 14 commits January 4, 2023 17:15

first iteration of the change for the release pipeline

1cd4dc3

add goreleaser to the pre-release pipeline

dd3e34f

Remove e2e correctly

4f74a07

fix prerelease pipeline

7a3280e

Fix goreleaser conditionals

0fbbe58

remove signing

210ac61

fix package naming

0c5e1fa

Fix naming and path expansion

6acfc39

fix packaging

061151d

fix version templating at build time

56917d9

remove snapshot from goreleaser

baea197

Fix packaging version

90e812d

The release was done in the previous step so now is false

096b682

Fix tag naming in prerelase pipeline

e6477e3

kang-makes and others added 27 commits January 4, 2023 17:16

Test the last endpoint

4bf5869

test package upload

ea0854f

where are my artifacts

6b77c80

Create release before publishing artifacts

f2c1884

fix release tag name

777cfe4

fix cpu arch on red hat based distros

f64878c

where are my artifacts (again)

66b3ed5

fix archiving paths and create Windows ZIPs

6852a3a

where is zip

53786ae

switch to 7z

c30d5d5

Uncomment all step made for testing and to speed up the pipeline

d9b9ea8

removed a line of wix project by mistake

5a179db

rename jobs and steps

6a0061a

test publish to s3 using testing environment

eeb349d

use proper path for linux artifacts in the manual generation action

9fab61d

sign packages after copying them to the expected location

b54c7a3

cleanup pre-release pipeline workflow file and restore powerdens expo…

2dd99d5

…rter file

add sles 15.4 to s3 publish schema

e64109e

update release workflow to create s3 schema before sending assets

094a921

include additional comments in package.sh

85482ee

improve package script comments

3eb8533

fix release schema_path value

6693a88

set goreleaser version as env variable

dbda3a8

move mail variable to the top

fb166fc

add access_point_host for publish action

b799537

cleanup prerelease jobs

b9d7daf

remove outdated comments

052bcaf

sigilioso force-pushed the pipeline/add-windows-arm-support-prerelease branch from 696b499 to 052bcaf Compare January 4, 2023 16:18

sigilioso merged commit 09d0e79 into main Jan 4, 2023

sigilioso deleted the pipeline/add-windows-arm-support-prerelease branch January 4, 2023 16:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Windows and ARM support to prerelease pipeline (NR-52415) #147

Add Windows and ARM support to prerelease pipeline (NR-52415) #147

kang-makes commented Jan 3, 2023 •

edited

Loading

sigilioso left a comment

sigilioso commented Jan 4, 2023 •

edited

Loading

kang-makes left a comment

alvarocabanas left a comment

paologallinaharbur commented Jan 4, 2023

sigilioso commented Jan 4, 2023

paologallinaharbur Jan 4, 2023

sigilioso Jan 5, 2023

paologallinaharbur Jan 4, 2023 •

edited

Loading

kang-makes Jan 4, 2023

paologallinaharbur left a comment

Add Windows and ARM support to prerelease pipeline (NR-52415) #147

Add Windows and ARM support to prerelease pipeline (NR-52415) #147

Conversation

kang-makes commented Jan 3, 2023 • edited Loading

sigilioso left a comment

Choose a reason for hiding this comment

sigilioso commented Jan 4, 2023 • edited Loading

kang-makes left a comment

Choose a reason for hiding this comment

alvarocabanas left a comment

Choose a reason for hiding this comment

paologallinaharbur commented Jan 4, 2023

sigilioso commented Jan 4, 2023

paologallinaharbur Jan 4, 2023

Choose a reason for hiding this comment

sigilioso Jan 5, 2023

Choose a reason for hiding this comment

paologallinaharbur Jan 4, 2023 • edited Loading

Choose a reason for hiding this comment

kang-makes Jan 4, 2023

Choose a reason for hiding this comment

Another long story...

Downloading or deleting artifacts

paologallinaharbur left a comment

Choose a reason for hiding this comment

kang-makes commented Jan 3, 2023 •

edited

Loading

sigilioso commented Jan 4, 2023 •

edited

Loading

paologallinaharbur Jan 4, 2023 •

edited

Loading