Publish precompiled to s3 on releasing #12242

gilacost · 2022-11-15T19:01:33Z

Uploads Precompiled.zip to S3 as ~~elixir.zip~~ elixir-otp-${{ matrix.otp }}.zip
Updates build_txt and uploads it to S3
~~Uploads hexdocs to S3 for each application~~ purges fastly
closes Automate docs publishing to Hexdocs with GitHub Actions #12038

~~## Next steps~~
* we can create a new composite action that uploads HexDocs to s3
* use this action in release.yml and repo.hex.pm.yml

Co-authored-by: wingyplus <wingyminus@gmail.com>

josevalim · 2022-11-15T19:07:12Z

.github/workflows/release.yml

          gh release upload --clobber "${{ github.ref_name }}" \
            Docs.zip \
            Docs.zip.sha{1,256}sum
+      - name: Upload Precompiled to S3


I am thinking we should keep those separate:

We can move this to the repo.hex.pm task

We can change repo.hex.pm to run on tags too

WDYT?

The idea is to keep this only specific to GitHub and the other one to Hex.

* moves Precompiled.zip to elixir.zip * uploads elixir.zip to S3 * uploads build_txt to S3

josevalim · 2022-11-15T19:15:10Z

.github/workflows/repo.hex.pm.yml

+          sed -i "/^${ref_name}${{ matrix.otp }} /d" builds.txt
+          echo -e "${ref_name}${{ matrix.otp }} ${{ github.sha }} ${date} ${build_sha256} \n$(cat builds.txt)" > builds.txt
+          sort -u -k1,1 -o builds.txt builds.txt
+          aws s3 cp builds.txt s3://${{ env.AWS_S3_BUCKET }}/builds/elixir/builds.txt --cache-control "public,max-age=3600" --metadata '{"surrogate-key":"builds","surrogate-control":"public,max-age=604800"}'


Shouldn't we also purge or is that not necessary?

I don't think so, at least I can't see it in upload_builds_txt.

We need to purge, it's happening here: https://github.com/hexpm/bob/blob/ab6e9db2df0ed0abcf1b5c81178d2c83237b267c/priv/scripts/elixir/elixir.sh#L43.

4ac9cd6 @ericmj 🚀

josevalim · 2022-11-15T19:32:46Z

/cc @ericmj @wojtekmach

ericmj

Just left one comment about the purging, otherwise it looks great. Thank you @gilacost!

.github/workflows/repo.hex.pm.yml

gilacost · 2022-11-17T06:33:33Z

Just left one comment about the purging, otherwise it looks great. Thank you @gilacost!

😍

Co-authored-by: Eric Meadows-Jönsson <eric.meadows.jonsson@gmail.com>

…easing

josevalim · 2022-11-17T17:53:06Z

.github/workflows/repo.hex.pm.yml

+          AWS_REGION: ${{ secrets.HEX_AWS_REGION }}
+          AWS_S3_BUCKET: ${{ secrets.HEX_AWS_S3_BUCKET }}
+        run: |
+          mv elixir-otp-${{ matrix.otp }}.zip elixir.zip


I don't think we need this mv right? We can directly aws s3 cp the first one?

it might not be necessary but as I can not list the items in the bucket I limited myself to copy and paste from bob.

.github/workflows/repo.hex.pm.yml

Co-authored-by: José Valim <jose.valim@gmail.com>

.github/workflows/repo.hex.pm.yml

Co-authored-by: José Valim <jose.valim@gmail.com>

gilacost · 2022-11-18T07:05:25Z

I am trying to understand this with no luck 😅. Could you bring some light here @ericmj 🙏

ericmj · 2022-11-18T08:49:59Z

I am trying to understand this with no luck 😅. Could you bring some light here @ericmj 🙏

I updated the code after your comment, maybe it makes more sense now. For the build on the oldest OTP version we also upload a version without the -otp- suffix and add it to the builds.txt file.

ericmj · 2022-11-19T00:12:32Z

.github/workflows/repo.hex.pm.yml

+            purge_key $1
+          }
+          EOF
+          chmod +x utils.sh


Nice solution to the code duplication.

gilacost · 2022-11-19T07:52:49Z

Is there anything else that we need to do to get this merged? 😄 Please let me know if I can help in any way 😛

josevalim · 2022-11-19T08:39:42Z

💚 💙 💜 💛 ❤️

Josep Lluis Giralt D'Lacoste and others added 3 commits November 15, 2022 20:50

Uploads Precompiled.zip to S3 as elixir.zip

3f63f78

Co-authored-by: wingyplus <wingyminus@gmail.com>

Updates build_txt and uploads it to S3

5e51688

Co-authored-by: wingyplus <wingyminus@gmail.com>

Uploads hexdocs to S3 for each application

8707d4d

Co-authored-by: wingyplus <wingyminus@gmail.com>

gilacost changed the title ~~publish precompiled to s3 on releasing~~ Publish precompiled to s3 on releasing Nov 15, 2022

gilacost mentioned this pull request Nov 15, 2022

Automate docs publishing to Hexdocs with GitHub Actions #12038

Closed

josevalim reviewed Nov 15, 2022

View reviewed changes

Move steps to repo.hex.pm.yml

3a18d82

* moves Precompiled.zip to elixir.zip * uploads elixir.zip to S3 * uploads build_txt to S3

gilacost force-pushed the publish-precompiled-to-s3-on-releasing branch from 6112488 to 3a18d82 Compare November 15, 2022 19:14

josevalim reviewed Nov 15, 2022

View reviewed changes

PR feedback, builds key needs to be purged

4ac9cd6

ericmj approved these changes Nov 16, 2022

View reviewed changes

.github/workflows/repo.hex.pm.yml Outdated Show resolved Hide resolved

ericmj reviewed Nov 16, 2022

View reviewed changes

.github/workflows/repo.hex.pm.yml Show resolved Hide resolved

gilacost and others added 3 commits November 17, 2022 08:34

Apply suggestions from code review

cdd2b11

Co-authored-by: Eric Meadows-Jönsson <eric.meadows.jonsson@gmail.com>

Merge branch 'elixir-lang:main' into publish-precompiled-to-s3-on-rel…

d8c4a7f

…easing

Move the purge dance to utils.sh

8c4ceb9

josevalim reviewed Nov 17, 2022

View reviewed changes

.github/workflows/repo.hex.pm.yml Outdated Show resolved Hide resolved

gilacost and others added 2 commits November 17, 2022 21:32

Update .github/workflows/repo.hex.pm.yml

82b8ad4

Co-authored-by: José Valim <jose.valim@gmail.com>

PR feedback, remove unneeded 'mv'

6e75af9

josevalim reviewed Nov 17, 2022

View reviewed changes

.github/workflows/repo.hex.pm.yml Outdated Show resolved Hide resolved

gilacost and others added 4 commits November 17, 2022 21:44

Update .github/workflows/repo.hex.pm.yml

266196c

Co-authored-by: José Valim <jose.valim@gmail.com>

Pending from removing mv elixir.zip

2cdb766

Fix filenames in builds.txt

88bc2d5

Upload generic version

12ebb9c

gilacost requested review from ericmj and josevalim and removed request for ericmj November 18, 2022 07:09

gilacost requested review from ericmj and removed request for josevalim November 18, 2022 07:09

Remove -otp- from generic version

5e45277

ericmj approved these changes Nov 19, 2022

View reviewed changes

gilacost requested a review from josevalim November 19, 2022 07:54

josevalim merged commit b9bcfae into elixir-lang:main Nov 19, 2022

Publish precompiled to s3 on releasing #12242

Publish precompiled to s3 on releasing #12242

Uh oh!

Conversation

gilacost commented Nov 15, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josevalim commented Nov 15, 2022

Uh oh!

ericmj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gilacost commented Nov 17, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gilacost commented Nov 18, 2022

Uh oh!

ericmj commented Nov 18, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gilacost commented Nov 19, 2022

Uh oh!

josevalim commented Nov 19, 2022

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

3 participants

gilacost commented Nov 15, 2022 •

edited

Loading