Automated Build for All Formats

[ike]

We should have an automated build of all formats, using the new Docker build system. Can we use GitHub actions for this?

c/f: #1294 (comment)

[elarlang]

Just in case mention, that it should work (and tested) with all translations.

[tghosth]

hey @ike,

I am working on a very basic Github action for this.

I have created PR #1679 but it does take quite a long time to run because of the time taken to build the container image.

[ike]

Hey @tghosth! That's what I found too -- I ended up pushing the image to docker hub and adjusting the run commands so it pulls before running. The action only takes ~2 mins with that change.

I am running into quite a few little problems, mostly related to the 4.0 docx files changing every time their built due to an internal time stamp.

I really like how you split things into multiple actions for 5.0/4.0 -- This solves quite a few odd behaviors I was experiencing.

I think we should create an ASVS repo on docker hub and publish the image there.

Feel free to borrow from any of this stuff in my branch: https://github.com/ike/asvs/tree/github-actions

[tghosth]

at this point I am less worried about the outputs being committed back into the repository itself. The current action just creates them as artefacts which can be downloaded from the repo:

e.g.
https://github.com/OWASP/ASVS/actions/runs/5475362014

[ike]

Okay, I created PR #1680 that uses GHCR for the image repository. We will need to create a package and associated secrets for the ASVS repo before merging.

[tghosth]

ok so would it make sense to create one action that creates and pushes the container and another action that generates the outputs? At that point, I don't see how the container is being created/published.

[ike]

@tghosth as it stands, the image is created and pushed manually. The action that builds the image could also push it to the repository. We would need to set up the correct access secrets for that.

[tghosth]

so can you update the PR to split it so one action creates and pushes the container and another action runs the output generation?

[ike]

@tghosth yep can do!

[ike]

Fixed with #1680

[tghosth]

Hi @ike, Can you check the logs, it looks like the image publish didn't succeed, maybe because it is going into ike instead of OWASP? Can you PR suggested changes to make it work?

[ike]

@tghosth yes, it looks like we'll need a couple of variables too. Create a personal access token, and add two variables to the settings of the OWASP/ASVS project:

• GHCR_TOKEN: should be set to the Personal Access Token
• GHCR_USERNAME: should be set to the username of the account that created the Personal Access Token

I am pretty sure that is all the settings changes this will need. When that's done I will change the code to push to the ASVS repo for the package registry. I think that'll work just fine.

[tghosth]

Thanks @ike I have now created those. I will merge the updated PR and see if it works.

[tghosth]

OK, the run failed:
https://github.com/OWASP/ASVS/actions/runs/5728422711/job/15522957437

Do I need an admin of the OWASP organization to create the package entity?

[ike]

@tghosth ugh -- I don't think so. Let me do a little more research on why this is failing.

[ike]

@tghosth okay, I have a new method up in #1704 -- we won't really be able to test without merging, unfortunately.

[ike]

Okay, looks like one permission needs to be amended on the ASVS repo:

from stack overflow
ensure that the Package settings (bottom right of the package page) allow access to actions running in the repository in question

[ike]

Also, if this merge works, we can remove the token from the repository variables.

[tghosth]

@ike sorry for the delay I have been on holiday. Please can you see whether the updated version now works?

[tghosth]

hi @ike how are we doing with this?

[ike]

@tghosth I have a new PR up with necessary changes to make this work. For now, it is manual only, since I don't want to create too much noise until we have tested it more thoroughly.

[tghosth]

Hi @ike,

Thanks for the new PR, I merged it but it seems to error on creating the container although the outputs seemed to create ok...
https://github.com/OWASP/ASVS/actions/runs/7626724616

[ike]

There is a problem with tlmgr mirrors, I think. Sometimes they randomly fail when tlmgr update --self is called. I am investigating whether there's a way to automatically re-try, or to pick another mirror.

Re-running the job built the image successfully: https://github.com/OWASP/ASVS/actions/runs/7626724616

[ike]

Created a #1830 new issue for the tlmgr update failure.