Implement GitHub Actions based CI. #118
Conversation
|
Random notes as I look at the built binaries:
Apart from that, the resulting Ubuntu artefacts look good to me. |
The example you link to only has higan binaries, not byuu binaries, but the nightly has both? |
There was a problem hiding this comment.
This all looks pretty straightforward, even the status-code stuff.
I definitely want the executable bit set on Ubuntu and macOS binaries; nightlies are a whole lot less useful without it. The other things aren't a big deal - if they're difficult or tedious, don't worry about 'em.
Ah, that's because it was produced with an earlier version of the script. I can go remake that tag to produce a newer build, but the build process is the same for nightlies, so it doesn't make a huge difference. |
That's OK, as long as there's an explanation for why the tag is different from the nightly. |
|
It's 3AM and I may have missed something — but that last force-push fixes everything besides the "linux binaries lack the executable bit" thing, right? |
Sorry, I went AFK (and I'm now at work) but, the last thing I did added a small hack to try to restore the executable bit for the ubuntu binaries. So if that didn't work, then maybe something is wrong with my zip invocation. |
|
OK, just took a quick look.
This should be good, although macOS may need a similar hack. Frustratingly, GitHub's own docs recommend just using a tarball artifact for passing data instead. I'll have to think more about this problem a bit more later on when I get back. |
|
I extended the permissions hack to macOS binaries as well. Probably could be cleaner but this should be OK for now. |
|
Thanks for working on this! It'll be good to have GitHub releases for everything from now on. |
This is lightly adapted from the implementation in Higan. For more information, see that PR in higan-emu/higan#118. The main difference is that we only compile one binary here, and packaging is similar to (but adapted from) packaging byuu.
This is lightly adapted from the implementation in Higan. For more information, see that PR in higan-emu/higan#118. The main difference is that we only compile one binary here, and packaging is similar to (but adapted from) packaging byuu.
This is lightly adapted from the implementation in Higan. For more information, see that PR in higan-emu/higan#118. The main difference is that we only compile one binary here, and packaging is similar to (but adapted from) packaging byuu.
This is lightly adapted from the implementation in Higan. For more information, see that PR in higan-emu/higan#118. The main difference is that we only compile one binary here, and packaging is similar to (but adapted from) packaging byuu.
Summary
This pull request implements a continuous build process for building Higan.
Features
Design
The Matrix
The build matrix has two dimensions: the programs (higan-ui, byuu, icarus) and the OS (ubuntu, windows, and macos.) The builder always passes
build=performanceandlocal=falseto themakeinvocation. Unlike Cirrus CI, we do not set theprofilevalue at all, since it defaults to what is expected for each UI anyways.local=falseis needed because some workers arbitrarily have a newer Xeon that supports AVX512, which has two undesirable side-effects:In any case, this was already set by the Cirrus CI runner, and is idiomatically the correct thing to do, but those caveats are especially worth mentioning here.
Packaging
GitHub's artifacts mechanism is a little tricky at first. The first interesting behavior is that within a single workflow, artifacts with the same name are merged into one unit. The second interesting behavior is that we can download all artifacts at once, and each will appear in a subdirectory with the name of the artifact in the path we request (it defaults to
$GITHUB_WORKSPACEwhich is, coincidentally, also the default$PWD.)Packaging varies between
higan-uiandbyuu, but it doesn't actually vary by much between platforms. In fact, here the packaging code is actually identical across platforms, and so we just loop over each platform and do the same exact thing.Release Step
GitHub has some helper actions, written using their platform-agnostic JavaScript action runner, that can perform certain tasks for you, like create a new release. Unfortunately, we need to do some things that do not fit well within the capabilities available from existing actions. (I will detail exactly what those are in a moment.) In interest of broader maintainability, my initial code which did work in TypeScript has been ported to shell script and subsequently merged into the workflow itself.
Much of what is needed to make GitHub REST API calls across platforms is pretty consistent across different endpoints, so we define a function
github_restwhich takes an HTTP method (sometimes known as a verb) and a path, and then forwards remaining arguments tocurl. Authentication and API versioning are handled withingithub_restto reduce duplication. We use cURL's--failmode to make server errors return errors, which, usingerrquit, should propagate up to a build failure.Helper functions do actual GitHub API calls, like
github_get_release_id_for_tag. We parse JSON usingjqwhich is preinstalled on GitHub Action's Ubuntu runner. Since pipelines effectively bypasserrquit, we capture the output into a variable and runjqseparately; it might be possible to do this in a less awful fashion. Some othererrquitworkaround shenanigans appear in the code as well; when in doubt, assume the weird thing I'm doing is working around subtle shell behavior.Finally, the release step passes the upload URL for artifacts into
UPLOAD_URLusing the GitHub Runner environment file mechanism, and this can then be used by subsequent stages to upload the artifacts.Nightly Release
GitHub Actions artifacts are useful, but they come with caveats.
We would ideally like to have a static URL for the latest build, and also, it would be ideal if said builds were available even to people without GitHub accounts. Therefore, it is ideal to have a 'rolling' nightly release that is updated on every push to
master.To facilitate this, the bash code aforementioned deletes and recreates a release with the
nightlytag on every build, and then new build artifacts are attached to that.This does have the downside of being very noisy for users who are trying to watch the repository using the "Releases Only" option, although it has little effect for ordinary watchers who would see pull request merges anyways.
Performance
Compared to Cirrus CI, builds are faster. There is also slightly more parallelism, as Icarus is compiled in a separate VM, although this is more a move to simplify the code than it is to improve performance.
Since GitHub Actions limits concurrency to 8 simultaneous jobs, one job will start stalled, but it will only take a minute or two for it to pick up. Typical runtimes look like around 17 minutes from start to finish. macOS jobs are the fastest, although also seem to be fairly variable.
Comparing to Cirrus CI
This setup has some disadvantages to the current Cirrus CI setup:
It has some advantages as well:
Above all, the primary advantage that seems to make it appealing is just the simplicity. Cirrus CI is fine, but it is another 'thing.' If it can be dropped, it is one less 'thing' to worry about.
Currently, I believe we should continue running both CIs simultaneously for a period of time.