-
Notifications
You must be signed in to change notification settings - Fork 111
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement GitHub Actions based CI. #118
Conversation
Random notes as I look at the built binaries:
Apart from that, the resulting Ubuntu artefacts look good to me. |
The example you link to only has higan binaries, not byuu binaries, but the nightly has both? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This all looks pretty straightforward, even the status-code stuff.
I definitely want the executable bit set on Ubuntu and macOS binaries; nightlies are a whole lot less useful without it. The other things aren't a big deal - if they're difficult or tedious, don't worry about 'em.
Ah, that's because it was produced with an earlier version of the script. I can go remake that tag to produce a newer build, but the build process is the same for nightlies, so it doesn't make a huge difference. |
That's OK, as long as there's an explanation for why the tag is different from the nightly. |
It's 3AM and I may have missed something — but that last force-push fixes everything besides the "linux binaries lack the executable bit" thing, right? |
Sorry, I went AFK (and I'm now at work) but, the last thing I did added a small hack to try to restore the executable bit for the ubuntu binaries. So if that didn't work, then maybe something is wrong with my zip invocation. |
I extended the permissions hack to macOS binaries as well. Probably could be cleaner but this should be OK for now. |
Thanks for working on this! It'll be good to have GitHub releases for everything from now on. |
This is lightly adapted from the implementation in Higan. For more information, see that PR in higan-emu/higan#118. The main difference is that we only compile one binary here, and packaging is similar to (but adapted from) packaging byuu.
This is lightly adapted from the implementation in Higan. For more information, see that PR in higan-emu/higan#118. The main difference is that we only compile one binary here, and packaging is similar to (but adapted from) packaging byuu.
This is lightly adapted from the implementation in Higan. For more information, see that PR in higan-emu/higan#118. The main difference is that we only compile one binary here, and packaging is similar to (but adapted from) packaging byuu.
This is lightly adapted from the implementation in Higan. For more information, see that PR in higan-emu/higan#118. The main difference is that we only compile one binary here, and packaging is similar to (but adapted from) packaging byuu.
Summary
This pull request implements a continuous build process for building Higan.
Features
Design
The Matrix
The build matrix has two dimensions: the programs (higan-ui, byuu, icarus) and the OS (ubuntu, windows, and macos.) The builder always passes
build=performance
andlocal=false
to themake
invocation. Unlike Cirrus CI, we do not set theprofile
value at all, since it defaults to what is expected for each UI anyways.local=false
is needed because some workers arbitrarily have a newer Xeon that supports AVX512, which has two undesirable side-effects:In any case, this was already set by the Cirrus CI runner, and is idiomatically the correct thing to do, but those caveats are especially worth mentioning here.
Packaging
GitHub's artifacts mechanism is a little tricky at first. The first interesting behavior is that within a single workflow, artifacts with the same name are merged into one unit. The second interesting behavior is that we can download all artifacts at once, and each will appear in a subdirectory with the name of the artifact in the path we request (it defaults to
$GITHUB_WORKSPACE
which is, coincidentally, also the default$PWD
.)Packaging varies between
higan-ui
andbyuu
, but it doesn't actually vary by much between platforms. In fact, here the packaging code is actually identical across platforms, and so we just loop over each platform and do the same exact thing.Release Step
GitHub has some helper actions, written using their platform-agnostic JavaScript action runner, that can perform certain tasks for you, like create a new release. Unfortunately, we need to do some things that do not fit well within the capabilities available from existing actions. (I will detail exactly what those are in a moment.) In interest of broader maintainability, my initial code which did work in TypeScript has been ported to shell script and subsequently merged into the workflow itself.
Much of what is needed to make GitHub REST API calls across platforms is pretty consistent across different endpoints, so we define a function
github_rest
which takes an HTTP method (sometimes known as a verb) and a path, and then forwards remaining arguments tocurl
. Authentication and API versioning are handled withingithub_rest
to reduce duplication. We use cURL's--fail
mode to make server errors return errors, which, usingerrquit
, should propagate up to a build failure.Helper functions do actual GitHub API calls, like
github_get_release_id_for_tag
. We parse JSON usingjq
which is preinstalled on GitHub Action's Ubuntu runner. Since pipelines effectively bypasserrquit
, we capture the output into a variable and runjq
separately; it might be possible to do this in a less awful fashion. Some othererrquit
workaround shenanigans appear in the code as well; when in doubt, assume the weird thing I'm doing is working around subtle shell behavior.Finally, the release step passes the upload URL for artifacts into
UPLOAD_URL
using the GitHub Runner environment file mechanism, and this can then be used by subsequent stages to upload the artifacts.Nightly Release
GitHub Actions artifacts are useful, but they come with caveats.
We would ideally like to have a static URL for the latest build, and also, it would be ideal if said builds were available even to people without GitHub accounts. Therefore, it is ideal to have a 'rolling' nightly release that is updated on every push to
master
.To facilitate this, the bash code aforementioned deletes and recreates a release with the
nightly
tag on every build, and then new build artifacts are attached to that.This does have the downside of being very noisy for users who are trying to watch the repository using the "Releases Only" option, although it has little effect for ordinary watchers who would see pull request merges anyways.
Performance
Compared to Cirrus CI, builds are faster. There is also slightly more parallelism, as Icarus is compiled in a separate VM, although this is more a move to simplify the code than it is to improve performance.
Since GitHub Actions limits concurrency to 8 simultaneous jobs, one job will start stalled, but it will only take a minute or two for it to pick up. Typical runtimes look like around 17 minutes from start to finish. macOS jobs are the fastest, although also seem to be fairly variable.
Comparing to Cirrus CI
This setup has some disadvantages to the current Cirrus CI setup:
It has some advantages as well:
Above all, the primary advantage that seems to make it appealing is just the simplicity. Cirrus CI is fine, but it is another 'thing.' If it can be dropped, it is one less 'thing' to worry about.
Currently, I believe we should continue running both CIs simultaneously for a period of time.