Migrate to GitHub Actions and GitHub Container Registry#79
Merged
lemonsaurus merged 7 commits intoNov 18, 2020
Conversation
I've migrated the build pipeline to GitHub Actions and changed the container registry to GitHub Container Registry. In the process, I've made some changes to our docker setup and caching: - We are now using a single multi-stage Dockerfile Instead of three separate dockerfiles, we are now using a single multi-stage Dockerfile that can be used to build the three images we want using build targets. In part, this is because we're now using the docker buildx build action currently recommended by docker. This new engine runs in a sandboxed mode, meaning that while it can export built images to `docker` running in the host, it cannot import local images from it to base builds on. - Docker builds are now cached within GitHub Actions The builds are now cached using the GitHub Actions cache of the build cache directory. The cache keys try to match a cache generated by a build that matches the current build as closely as possible. In case of a cache miss, we fall back to caching from the latest image pushed to the container repository. - The `base` and `venv` images now have an inline cache manifest In order to fall back intelligently to caching from the repository, the final build and push action for the `base` and `venv` images includes an "inline" cache manifest. This means that the build process can inspect, without pulling, if it makes sense to pull layers to speed up the build. The other options, pushing a cache manifest separately (not inline), is currently not supported by GHCR. The custom caching script has been removed. - Linting errors are now added as GitHub Actions annotations Just like for some of our other pipelines, linting now generates annotations if linting errors are observed. - Coverage is pushed to coveralls.io A coverage summary is now pushed to coveralls.io. Each CI run will get a unique job that's linked in the CI output. If the run is attached to a PR, coveralls.io will automatically add a check link with the coverage result to the PR as well. - The README.md, Pipfile, docker-compose, and scripts have been updated As we now need to pull from and link to the GHCR, I've updated the other files to reflect these changes, including Pipfile run commands. I've also changed the CI badge and added a coveralls.io badge.
Now we've migrated to GitHub Actions, we don't need have XML reports of our unit tests as we're no longer using the Azure test result application.
SebastiaanZ
added
area: CI
SebastiaanZ
commented
Nov 18, 2020
jb3
approved these changes
Nov 18, 2020
I accidentally escaped a single quote in a run command; I've removed it now. I also changed the job name to `lint-test-build-push` to better reflect the contents of the job.
MarkKoz
requested changes
Nov 18, 2020
Contributor
MarkKoz
left a comment
MarkKoz
left a comment
There was a problem hiding this comment.
Since there's only one Dockerfile now, it's redundant to keep it within the Docker subdirectory.
One problem that our master builds may have is that they retain more and more layers of old builds, as there is no easy way of purging them from the cache. As such master cache would not have benefits over using repository-based caching, I've removed persistent local caching for non-PR builds.
I've removed the redundant intermediate image build commands from the Pipfile. Since everything is now contained in one Dockerfile, we can simply build the final image in one go.
MarkKoz
reviewed
Nov 18, 2020
MarkKoz
requested changes
Nov 18, 2020
I've fixed paths still pointing to the old Dockerfile location. I've also reverted an error that somehow got committed to the Dockerfile.
SebastiaanZ
force-pushed
the
sebastiaan/backend/migrate-ci-to-github-actions
branch
from
0ca42be to
8135eee
Compare
MarkKoz
approved these changes
Nov 18, 2020
lemonsaurus
deleted the
sebastiaan/backend/migrate-ci-to-github-actions
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I've migrated the build pipeline to GitHub Actions and changed the container registry to GitHub Container Registry. In the process, I've also made some changes to our docker setup and caching.
Note: This PR does not yet take Add pep8-naming and more pre-commit hooks #64 into account. I will update that PR later.
Note II: We need to get to a mergeable state sooner rather than later, as this is currently blocking our migration to the Kubernetes cluster.
Most notable aspects of this PR:
We are now using a single multi-stage Dockerfile
Instead of three separate dockerfiles, we are now using a single multi-stage Dockerfile that can be used to build the three images
we want using build targets.
In part, this is because we're now using the docker buildx build action currently recommended by docker. This new engine runs in a sandboxed mode, meaning that while it can export built images to
dockerrunning in the host, it cannot import local images from it to base builds on. Building truly separate images, from different dockerfiles, would mean we'd have to push and pull the intermediate images.By having a single multi-stage Dockerfile, we can still build the intermediate containers and cache their layers, but we don't have to reference other image tags in the Dockerfile.
This new builder back-end comes with some interesting additional options in terms of caching that should work similar to the custom caching script @MarkKoz wrote for Azure (see below).
Docker builds are now cached within GitHub Actions
The builds are now cached using the GitHub Actions cache of the build cache directory. The cache keys try to match a cache generated by a build that matches the current build as closely as possible. In case of a cache miss, we fall back to caching from the latest image pushed to the container repository.
The
baseandvenvimages now have an inline cache manifestIn order to fall back intelligently to caching from the repository, the final build and push action for the
baseandvenvimages now includes an "inline" cache manifest with the image. This means that the build process can inspect, without pulling the entire image, if it makes sense to pull layers to speed up the build.The other options, pushing a cache manifest separately (not inline), is currently not supported by GHCR.
The custom caching script has been removed.
Linting errors are now added as GitHub Actions annotations
Just like for some of our other pipelines, linting now generates annotations if linting errors are observed.
Coverage is pushed to coveralls.io
A coverage summary is now pushed to coveralls.io. Each CI run will get a unique job that's linked in the CI output. If the run is attached to a PR, coveralls.io will automatically add a check link with the coverage result to the PR as well.
The README.md, Pipfile, docker-compose, and scripts have been updated
As we now need to pull from and link to the GHCR, I've updated the other files to reflect these changes, including Pipfile run commands. I've also changed the CI badge and added a coveralls.io badge.