Reduce GitLab CI pipeline depth #577
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Source prep is hitting that btrfs dev problem I've seen mentioned somewhere else... |
thozza
reviewed
Apr 11, 2024
|
I would like to add one suggestion: It would be great to not run Gitlab CI at all if |
This is a good suggestion. I wanted to get away from what we do in osbuild-composer which is gate gitlab on all the unit tests, which can take some time. But the source-prep stage, especially now that it's generating the CI config, should be in the same workflow as the GitLab trigger now. I'll move some stuff around. |
We will use this script to generate the whole CI config.
Add the init, verify, and finish jobs to the base config, so that we can generate the top-level gitlab-ci config with all the stages we need.
Sort the image configurations so that the generator is stable.
Check in the generated gitlab-ci.yml so it becomes the top-level CI config.
Generate .gitlab-ci.yml when running prepare-source. This will make sure we catch any changes to image configurations (distros, arches, image types) that aren't reflected in the CI config.
The source prep script now runs a python script that import imgtestlib, which uses newer python3 features (match). Run on ubuntu-latest so we have a newer python version and update all runners to match.
achilleas-k
force-pushed
the
ci/pop-a-level
branch
from
9408d05
to
bf66b67
Compare
|
Done and done. |
The source preparation script now generates the gitalb-ci.yml and checks that it's up to date with new image configurations. Don't run gitlab CI tests if the source prep check doesn't succeed since it might end up testing the wrong thing (distros that have been removed, for example). Also, we need build dependencies to run the script now since it compiles and runs the list-images command. Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
achilleas-k
force-pushed
the
ci/pop-a-level
branch
from
bf66b67
to
ec3de25
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, our top-level gitlab-ci.yml included a call to
./test/scripts/configure-generatorswhich was used to generate a dynamic pipeline with jobs that would generate manifests for each distro/arch/image-type configuration. Then, each of those manifest generation stages in the dynamic pipeline would generate another level of dynamic pipelines based on the manifest changes.This PR reduces the depth of dynamic pipelines by 1 (to 1). The gitlab-ci.yml in the repository is replaced by the configuration that was previously dynamically generated at the first level, with the other previous top-level jobs added (
init,verify,finish). A CI job makes sure that the generated config is up to date.This change has two benefits:
configure-generatorsstep would take 3-4 minutes and would run every time. Now it's done offline and a local run takes a few seconds. The runtime of the job in CI was mostly spent setting up the runner.Includes #576