Speed up CI #5592
Merged
Speed up CI #5592
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We need to explicitly install python2, since it is no longer the default. Eventually we may want to drop this completely, but we're not there yet. Signed-off-by: Edwin Török <edwin.torok@cloud.com>
We do not publish or use the opam packages from this repo, and therefore pinning is unnecessary. 'opam install --deps-only --with-test .' works without pinning on the version of 'opam' we have here. Pinning ~71 packages is very slow since it makes a full copy of the source+.git folder every time. Skip pinning, this also saves a few gigabytes of disk space. This also makes the cleanup-xapi-environment action a no-op, so remove it. (in fact it'd fail because the output of opam list with master pins is empty). For testing the correctness of opam packages: * we still have the xs-opam CI * we can introduce opam-dune-lint that statically checks opam and dune for missing dependencies (needs new release first) This reduces CI times: * Run OCaml tests: from 15m24s to 10m16s * Setup XenAPI environment: from 7m1s to 5m55s Signed-off-by: Edwin Török <edwin.torok@cloud.com>
Setting DUNE_CACHE_STORAGE_MODE=copy is needed during 'opam install', otherwise dune builds inside opam packages won't be cached. The sandbox used by opam is another partition/device and hardlinking fails with EXDEV. Signed-off-by: Edwin Török <edwin.torok@cloud.com>
Now that we no longer pin the local opam packages we have a few more GiB of disk space, and we don't need to rm pre-existing files. This speeds up this step a little. Signed-off-by: Edwin Török <edwin.torok@cloud.com>
Codecov ReportAll modified and coverable lines are covered by tests ✅ Additional details and impacted files@@ Coverage Diff @@
## master #5592 +/- ##
========================================
- Coverage 53.6% 44.7% -8.9%
========================================
Files 16 16
Lines 2000 2206 +206
========================================
- Hits 1072 988 -84
- Misses 928 1218 +290 see 17 files with indirect coverage changes
Flags with carried forward coverage won't be shown. Click here to find out more. |
The dune cache inside 'setup-ocaml' is trimmed to 5GB/jobcount, which is currently about 10. But not all jobs here use the dune cache, so separate the workflows. (Also the dune cache will get compressed afterwards). The SDK build does use the dune cache, but eventually it might become smaller, e.g. if we change it to install just the dependencies needed for SDK gen, and not everything. It does have a lot of unrelated jobs though. The workflows need a unique name, otherwise they cancel each-other. The dune cache is ~1.4GB now compressed, which should speed up the setup-ocaml step on next run as more dependencies will be cached. Setup XenAPI environment now only takes 2m51s instead of 5m55s. Signed-off-by: Edwin Török <edwin.torok@cloud.com>
edwintorok
force-pushed
the
private/edvint/speedci
branch
from
373f305
to
3907e21
Compare
lindig
approved these changes
Apr 25, 2024
Vincent-lau
approved these changes
Apr 25, 2024
We should eventually separate the opam dependency install into its own job that uploads an artifact/cache that other jobs can reuse (and that should have a bigger dune cache), however for now just trim the size of the cache. The default is 5GiB, which results in ~1.4GiB compressed, which would hit the 10GiB cache size limit in GitHub way too often. We may need to tweak this value in the future. Signed-off-by: Edwin Török <edwin.torok@cloud.com>
stephenchengCloud
added a commit
to stephenchengCloud/xen-api
that referenced
this pull request
May 7, 2024
- In the master branch PR xapi-project#5592 moved python CI config from main.yml to other.yml, need to apply feature/py3 main.yml changes to other.yml Signed-off-by: Stephen Cheng <stephen.cheng@cloud.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The 'Run OCaml tests' CI step now takes ~6m15s (on a cached run) instead of 15m24s (master).
'opam pin' is very very slow (needs to copy the source repository ~71 times for each opam package), and unnecessary, disabling it yields the biggest savings.
The dune cache is also bigger now (1.4GB) compressed, but saves a lot of time.