♻️ CI build times: leverage docker buildx github registry caching and transmit builds to int/sys tests

[sanderegg]

What do these changes do?

This PR changes how the simcore stack is built in the inegration/system tests in the CI:

Before

• build-test-images job: would build the simcore docker images and push them to the Dockerhub registry (if on upstream branch), or do nothing (if on a PR branch due to the lack of available credentials due to security)
• [int] tests or [sys] tests: would either pull the simcore docker images (from the build-test-images job) or build the whole stack on the fly (taking about 10-12minutes each time), then run the tests

After

• build-test-images job: leverages --cache-from and --cache-to options of docker buildx. This allows to cache builds between CI runs on the Github Registry and therefore permits faster builds on subsequent pushes on both PRs and upstream branch as they do not need credentials. Observed time gains on builds goes from 10 minutes on the first commit to 3 minutes on the subsequent pushes (with still quite some potential as these 3 minutes mean we are bad with layer caching). Of course it depends on how many changes were done between 2 CI push.
• build-test-images job: additionally the job pushes the images as github artifacts, so that subsequent jobs can use it
• [int] tests or [sys] tests: now download the docker images from github artifacts, then load them in the github runner docker registry (takes about 1 minute) and then run the tests

--> performance gain: all integration/system tests time to run reduce by about 10-12 minutes. Also since the build-test-images job runs alone for some time it reduces the number of github runners needed by each CI push, thus liberating some compute power to have more CI pushes at the same time.

Bonus:

• no need to build dask-scheduler image, it is the same as the dask-sidecar
• no need to start ops tools in e2e tests
• a few tests marked as flaky (list of flaky tests in CI #3421)

Potential remaining improvements

• frontend systematically recompiles all the sources with each change of the GIT_SHA (which happens at every commit), and takes a long time
• use github cache instead of upload-artifact action as it is much faster to upload
• push to ghcr.io registry (but not sure this is accessible from pull requests), this would allow using docker-compose pull/push and maybe scratch another minute?

Related issue/s

How to test

Checklist

[codecov]

Codecov Report

Merging #3369 (aae7966) into master (eeaf834) will decrease coverage by 0.0%.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           master   #3369     +/-   ##
========================================
- Coverage    83.2%   83.2%   -0.1%     
========================================
  Files         823     823             
  Lines       34866   34866             
  Branches     1365    1365             
========================================
- Hits        29042   29030     -12     
- Misses       5638    5650     +12     
  Partials      186     186             

Flag	Coverage Δ
integrationtests	68.2% <ø> (-0.1%)	⬇️
unittests	80.1% <ø> (-0.1%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../simcore_service_catalog/db/repositories/groups.py	64.8% <0.0%> (-13.6%)	⬇️
...rvice_datcore_adapter/utils/requests_decorators.py	75.6% <0.0%> (-8.2%)	⬇️
...rary/src/servicelib/fastapi/requests_decorators.py	80.3% <0.0%> (-4.0%)	⬇️
...rc/simcore_service_catalog/db/repositories/dags.py	44.4% <0.0%> (-2.8%)	⬇️
...mcore_service_webserver/garbage_collector_utils.py	85.8% <0.0%> (-1.3%)	⬇️
.../simcore_service_catalog/services/access_rights.py	86.2% <0.0%> (-1.3%)	⬇️
...simcore_service_director_v2/modules/node_rights.py	98.1% <0.0%> (-1.0%)	⬇️
...imcore_service_webserver/garbage_collector_core.py	69.8% <0.0%> (+1.2%)	⬆️

[mrnicegyu11]

awesome, thx!!

[pcrespov]

🎉 fantastic! This will be a game-changes reducing the time-to-deploy! Thx so much SAN!

[sanderegg]

🎉 fantastic! This will be a game-changes reducing the time-to-deploy! Thx so much SAN!

well hold your horses... it's a bit better, it's not yet blazing fast...

[GitHK]

👍 Very welcomed changes! Thanks!

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication