The nightly variant of build-test-image.yml consistently takes substantially longer than 2023.09. Observed on commit 593c05f:
| Variant |
Build time |
build 2023.09 |
6m56s |
build nightly |
~24m (gated the whole CI workflow at 42m total) |
Since the nightly image is rebuilt on every change to tests/environment.*.yml, docker/Dockerfile.nmisp-test, or the workflow itself, this gating cost is paid often.
Suggested investigation
- Compare
tests/environment.2023.09.yml and tests/environment.nightly.yml — likely the nightly env pulls heavier ML deps (TensorFlow / PyTorch nightly wheels are large).
- Check whether
docker/build-push-action's GHA cache (cache-from: type=gha, cache-to: type=gha,mode=max) is actually hitting on these builds — if cache is invalidated by every Dockerfile change the cache provides little value.
- Consider splitting the heavy ML install into its own Dockerfile stage so it caches independently from the rest of the env.
Why
The nightly build dominates the total wall-clock of any CI change that touches the Dockerfile/env. Halving it would meaningfully tighten the iteration loop on container/CI changes.
The
nightlyvariant ofbuild-test-image.ymlconsistently takes substantially longer than2023.09. Observed on commit 593c05f:build 2023.09build nightlyCIworkflow at 42m total)Since the nightly image is rebuilt on every change to
tests/environment.*.yml,docker/Dockerfile.nmisp-test, or the workflow itself, this gating cost is paid often.Suggested investigation
tests/environment.2023.09.ymlandtests/environment.nightly.yml— likely the nightly env pulls heavier ML deps (TensorFlow / PyTorch nightly wheels are large).docker/build-push-action's GHA cache (cache-from: type=gha,cache-to: type=gha,mode=max) is actually hitting on these builds — if cache is invalidated by every Dockerfile change the cache provides little value.Why
The nightly build dominates the total wall-clock of any CI change that touches the Dockerfile/env. Halving it would meaningfully tighten the iteration loop on container/CI changes.