Skip to content

CI: profile nightly image build (3-4x slower than 2023.09) #370

@kwlee2025cpp

Description

@kwlee2025cpp

The nightly variant of build-test-image.yml consistently takes substantially longer than 2023.09. Observed on commit 593c05f:

Variant Build time
build 2023.09 6m56s
build nightly ~24m (gated the whole CI workflow at 42m total)

Since the nightly image is rebuilt on every change to tests/environment.*.yml, docker/Dockerfile.nmisp-test, or the workflow itself, this gating cost is paid often.

Suggested investigation

  1. Compare tests/environment.2023.09.yml and tests/environment.nightly.yml — likely the nightly env pulls heavier ML deps (TensorFlow / PyTorch nightly wheels are large).
  2. Check whether docker/build-push-action's GHA cache (cache-from: type=gha, cache-to: type=gha,mode=max) is actually hitting on these builds — if cache is invalidated by every Dockerfile change the cache provides little value.
  3. Consider splitting the heavy ML install into its own Dockerfile stage so it caches independently from the rest of the env.

Why

The nightly build dominates the total wall-clock of any CI change that touches the Dockerfile/env. Halving it would meaningfully tighten the iteration loop on container/CI changes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions