Skip to content

[CI] Docker dependency layers (cargo-chef / cache mounts) #45

Description

@brendanPro

Summary: Speed up rebuilds with proper Docker/cargo layer caching, and stop Buildx GHA cache export from failing a job after a successful image push.

Parent epic: #44

Objective / Problem

Two related cache problems:

  1. Compile layersdeploy/Dockerfile copies the whole workspace then runs dx build / cargo build --release. Any crate change invalidates the compile layer; cold CI also pays full dep compile twice (UI stage + controller stage).
  2. GHA cache-to is unsafe at our size — first multi-arch run (ci: Kustomize product package + just check + GHCR publish #22 / Actions run 30160511674) built and pushed main + sha-71ac73a to GHCR, then failed for ~4h36 wall time with:
    #41 exporting to GitHub Actions Cache
    #41 ERROR: not_found
    ERROR: failed to build: failed to solve: not_found
    
    Likely causes: GHA cache size/limits or Buildx type=gha export race on a huge multi-arch Rust cache. Current workflow uses cache-from: type=gha and cache-to: type=gha,mode=max in image.yml.

Proposed Solution

Decide with data; do not assume one approach:

A. Reliable cache export (must fix redness)

  1. Prefer registry cache (type=registry) on GHCR, or drop cache-to until layering is sane.
  2. Or keep cache-from only; never fail the job on cache export (if Buildx/action supports it).
  3. Avoid mode=max for multi-arch Rust if it blows past GHA cache limits.

B. Dependency layering (compile speed)

  1. cargo-chef (or BuildKit cache mounts for cargo registry + target) in UI and builder stages.
  2. Recipe/cook so deps reuse across commits when Cargo.lock is unchanged.
  3. Keep distroless final stage unchanged.

Evidence to preserve

QA

  • Successful image push ⇒ green workflow even if cache export is skipped/fails.
  • Second CI image build with only app-code diff reuses dep layers (buildx timings).
  • Image still runs; UI assets embedded.
  • Lockfile / dep changes still rebuild deps correctly.
  • Document chosen cache strategy in deploy/README.md or workflow comments.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions