Reorder Dockerfile builder: install deps before copying src#128
Open
Chr96er wants to merge 1 commit into
Open
Conversation
The builder stage copied `src/` before running the dependency install, so any source edit invalidated the (expensive) `uv pip install -r requirements.txt` layer -- every rebuild re-resolved/reinstalled apache-beam[gcp] + transitives. Move `COPY src ./src` after the requirements install so a source-only change only re-runs the package-install layer (`--no-deps .` + `.`), turning cached source rebuilds from ~1-2 min into seconds. The deps layer is now keyed on pyproject.toml / requirements.txt only. This matches the layering anchorages_pipeline already uses. No behavioural change to the resulting image -- same install commands, only the layer ordering changed.
This was referenced May 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Move
COPY src ./srcin thebuilderstage to after theuv pip install -r requirements.txtstep.Why
The builder currently copies
src/before the dependency install:Docker layer caching is sequential, so editing any file under
src/invalidates therequirements.txtinstall layer — every rebuild re-resolves and reinstallsapache-beam[gcp]and its transitives, even though dependencies didn't change.After the reorder, a source-only change re-runs only the package-install layer (
--no-deps .+.), turning cached source rebuilds from ~1–2 min into seconds. The deps layer is keyed onpyproject.toml/requirements.txtonly. This is the same layeringanchorages_pipelinealready uses (its Dockerfile even documents it).Safety
No behavioural change to the resulting image — identical install commands, only the layer ordering changed. The
--no-deps .install still has everything it needs (pyproject.toml,README.md,MANIFEST.incopied earlier;src/copied just before it).Context
This speeds up dit's auto-built worker images (it builds this Dockerfile's
prodtarget per unreviewed run); with the current layering those rebuilds pay the full deps cost on every source change.