build(docker): move both image builds' sccache from Redis to R2 (JEF-584) - #290
Merged
Conversation
…584) Dockerfile and agent/Dockerfile baked `ENV SCCACHE_REDIS=redis://sccache-redis.dev...` and build inside the BuildKit pod, which never sees the runner pod's env — so the JEF-564 runner-side cutover left this half of the cache on redis. That split the shared cache and is one of the things blocking redis from being retired. Adopts the pattern murmurify landed first in JEF-589 rather than inventing a second one: - The R2 config + bucket-scoped token arrive as BuildKit build SECRETS, never ENV or a build-arg. An `ENV AWS_SECRET_ACCESS_KEY=…` or an ARG consumed in a layer persists in `docker history` for every image we push to ghcr — that would publish the token. A build secret is mounted for one RUN, lands in no layer, and is not part of the layer cache key. - Both workflows use `secret-envs`, reading straight from the RUNNER POD's env where the `sccache-r2` Secret is injected via envFrom. - scripts/start-sccache-docker.sh (copied from murmurify, with the fallback dir parameterised as SCCACHE_LOCAL_CACHE_DIR) probes, retries, then degrades. agent/Dockerfile overrides it to /app/agent/target/.sccache-local, which is where THAT image's BuildKit target cache mount lives. NO LONGER A HARD GATE — deliberately. Redis was an in-cluster Service reachable with no credentials, so "can't reach it" really did mean "this builder is misconfigured". R2 is a remote bucket behind a rotatable token: a blip or an expired key would now fail a build for a reason unrelated to the code, while degrading costs one cold compile. SCCACHE_DISABLE therefore becomes belt-and-braces rather than load-bearing — scripts/e2e.sh's ubuntu-latest build would now pass without it, but it still skips the sccache path on a builder with no reason to touch the shared cache. agent/Dockerfile also needed `COPY scripts/start-sccache-docker.sh` — unlike the main image (COPY . .) it copies only agent/ + behavior/. Verified: --selftest passes all 5 fixtures in both copies, the SCCACHE_LOCAL_CACHE_DIR override takes effect, `docker buildx build --call=check` reports no warnings for BOTH Dockerfiles, no comment line survives inside a RUN continuation (it would be a shell comment swallowing the joined command, not a Dockerfile comment), and scripts/ is not excluded by .dockerignore. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013cYVqzH7Xfwea7fAozdQK7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dockerfileandagent/DockerfilebakedENV SCCACHE_REDIS=redis://sccache-redis.dev...and build inside the BuildKit pod, which never sees the runner pod's env — so the JEF-564 runner-side cutover left this half of the cache on redis. That split the shared cache and blocks retiring redis.Adopts the pattern murmurify landed first in JEF-589 (murmurify#480) rather than inventing a second one.
Credential delivery
The R2 config + bucket-scoped token arrive as BuildKit build secrets, never
ENVor a build-arg — anENV AWS_SECRET_ACCESS_KEY=…persists indocker historyfor every image pushed to ghcr. A build secret is mounted for exactly oneRUN, lands in no layer, and isn't part of the layer cache key.Both workflows use
secret-envs, reading straight out of the runner pod's env where thesccache-r2Secret is injected viaenvFrom.No longer a hard gate — deliberately
Redis was an in-cluster Service reachable with no credentials, so "can't reach it" really did mean "this builder is misconfigured". R2 is a remote bucket behind a rotatable token: a blip or an expired key would fail a build for a reason unrelated to the code, while degrading costs one cold compile.
SCCACHE_DISABLEtherefore becomes belt-and-braces rather than load-bearing —scripts/e2e.sh's ubuntu-latest build would now pass without it, but it still skips the sccache path on a builder with no reason to touch the shared cache.Two repo-specific details
agent/DockerfileneededCOPY scripts/start-sccache-docker.sh— unlike the main image (COPY . .) it copies onlyagent/+behavior/.SCCACHE_LOCAL_CACHE_DIRto/app/agent/target/.sccache-local, where that image's BuildKit target cache mount lives, so a degraded build still warms something.Verification
scripts/start-sccache-docker.sh --selftest→ 5/5 fixtures passSCCACHE_LOCAL_CACHE_DIRoverride takes effectdocker buildx build --call=check→ no warnings for both DockerfilesRUNcontinuation — there it would be a shell comment swallowing the rest of the joined command, not a Dockerfile commentscripts/is not excluded by.dockerignore🤖 Generated with Claude Code
https://claude.ai/code/session_013cYVqzH7Xfwea7fAozdQK7