chore(ci): switch sccache from local disk to memcached backend#68
Merged
Conversation
Replace SCCACHE_DIR with SCCACHE_MEMCACHED_ENDPOINT (tcp://memcached.sccache:11211) across CI configs, Dockerfiles, and build scripts. Docker builds receive the endpoint as a build arg so CI gets shared caching while local builds degrade gracefully.
Read SCCACHE_MEMCACHED_ENDPOINT from GitHub repository variables instead of hardcoding the value. Revert GitLab CI changes since that environment is not configured for memcached yet. Remove the mise.toml default so local dev without memcached falls back to local sccache storage.
Unset empty SCCACHE_MEMCACHED_ENDPOINT in cross-build.sh so sccache falls back to local disk cache. Remove the ENV line from Dockerfiles since the ARG alone is sufficient for RUN layers.
BuildKit cache mounts can retain stale .rmeta files from prior builds with different dependency versions. If cargo build fails, clean the target directory and retry once so builds self-heal.
With sccache backed by memcached (CI) or local disk (dev), compilation caching is handled externally. The BuildKit cargo-target cache mounts were the sole source of stale .rmeta files that caused build failures when dependencies changed. Removing them eliminates the staleness problem entirely and the touch-to-invalidate workarounds they required.
…t mounts Replace cargo-target BuildKit cache mounts with sccache local disk cache mounts (/tmp/sccache). This avoids stale .rmeta artifacts that plague target/ mounts across dependency changes, while still giving local dev builds fast incremental rebuilds. In CI, sccache uses memcached and ignores the local disk mount. cross-build.sh defaults SCCACHE_DIR to /tmp/sccache to align with the Dockerfile mount targets.
Bring back cargo-target BuildKit cache mounts alongside sccache disk mounts. Without target/, cargo recompiles all dependencies on every source change since it has no fingerprint state. The target mount gives true incremental rebuilds locally; sccache provides a second cache layer that survives target/ invalidation from dep changes. The touch workarounds are restored to handle stale proto/build.rs artifacts in the target mount.
If cargo build fails (e.g. stale .rmeta in the BuildKit cargo-target cache mount from a dependency version change), clean the target dir and retry. The sccache layer still has compiled objects so the clean rebuild is fast. This self-heals without manual docker builder prune.
Cache target/ directory and cargo registry between CI runs so unchanged dependencies skip compilation entirely instead of relying solely on sccache for individual compilation units.
The aqua backend for sccache does not support linux/arm64 despite arm64 binaries being available on GitHub releases. Switch to the ubi backend which downloads directly from GitHub releases and correctly resolves the architecture. Also add mise install steps to checks workflow to ensure tools are available when the CI image may not have them pre-installed.
The ubi backend installed a broken sccache binary on linux/amd64 that did not support RUSTC_WRAPPER mode. The github backend properly extracts the release tarball and provides the correct binary.
Swatinem/rust-cache runs rustc -vV to compute the cache key, but rustup has no default toolchain configured in the CI image (mise manages Rust via shims). Without this, the cache action silently fails and every run compiles from scratch.
drew
force-pushed
the
sccache-memcached-backend/an
branch
from
0451b81 to
798d150
Compare
The previous approach wrote 'export KEY=value' shell syntax directly to $GITHUB_ENV, which only accepts KEY=value format. This caused PATH not to be updated, so sccache resolved to the mise shim instead of the installed binary, failing with 'No version is set for shim: sccache'. Split activation into: - mise env --dotenv for env vars (proper KEY=VALUE format) - GITHUB_PATH for PATH entries (correct GH Actions mechanism)
The mise shim at /opt/mise/shims/sccache cannot resolve the tool version when cargo spawns it as a subprocess. Use 'mise which sccache' to get the absolute binary path, bypassing the shim entirely.
mise which sccache fails on amd64 with 'sccache is a mise bin however it is not currently active'. Run mise use -g to register the tool globally so both the shim and mise which can resolve it.
The mise shim maps the sccache binary to ubi:mozilla/sccache, but the tool was installed as github:mozilla/sccache. This backend mismatch causes the shim to fail with 'No version is set for shim: sccache' on amd64. The github: backend is deprecated in favor of ubi: in newer mise versions — switching aligns the tool declaration with what the shim expects. Also removes the now-unnecessary 'Activate mise environment' CI step.
ubi:mozilla/sccache picks sccache-dist instead of sccache on x86_64 linux because both release assets match the platform and ubi selects sccache-dist first alphabetically. The sccache-dist binary is the distributed build scheduler, not the compiler cache wrapper, causing RUSTC_WRAPPER to fail with 'unrecognized subcommand'. Use matching="sccache-v" to filter release assets at download time, ensuring ubi selects sccache-v0.14.0-* instead of sccache-dist-v0.14.0-*. The exe option alone is insufficient because ubi partial-matches sccache-dist containing sccache. arm64 is unaffected since sccache-dist has no arm64 release asset.
drew
force-pushed
the
sccache-memcached-backend/an
branch
from
e7b2512 to
6219fc6
Compare
drew
added a commit
that referenced
this pull request
Mar 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SCCACHE_DIR(local disk cache) withSCCACHE_MEMCACHED_ENDPOINT(tcp://memcached.sccache:11211) for shared compilation caching across CI runnersactions/cache) and GitLab CI cache configChanges
mise.tomlSCCACHE_DIR→SCCACHE_MEMCACHED_ENDPOINT(with env override)Dockerfile.serverENV SCCACHE_DIR→ARG/ENV SCCACHE_MEMCACHED_ENDPOINT; remove sccache cache mountsDockerfile.base(sandbox)Dockerfile.python-wheels.github/workflows/checks.ymlSCCACHE_MEMCACHED_ENDPOINTenv; remove disk cache step.gitlab-ci.ymlSCCACHE_MEMCACHED_ENDPOINTvariable; remove sccache disk cache entrydocker-build-component.shSCCACHE_MEMCACHED_ENDPOINTas--build-argdocker-publish-multiarch.shSCCACHE_MEMCACHED_ENDPOINTas--build-argarchitecture/build-containers.mdPrerequisites
tcp://memcached.sccache:11211from CI runners and Docker build network