Proposal: content-addressed build caching across CI
Status: proposal, needs a storage decision
Context: follows the measurement work in #36942 / #36945
The principle
Nothing should be built twice from identical inputs — across runs, across
branches, across PRs.
Today every run rebuilds everything: the whole nx workspace, all 24 Maven
modules, and the Docker image, from scratch, on a fresh runner. The
Initial Artifact Build that does most of this is the serial prefix gating
all ~25 test jobs.
The fix is content-addressed caching: hash a unit of work's real inputs, and if
that hash has been seen, fetch the output instead of recomputing it. Three
layers, three tools.
| Layer |
Tool |
Today |
Frontend (nx run-many -t build) |
nx remote cache |
local only, discarded with the runner |
| Java modules (CLI, core, all 24) |
Apache Maven Build Cache Extension |
not used |
| Docker image |
buildx registry cache |
no layer cache at all |
The three are independent and can land in parallel.
Why actions/cache cannot be the answer
GitHub Actions caches are branch-scoped. A run reads caches from its own
branch and the default/base branch — never a sibling branch. Two PRs building
identical code share nothing. That is a deliberate security boundary.
So actions/cache only supports save-on-trunk, restore-everywhere: a PR
benefits only from what main has already built.
Second problem: .nx/cache is 2.0 GB on a working machine. The Actions
cache is 10 GB per repo with LRU eviction, already shared with the Maven
repository, pnpm store and Node binary caches. A large, frequently-rewritten
entry would evict the others and could make builds slower.
A real remote cache is keyed purely on the content hash — no branch dimension,
no 10 GB ceiling.
Runner geography (measured, not assumed)
12 samples via PR #36948, cross-checked two ways (Azure IMDS .location and the
Azure Region: line GitHub prints in every job log header — they agreed 100%):
centralus x3 Iowa
westus2 x3 Washington
eastus x2 Virginia
northcentralus x2 Illinois
westus x1 California
eastus2 x1 Virginia
Runners are spread across six US Azure regions, coast to coast. No region
dominates. Consequences:
- A single-region bucket is the wrong shape — fast for roughly one job in
six, slow for the rest (westus↔eastus is ~60–70ms RTT).
- This favours a managed multi-region cache, or a bucket behind a CDN. If
self-hosting in one region anyway, pick a central compromise (us-east-2),
not us-east-1.
- Small per-task entries (nx outputs, module jars) tolerate this well; large
artifacts do not — see the Docker section.
Incidental finding: all runners are 4 vCPU / 15 GB, but the VM SKU varies by
generation (Standard_D4ads_v5 ×10, D4ds_v6 ×1, D4ds_v7 ×1). That is a
concrete cause of run-to-run wall-time noise, and a reason to judge any change
over several runs.
Layer 1 — frontend (nx)
Nx already hashes task inputs; it only lacks somewhere to keep results.
- Nx Cloud — free plan, multi-region edge (which the runner spread makes
genuinely valuable), nothing to operate. Build outputs leave our
infrastructure, so it needs a security review.
- Self-hosted — Nx ships S3/GCS/Azure adapters; self-hosted caching moved
from paid Powerpack to free (activation key). We already have AWS
credentials in CI (deploy-javadoc → s3://static.dotcms.com). Confirm the
adapter's current packaging for Nx 23 before committing — the Nx 23 docs lead
with Nx Cloud and with "build your own server".
- Custom server — stable OpenAPI contract (
PUT/GET /v1/cache/{hash})
plus NX_SELF_HOSTED_REMOTE_CACHE_SERVER and
NX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN. Only if the above are rejected.
Layer 2 — Java modules (Maven Build Cache Extension)
The Apache Maven Build Cache Extension
fingerprints each module's inputs by content digest, caches its outputs, and
skips the module on a hit. Remote cache works over plain HTTP PUT/GET/HEAD,
or anything Maven Wagon speaks (S3, SSH).
This is the answer to "we don't need to build the CLI every build." Rather
than hand-maintaining which modules a PR needs, the hash decides — and it covers
dotcms-cli, dotcms-api-data-model, dotcms-core and the other 21 modules at
once.
It also subsumes two other pieces of work:
Earlier I argued that reusing a prior WAR needs a provably complete predicate
for "did anything relevant change". Content hashing is that predicate,
computed rather than curated, which removes the class of bug where a hand-written
filter drifts and a PR silently tests a stale binary.
The real work: hidden inputs
Cache correctness depends on the hash covering every real input. Plugins that
read undeclared state produce hits that should have been misses. dotCMS has
several to handle explicitly:
- git metadata read during the build (
git rev-parse for ShortRevision /
scmBranch)
openapi.yaml generated by swagger-maven-plugin at compile- starter zip assembly
- the
process-annotations immutables pass
Also enable project.build.outputTimestamp — timestamps embedded at package
time make outputs non-reproducible and silently destroy hit rate.
Layer 3 — Docker image
Currently there is no layer cache: every build runs docker build on a fresh
runner with an empty local cache. buildx registry caching
(--cache-to/--cache-from type=registry) fixes this and transfers only changed
layers.
Do not try to cache the image through the Maven or nx cache. The docker save
tar is ~1 GB, and with runners spread across six regions, pulling that from a
distant cache can cost more than rebuilding. Layer-level caching against a
registry is the right granularity.
What must NOT be cached (at least not first)
Test execution. The Maven extension caches package and later phases by
default and can cache test. Memoised test results mean a green run may mean
"we did not run", not "it passed". We have already measured real flakiness in
this suite, plus a hang that burned a 122m job timeout — masking that is worse
than the time it saves.
Cache compile/package. Run tests. Revisit only once hashing is demonstrably
correct.
Security
A cache that untrusted code can write is a remote code execution vector: a
poisoned entry is replayed as a build output on trunk.
- PR builds get read-only tokens. The nx OpenAPI contract returns
403 when
a read-only token attempts a write; the Maven extension supports a read-only
remote cache the same way.
- Only trusted branches write — trunk and release populate, PRs consume.
- Fork PRs never receive a write token.
- Hashes must include toolchain versions, so a Node or JDK bump cannot collide
with an older entry.
This costs some hit rate — a PR cannot reuse another PR's novel work — but
keeps the case that matters: unchanged subsystems.
Order
- nx remote cache. Biggest single win, best-understood tool, frontend only.
- Maven build cache,
package-and-later, read-only on PRs. Measure hit rate
before trusting it; budget the real effort for hidden inputs above.
- buildx registry cache for Docker. Independent of 1 and 2, and currently a
complete gap.
- Only then consider caching test phases.
Land #36946 first and re-measure — it stops building SDK projects the WAR never
contains, so it shrinks both the serial prefix and the 33m frontend job, and
today's baselines will not hold afterwards.
Proposal: content-addressed build caching across CI
Status: proposal, needs a storage decision
Context: follows the measurement work in #36942 / #36945
The principle
Nothing should be built twice from identical inputs — across runs, across
branches, across PRs.
Today every run rebuilds everything: the whole nx workspace, all 24 Maven
modules, and the Docker image, from scratch, on a fresh runner. The
Initial Artifact Buildthat does most of this is the serial prefix gatingall ~25 test jobs.
The fix is content-addressed caching: hash a unit of work's real inputs, and if
that hash has been seen, fetch the output instead of recomputing it. Three
layers, three tools.
nx run-many -t build)The three are independent and can land in parallel.
Why
actions/cachecannot be the answerGitHub Actions caches are branch-scoped. A run reads caches from its own
branch and the default/base branch — never a sibling branch. Two PRs building
identical code share nothing. That is a deliberate security boundary.
So
actions/cacheonly supports save-on-trunk, restore-everywhere: a PRbenefits only from what
mainhas already built.Second problem:
.nx/cacheis 2.0 GB on a working machine. The Actionscache is 10 GB per repo with LRU eviction, already shared with the Maven
repository, pnpm store and Node binary caches. A large, frequently-rewritten
entry would evict the others and could make builds slower.
A real remote cache is keyed purely on the content hash — no branch dimension,
no 10 GB ceiling.
Runner geography (measured, not assumed)
12 samples via PR #36948, cross-checked two ways (Azure IMDS
.locationand theAzure Region:line GitHub prints in every job log header — they agreed 100%):Runners are spread across six US Azure regions, coast to coast. No region
dominates. Consequences:
six, slow for the rest (westus↔eastus is ~60–70ms RTT).
self-hosting in one region anyway, pick a central compromise (
us-east-2),not
us-east-1.artifacts do not — see the Docker section.
Incidental finding: all runners are 4 vCPU / 15 GB, but the VM SKU varies by
generation (
Standard_D4ads_v5×10,D4ds_v6×1,D4ds_v7×1). That is aconcrete cause of run-to-run wall-time noise, and a reason to judge any change
over several runs.
Layer 1 — frontend (nx)
Nx already hashes task inputs; it only lacks somewhere to keep results.
genuinely valuable), nothing to operate. Build outputs leave our
infrastructure, so it needs a security review.
from paid Powerpack to free (activation key). We already have AWS
credentials in CI (
deploy-javadoc→s3://static.dotcms.com). Confirm theadapter's current packaging for Nx 23 before committing — the Nx 23 docs lead
with Nx Cloud and with "build your own server".
PUT/GET /v1/cache/{hash})plus
NX_SELF_HOSTED_REMOTE_CACHE_SERVERandNX_SELF_HOSTED_REMOTE_CACHE_ACCESS_TOKEN. Only if the above are rejected.Layer 2 — Java modules (Maven Build Cache Extension)
The Apache Maven Build Cache Extension
fingerprints each module's inputs by content digest, caches its outputs, and
skips the module on a hit. Remote cache works over plain HTTP
PUT/GET/HEAD,or anything Maven Wagon speaks (S3, SSH).
This is the answer to "we don't need to build the CLI every build." Rather
than hand-maintaining which modules a PR needs, the hash decides — and it covers
dotcms-cli,dotcms-api-data-model,dotcms-coreand the other 21 modules atonce.
It also subsumes two other pieces of work:
Earlier I argued that reusing a prior WAR needs a provably complete predicate
for "did anything relevant change". Content hashing is that predicate,
computed rather than curated, which removes the class of bug where a hand-written
filter drifts and a PR silently tests a stale binary.
The real work: hidden inputs
Cache correctness depends on the hash covering every real input. Plugins that
read undeclared state produce hits that should have been misses. dotCMS has
several to handle explicitly:
git rev-parseforShortRevision/scmBranch)openapi.yamlgenerated byswagger-maven-pluginat compileprocess-annotationsimmutables passAlso enable
project.build.outputTimestamp— timestamps embedded at packagetime make outputs non-reproducible and silently destroy hit rate.
Layer 3 — Docker image
Currently there is no layer cache: every build runs
docker buildon a freshrunner with an empty local cache. buildx registry caching
(
--cache-to/--cache-from type=registry) fixes this and transfers only changedlayers.
Do not try to cache the image through the Maven or nx cache. The
docker savetar is ~1 GB, and with runners spread across six regions, pulling that from a
distant cache can cost more than rebuilding. Layer-level caching against a
registry is the right granularity.
What must NOT be cached (at least not first)
Test execution. The Maven extension caches
packageand later phases bydefault and can cache
test. Memoised test results mean a green run may mean"we did not run", not "it passed". We have already measured real flakiness in
this suite, plus a hang that burned a 122m job timeout — masking that is worse
than the time it saves.
Cache compile/package. Run tests. Revisit only once hashing is demonstrably
correct.
Security
A cache that untrusted code can write is a remote code execution vector: a
poisoned entry is replayed as a build output on trunk.
403whena read-only token attempts a write; the Maven extension supports a read-only
remote cache the same way.
with an older entry.
This costs some hit rate — a PR cannot reuse another PR's novel work — but
keeps the case that matters: unchanged subsystems.
Order
package-and-later, read-only on PRs. Measure hit ratebefore trusting it; budget the real effort for hidden inputs above.
complete gap.
Land #36946 first and re-measure — it stops building SDK projects the WAR never
contains, so it shrinks both the serial prefix and the 33m frontend job, and
today's baselines will not hold afterwards.