Found while implementing #3523 (PR #3722), which copies this job's should_run shape into four other jobs. Filed rather than fixed there: #3722's ruling scopes it to the merge_group triggers and the ci.yml/lint.yml paths-ignore, and this is a pre-existing defect in a different direction.
The code
.github/workflows/ci.yml, the docs job (Build Docs), step Check for docs changes:
CHANGED=$(git diff --name-only ${{ github.event.pull_request.base.sha }}...${{ github.event.pull_request.head.sha }} -- \
'apps/site/' 'content/' 2>/dev/null || echo "")
if [ -n "$CHANGED" ]; then should_run=true else should_run=false fi
2>/dev/null || echo "" collapses two different facts into one value:
- "the diff succeeded and nothing under
apps/site/ or content/ changed" — correctly should_run=false;
- "the diff could not be computed at all" — a checkout that did not fetch deep enough, a transient git failure, a malformed sha — which also yields
should_run=false.
The second case skips the entire site build and the job still reports success. There is no red step, no warning, nothing in the summary: the check says the docs built when nothing was built.
Measured, both directions
Against a fixture repository, feeding an unreachable base sha into the two spellings of the same gate (PR #3722 ran this as its reverse verification):
| Spelling |
Unreachable base sha |
fail-OPEN — if ! CHANGED=$(git diff …); then should_run=true; exit 0; fi |
should_run=true — runs everything |
fail-CLOSED — … 2>/dev/null || echo "" (today's docs job) |
should_run=false — silent full skip |
Why it matters beyond this job
objectstack#4928 named this "the filter contract" after exactly this shape produced a fully green, zero-job pull request with no red signal anywhere — a flake in the filter, not in the code, decided that nothing ran. The rule it settled on is: when the filter cannot tell, RUN.
Today the blast radius here is bounded — Build Docs is not on the list of contexts #3523 found safe to require, so a false green on it does not merge anything by itself. But the shape is the one this repository keeps paying for, and #3722 has just made this job the odd one out: the four gates it added (type-check, test, e2e, lint) all fail open, and scripts/__tests__/merge-queue-reporting.test.ts pins them that way. This one still does not.
Suggested fix
Rewrite the capture as the fail-open form the new gates use, so the job runs the site build when it cannot establish that nothing docs-related changed. One step, no new mechanism. While there, the same step's push branch was already corrected by #3722 to != 'pull_request' so queue builds are covered — the diff itself is the only part left.
Found while implementing #3523 (PR #3722), which copies this job's
should_runshape into four other jobs. Filed rather than fixed there: #3722's ruling scopes it to themerge_grouptriggers and theci.yml/lint.ymlpaths-ignore, and this is a pre-existing defect in a different direction.The code
.github/workflows/ci.yml, thedocsjob (Build Docs), stepCheck for docs changes:2>/dev/null || echo ""collapses two different facts into one value:apps/site/orcontent/changed" — correctlyshould_run=false;should_run=false.The second case skips the entire site build and the job still reports success. There is no red step, no warning, nothing in the summary: the check says the docs built when nothing was built.
Measured, both directions
Against a fixture repository, feeding an unreachable base sha into the two spellings of the same gate (PR #3722 ran this as its reverse verification):
if ! CHANGED=$(git diff …); then should_run=true; exit 0; fishould_run=true— runs everything… 2>/dev/null || echo ""(today'sdocsjob)should_run=false— silent full skipWhy it matters beyond this job
objectstack#4928 named this "the filter contract" after exactly this shape produced a fully green, zero-job pull request with no red signal anywhere — a flake in the filter, not in the code, decided that nothing ran. The rule it settled on is: when the filter cannot tell, RUN.
Today the blast radius here is bounded —
Build Docsis not on the list of contexts #3523 found safe to require, so a false green on it does not merge anything by itself. But the shape is the one this repository keeps paying for, and #3722 has just made this job the odd one out: the four gates it added (type-check,test,e2e,lint) all fail open, andscripts/__tests__/merge-queue-reporting.test.tspins them that way. This one still does not.Suggested fix
Rewrite the capture as the fail-open form the new gates use, so the job runs the site build when it cannot establish that nothing docs-related changed. One step, no new mechanism. While there, the same step's
pushbranch was already corrected by #3722 to!= 'pull_request'so queue builds are covered — the diff itself is the only part left.