Skip to content

fix(ci): stop exempting the gate's own fixture corpus from full-gate-build - #151

Merged
forkwright merged 2 commits into
mainfrom
fix/122-gate-fixture-corpus-exemption
Aug 16, 2026
Merged

fix(ci): stop exempting the gate's own fixture corpus from full-gate-build#151
forkwright merged 2 commits into
mainfrom
fix/122-gate-fixture-corpus-exemption

Conversation

@forkwright

@forkwright forkwright commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Finding

.github/workflows/gate-attestation.yml:137 passed docs_only_exemption: true to
forkwright/.github's hybrid-gate.yml reusable, pinned at
84a39d3344660e4010c32755b33a6a88912ad929. That reusable's check-trailer job, in
its "Check for docs-only changeset" step, classifies any diff where every changed
path matches *.md/docs/*/llms.txt as docs-only and skips full-gate-build
entirely. The step runs hybrid-gate.yml:155-219 at the pinned SHA (:205-276 at
current forkwright/.github main); the case-pattern match itself is at :208 pinned
/ :266 main — verified both by git show <ref>:.github/workflows/hybrid-gate.yml
against the pinned SHA and origin/main. The pattern-match arm (*.md|docs/*|llms.txt)
is byte-identical between the two; only the diff-range computation feeding it changed
(two-dot origin/${BASE_REF}..HEAD at the pinned SHA → three-dot ...HEAD on main,
an unrelated merge-base-vs-moving-base fix), which does not change what counts as a
match for a fixed diff like this repo's. examples/*/content/**.md matches *.md,
but it is not documentation: it is the fixture corpus ci/run-fixtures.sh builds and
asserts against via bin/typikon-check (ci/run-fixtures.sh:16-17). A diff confined
to that corpus is therefore classified exempt, when it is exactly the diff most in
need of the gate.

Evidence

PR #120 touched only two fixture-corpus files:

examples/sample-shop/content/sizing.md
examples/sample-shop/content/sizing-timed.md

Run 31339059510:
ai-attribution success, check-trailer success, gate success, full-gate-build
skipped. Three green checks and an aggregate gate pass with the one job that
builds anything never having run.

Why this matters

A fixture is how this repo proves a claim. A build failure #120's fixture would have
introduced — a schema violation, a broken link, a pa11y regression — would have
reached main with four green checks behind it, discovered only by the next PR that
happens to touch a .sh/.ts file. The classifier lives in a separate shared repo
(forkwright/.github) and keys on file extension alone; it has no notion of "gate
input" vs. "documentation" to parameterize per caller, so no path-level fix is
possible from this repo. Teaching the shared classifier to exclude gate-input globs
fleet-wide is out of scope here by design — issue #122 itself flags that as needing
deliberate review across every consumer, not a drive-by edit riding this fix.

Desired correction

.github/workflows/gate-attestation.yml:137 now sets docs_only_exemption: false
with an inline comment recording why: no diff can be exempted here without the
shared classifier being able to distinguish examples/** from real docs, which it
currently cannot. full-gate-build now runs on every PR to this repo regardless of
which paths changed — the repo is public, so this trades no metered CI budget.

ci/check-fixture-corpus-exemption.sh is the negative-case fixture, wired into
ci/run-fixtures.sh:15 (so it runs as part of the gate's own check_cmd, not just
locally). It reads this repo's actual configured docs_only_exemption value out of
gate-attestation.yml (never hardcodes it) and, against PR #120's exact two-file
diff:

  1. Control — asserts the diff still matches the shared classifier's bare pattern
    (*.md|docs/*|llms.txt), i.e. it is a faithful reproduction of docs-only exemption skips the gate for changes to the gate's own fixture corpus #122's known-bad
    shape, not a pattern invented for this test.
  2. Regression guard — computes the same effective decision hybrid-gate.yml
    computes (pattern_match && docs_only_exemption == "true") and asserts it is
    false.

Ran directly against the un-fixed workflow (docs_only_exemption: true), it fails:

FAIL: gate-attestation.yml (docs_only_exemption: true,
hybrid-gate.yml@84a39d3344660e4010c32755b33a6a88912ad929) would classify a diff
confined to examples/sample-shop/content/sizing.md
examples/sample-shop/content/sizing-timed.md as docs-only and skip
full-gate-build -- this is #122

Against the fixed workflow in this PR, it passes:

{"checked":"fixture-corpus-exemption","status":"pass","docs_only_exemption":"false",
"pinned_sha":"84a39d3344660e4010c32755b33a6a88912ad929"}

A reviewer can disprove this by: reverting docs_only_exemption to true and
showing ci/check-fixture-corpus-exemption.sh still passes (it will not — that is
the fixture's whole job), or by showing the two cited fixture paths no longer match
the shared classifier's pattern (they do — both end in .md), or by showing
ci/run-fixtures.sh does not actually invoke the new script (it does, at line 15).

cargo fmt is not applicable (no Rust in this repo — fmt_cmd/clippy_cmd/
nextest_cmd are all no-ops per gate-attestation.yml's own header comment).
shellcheck ci/check-fixture-corpus-exemption.sh and shellcheck ci/run-fixtures.sh
are both clean. kanon lint . --all reports zero findings against every file this PR
touches (one pre-existing, unrelated PY/bare-except finding in
ci/asset-provenance-scan.py predates this branch and is untouched by it).

On #122's literal Done-when

#122's Done-when reads: "a PR touching only examples/*/content/**.md runs
full-gate-build, demonstrated by a run link." The evidence above is a local
shell-script simulation of the classifier plus PR #120's old run (which shows the
bug, not the fix) — neither is that run link. This is not an oversight; it is a
sequencing constraint worth stating plainly rather than arguing around:

gate-attestation.yml's trigger is pull_request: branches: [main] (:65-69) —
GitHub filters pull_request on the PR's base branch, so the workflow only fires
at all when a PR targets main. For a pull_request run, GitHub executes the
workflow file as it exists in the merge of head into base — which is why this PR's
own head already ran full-gate-build under its own docs_only_exemption: false
(run 31921071233, this
PR's current head 059fc7d — all 4 jobs pass) even though main doesn't have that
value yet. But the PR diff GitHub
shows is always head vs. base: as long as main lacks this fix, any PR built on top
of it that also touches only fixture content would carry this fix's own workflow and
script changes in that diff too — it would not be "a PR touching only
examples/*/content/**.md." There is no diff shape that is simultaneously (a)
confined to fixture content and (b) evaluated against the fixed classifier unless the
fix is already merged to main. The empirical demonstration #122 asks for is
therefore only producible after this PR merges, not before it.

Plan, so this doesn't get silently dropped: merge this PR, then open a follow-up PR
against the now-fixed main touching only a fixture-corpus file (e.g. a no-op
content edit to examples/sample-shop/content/sizing.md), and post that run's link
as a comment on #122 before closing it. #122 is intentionally NOT auto-closed by this
PR (see below) so that comment is the actual closing action, not this merge.

Addresses #122 — left open; closed separately once the run-link comment above lands,
per the plan in "On #122's literal Done-when."

forkwright added 2 commits August 15, 2026 20:23
…build

docs_only_exemption:true let the shared classifier's *.md pattern treat
examples/*/content/**.md as documentation, so a diff confined to typikon's
own fixture corpus (PR #120) skipped full-gate-build with three green
checks. The classifier is a shared reusable and cannot be parameterized
per-repo, so the local predicate is now false: full-gate-build always
runs here. ci/check-fixture-corpus-exemption.sh proves the corpus-only
diff both matches the docs-only pattern (control) and is no longer
exempt under this repo's configured value (regression guard).
…eaders

'WHAT this proves, as a negative case:' and 'Usage:' were freeform prose outside the closed comment-tag set (STANDARDS.md Structured comment tags). Retagged both as NOTE: -- the closest fit among WHY/WARNING/NOTE/PERF/SAFETY/INVARIANT for non-design-rationale context.
@forkwright
forkwright merged commit 8690c82 into main Aug 16, 2026
4 checks passed
@forkwright
forkwright deleted the fix/122-gate-fixture-corpus-exemption branch August 16, 2026 02:09
forkwright pushed a commit that referenced this pull request Aug 16, 2026
🤖 I have created a release *beep* *boop*
---


##
[0.4.0](v0.3.0...v0.4.0)
(2026-08-16)


### Features

* **schemas:** add a fail-closed consumer schema registry
([#154](#154))
([3d9432a](3d9432a))
* **templates:** make the favicon path configurable like every sibling
brand asset ([#141](#141))
([729ce2e](729ce2e))


### Bug Fixes

* **ci:** stop exempting the gate's own fixture corpus from
full-gate-build
([#151](#151))
([8690c82](8690c82))
* **templates:** give the home page a genuine top-level heading
([#152](#152))
([880f711](880f711))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
forkwright added a commit that referenced this pull request Aug 17, 2026
…d claim-accuracy defects (#171)

## Summary

Fixes 16 of the 18 findings in #92; the other 2 were already fixed on
`main`
by prior PRs (evidence below, per `#5a` in the fleet agent brief). One
additional fix (JSON-LD script-injection escaping) generalizes beyond
the
single file the issue cited to all six `partials/ld-*.html` partials,
since
every sibling shared the identical unescaped `json_encode` pattern —
fixing
only the named file would have left the same defect live in five others.

## Acceptance criteria

Issue #92: `Done when: every bullet is either fixed, or closed with a
stated
reason it is not a defect.`

### `bin/` (7)

1. **`bin/typikon-check:65` — `base_host()` single-quoted TOML.** Fixed
—
   `bin/typikon-check:103-112`. `base_host()`'s `sed` now matches both
`"..."` and `'...'` TOML string forms. Verified: piping a single-quoted
`base_url = 'https://example.com'` through the pre-fix pipeline produced
`base_url = 'https:` (garbage); through the fixed pipeline,
`example.com`
   for both quote styles.
2. **`bin/typikon-check:7` — "fail-fast" header claim.** Already fixed —
`bin/typikon-check:7-9` currently reads "Runs every gate stage in order.
   Stages do NOT stop at the first failure." Fixed in PR #103
   (`3. The header claimed fail-fast; it isn't`). No change in this PR.
3. **`bin/typikon-init:107` — "always refresh" comment vs skip-if-exists
code.** Fixed — `bin/typikon-init:131-137`. Corrected the comment: these
   files are create-once (matches `typikon-refresh`'s own documented
contract, which explicitly does NOT re-render `_headers`/`_redirects`).
4. **`bin/typikon-init:91` — swallowed theme-submodule refresh
failure.**
   Fixed — `bin/typikon-init:90-97`. A failed `git pull --ff-only` now
prints `warn themes/typikon: refresh failed, continuing with existing
checkout: <git's own error>` to stderr instead of `|| true`-swallowing
   it silently. Still non-fatal (an offline forge on a re-run must not
   abort the whole scaffold).
5. **`bin/typikon-validate:138` — JSON Pointer root emits `"/"` not
   `""`.** Fixed — `bin/typikon-validate:420-428`. A root-level failure
   (`err.absolute_path` empty) now emits `pointer: ""` per RFC 6901.
6. **`bin/typikon-validate:155` — `load_schemas()` raw traceback on
   malformed schema.** Fixed — `bin/typikon-validate:493-499` and
   `:592-599` (both the per-slug loop and the `.core.schema.json` loop
   carried the identical unwrapped `json.loads()`). Both now catch and
   report `{"error": "malformed schema <path>: <exc>"}` + exit 2.
7. **`bin/typikon-validate:29` — exit-code docstring contradicts
   implementation for malformed frontmatter.** Fixed —
`bin/typikon-validate:57-64`. Malformed frontmatter is folded into the
per-file `failures` list (exit 1), not the runtime-error class (exit 2);
the docstring said the opposite. Corrected the doc; the code was already
the intended behavior (same JSONL per-file reporting shape as a schema
   failure).

### `ci/` (4, + 1 bonus from the issue thread)

8. **`ci/csp-enforce.sh:46` — unbounded `grep`/argv, E2BIG risk.** Fixed
—
`ci/csp-enforce.sh:65-70`, `ci/csp-scan.py:106-124`. File paths now go
over stdin (bash builtin `printf`, no `execve()` of its own) instead of
argv; `csp-scan.py` reads newline-delimited paths from stdin when argv
   is empty.
9. **`ci/kanon-ci.toml.tmpl:40` — missing TLS pinning vs the GitHub
template it claims to mirror.** Fixed — `ci/kanon-ci.toml.tmpl:41,75`.
Both `curl` calls (zola, lychee) now carry `--proto '=https' --tlsv1.2`,
   matching `ci/github-workflow.yml.tmpl:64,110`.
10. **`ci/lychee.toml:12` — false cross-run cache-persistence claim.**
    Fixed — `ci/lychee.toml:11-16`. Corrected: neither CI template
    persists `.lychee_cache/` across runs today (verified: no
`actions/cache` / forge equivalent in either template). `cache = true`
stays on — it still dedupes repeat hits to the same URL within one run.
11. **`ci/playwright.config.ts:10` — header claims `public/`, gate
    actually serves `public-local/`.** Already fixed —
    `ci/playwright.config.ts:16-21` currently describes `public-local/`
correctly, with an OS-allocated port (no hardcoded 8080 claim). Fixed
    in PR #116. No change in this PR.
12. *(Bonus, from the issue thread, not one of the 18 — fixed while
here)*
    **`ci/check-consumer-check-extension.sh:32-33` — freeform `# Usage:`
comment tag.** Fixed — `ci/check-consumer-check-extension.sh:32`, now
    `# NOTE: usage — ...`, per the fix already applied to the identical
    shape in PR #151.

### `schemas/` (1)

13. **`schemas/page.schema.json:66` — `og_image` extension allowlist
inconsistent across schemas.** Fixed —
`schemas/page.core.schema.json:64`
    and `schemas/journal-entry.schema.json:52` (the file split into
    `page.schema.json` + `page.core.schema.json` since the issue was
filed; the actual `og_image` pattern lives in `page.core.schema.json`
now). Both now accept `.jpeg`, matching `faq`/`sizing-guide`/`product`
    schemas, which already did. All 5 schemas' `og_image` patterns now
    agree.

### `templates/` (6)

14. **`templates/atom.xml:25` — `<updated>` from newest-DATED entry, not
most-recently-EDITED entry.** Fixed — `templates/atom.xml:24-45`. Now
    scans every entry's own `updated`-or-`date` and takes the max
    (`set_global` + epoch-int comparison, since Tera's `>` only compares
    numbers). Verified against a real `zola build`: an older-dated entry
    given a newer `updated` now correctly wins.
15. **`templates/faq.html:36` — auto-slugify anchor collision.** Fixed —
    `templates/faq.html:42-59`. A colliding anchor (explicit or
    slugify-derived) gets a `-{{ loop.index }}` suffix instead of
    colliding on `id=`. Verified against a real `zola build`: two
questions that previously rendered two
`id="do-you-ship-internationally"`
    elements now render `id="do-you-ship-internationally"` +
    `id="do-you-ship-internationally-2"`.
16. **`templates/journal-entry.html:16` — hardcoded `og_type`.** Fixed —
removed the override; `journal-entry.html` now inherits `page.html`'s
    `og_type` block (`templates/page.html:13`), which already reads
    `page.extra.og_type | default(value="article")` correctly. Verified:
    a real build's `og:type` meta still reads `article` by default with
    no regression.
17. **`templates/partials/ld-article.html:21` — hand-concatenated
    image/logo URLs instead of `get_url()`.** Fixed —
    `templates/partials/ld-article.html:20-28`,
    `templates/partials/ld-product.html:27-32`,
`templates/partials/ld-organization.html:29-34` (the issue: "**three**
JSON-LD partials"). All three now call `get_url(path=...,
trailing_slash=
false)`, matching the same field's resolution in
`base.html`/`page.html`.
    Verified: with `base_url = "https://example.com/"` (trailing slash),
    hand-concatenation produced `https://example.com//img/og.png`
(double-slash, broken); `get_url()` produces
`https://example.com/img/og.png`.
18. **`templates/partials/ld-organization.html:14` — no
`assert::required`
guard, unlike every sibling.** Fixed —
`templates/partials/ld-organization.html:13,26`.
Now imports `partials/assert.html` and asserts `config.extra.logo_path`,
    matching every sibling `ld-*.html`. Verified: pre-fix, an absent
`logo_path` still failed the build, but with a bare Tera engine error
    ("Variable `config.extra.logo_path` not found in context") naming
    neither the field's role nor which template needs it; post-fix, the
    same missing field now fails with `partials/ld-organization.html:
missing required metadata` — the same clear, actionable message every
    sibling partial already gives.
19. *(the issue's `ld-organization.html:19` finding — same file, listed
separately below since it's a distinct defect)* **JSON-LD `</script>`
    breakout via unescaped `json_encode`.** Fixed — new
    `templates/partials/ld-json.html` (shared `ldjson::str()` macro,
    escapes `/` → `\/`, RFC 8259 §7-legal) wired into **all six**
    `partials/ld-*.html` (article, breadcrumb, faq, howto, organization,
    product) — not only the file the issue cited, since every sibling
    shared the identical `json_encode | safe` pattern with no escaping.
Verified: a field value containing
`"Hello</script><script>alert(1)</script>"`
    round-trips through `json.loads()` to the exact original string, and
    the literal substring `</script` no longer occurs anywhere in the
    `<script type="application/ld+json">` payload.

Note: fixing `ld-organization.html`'s `assert::required` call surfaced a
real cross-file Tera scoping gap — `base.html` (which `{% include %}`s
`ld-organization.html` inside a block every page reaches via `super()`)
did not itself import `partials/assert.html`, and Tera resolves a macro
namespace by which template an `include` tag is textually written in,
not
by the included partial's own imports, once reached via `extends` +
`super()`. Fixed by importing `assert` in `templates/base.html:20` too;
documented + verified (see commit message).

**18/18 accounted for**: 16 fixed in this PR, 2 already fixed on `main`
(items 2 and 11 above) with cited evidence.

## Negative fixtures

> Negative fixture: `bin/typikon-check` (`base_host()`) — watched
failing by
> piping a single-quoted `base_url = 'https://example.com'` config line
> through the pre-fix `sed` pipeline, which produced the literal garbage
> `base_url = 'https:` before the fix and `example.com` after (both
quote
> styles), run against the exact shipped `sed` expression from each
> revision.

> Negative fixture: `bin/typikon-validate` (root JSON Pointer) — watched
> failing by running `origin/main`'s own shipped `bin/typikon-validate`
> against a fixture page missing the required `title` field, which
produced
> `"pointer": "/"` before the fix; the current worktree's shipped
> `bin/typikon-validate` against the identical fixture produces
> `"pointer": ""` after.

> Negative fixture: `bin/typikon-validate` (`load_schemas()` crash) —
> watched failing by running `origin/main`'s own shipped
> `bin/typikon-validate` against a schema directory with
`faq.schema.json`
> truncated mid-object (invalid JSON), which produced a raw
> `json.decoder.JSONDecodeError` Python traceback (exit 1) before the
fix;
> the current worktree's shipped `bin/typikon-validate` against the
> identical corrupted schema produces a clean
> `{"error": "malformed schema ...: Unterminated string ..."}` (exit 2)
> after.

> Negative fixture: `bin/typikon-init` (swallowed refresh failure) —
> watched failing by extracting the exact shipped refresh line from
> `origin/main`'s `bin/typikon-init` (`(cd themes/typikon && git pull
> --ff-only origin main 2>/dev/null) || true`) and running it verbatim
> against a `themes/typikon` checkout whose `origin` remote is
> unreachable, which produced silent success (exit 0, empty stdout AND
> stderr) before the fix; the current worktree's shipped replacement
lines
> against the identical unreachable remote produce exit 0 (still
> non-fatal, correctly) but with `warn   themes/typikon: refresh failed,
> continuing with existing checkout: fatal: ... does not appear to be a
> git repository ...` on stderr after.

> Negative fixture: `templates/atom.xml` (`<updated>`) — watched failing
> by building `examples/sample-blog` (real `zola build`, shipped
> templates) with `origin/main`'s `templates/atom.xml` against a fixture
> where the older-dated journal entry has a newer `updated` than the
> newer-dated entry's own value, which produced
`<updated>2026-02-10T00:00:00+00:00</updated>`
> (the newer entry's own date, ignoring the true most-recent edit)
before
> the fix; the current worktree's `templates/atom.xml` against the
> identical fixture produces
`<updated>2026-08-10T00:00:00+00:00</updated>`
> (the actual most-recent edit) after.

> Negative fixture: `templates/faq.html` (anchor collision) — watched
> failing by building a FAQ page (real `zola build`, shipped templates)
> with two questions that slugify identically, against `origin/main`'s
> `templates/faq.html`, which produced two
`id="do-you-ship-internationally"`
> elements before the fix; the current worktree's `templates/faq.html`
> against the identical fixture produces
`id="do-you-ship-internationally"`
> + `id="do-you-ship-internationally-2"` after.

> Negative fixture: `templates/partials/ld-organization.html`
> (`assert::required` + `get_url()`) — watched failing by building
> `examples/sample-blog` (real `zola build`) with
`config.extra.logo_path`
> removed from `config.toml`, against `origin/main`'s
> `templates/partials/ld-organization.html`, which produced a bare Tera
> engine error (`Variable 'config.extra.logo_path' not found in
context`)
> before the fix; the current worktree's version against the identical
> fixture produces `partials/ld-organization.html: missing required
> metadata 'config.extra.logo_path'` after — matching the message shape
> every sibling `ld-*.html` partial already uses.

> Negative fixture: every `partials/ld-*.html` (JSON-LD escaping) —
> watched failing by piping `"Hello</script><script>alert(1)</script>"`
> through the pre-fix `value | json_encode | safe` pattern (shared,
> unescaped, across all six partials), which left the literal substring
> `</script` intact in the `<script type="application/ld+json">`
payload;
> the current worktree's `partials/ld-json.html::str()` macro against
the
> identical input produces `Hello<\/script><script>alert(1)<\/script>` —
> `json.loads()`-equivalent to the original string, with `</script`
> nowhere in the byte stream. All 16 real JSON-LD blocks rendered across
> both `examples/*` sites (blog + shop, every template) were re-verified
> to still `json.loads()` cleanly after the change.

All ten fixtures ran the actual shipped file content at each git
revision
(`origin/main` vs this branch) — extracted verbatim (`git show
origin/main:<path>`) or symlinked directly at the theme root for a real
`zola build` — never a re-implementation.

## Verification

- `kanon lint <path> --all` — zero violations across all 21
changed/added
  files (individually checked).
- `python3 bin/typikon-validate examples/sample-blog` /
  `examples/sample-shop` — both `{"failed": 0}` against the updated
  schemas.
- Real `zola build` + `zola check` — both example sites build and check
  clean.
- `ci/check-xml-output.sh`, `ci/csp-enforce.sh` — both example builds
pass.
- Every `ci/check-*.py` / `ci/check-*.sh` regression script in the repo
  (triad-schema, consumer-schema-registry, page-template-cascade,
  font-coverage, release-config, asset-provenance, favicon-path,
  init-favicon-path, asset-path-existence, init-staging-scope,
  control-contrast, home-heading, content-heading-collision,
  interactive-contrast (+selftest), workflow-template,
consumer-check-extension, fixture-corpus-exemption, deploy-bundle-gate,
  cf-deploy-gate) — all pass.
- `TYPIKON_CHECK_MODE=dev bin/typikon-check examples/sample-blog` and
  `.../sample-shop` — every stage passes except `pa11y` (`pa11y-ci`
  genuinely absent on this box; `lychee` and `playwright-smoke`, both
  present, pass real link-checking and browser smoke tests against the
  fixed templates).

Not run: GitHub Actions CI (this box does not run cargo/full builds per
fleet policy; this repo's own CI is zola/python/node-based and this PR's
branch push will trigger it — this repo is public, so Actions minutes
are
unmetered).

Closes #92

Co-authored-by: forkwright <cody@forkwright.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant