tests: markup, framework-class, and visual baselines for chrome partials - #2719
Merged
chalin merged 41 commits intoAug 17, 2026
Merged
Conversation
This was referenced Aug 15, 2026
- Characterizes the breadcrumb's rendered markup (deep, mid, and single-crumb variants) before the semantic-class migration touches it, so class swaps review as golden diffs - Shares one fixture spec between the test and the update:markup-goldens refresh lane, mirroring the docsy.dev md-output golden shape
- Asserts partials on the cleared list emit only Docsy-owned (td-) classes, with the Bootstrap inventory derived from the installed bootstrap.css rather than a hand-kept list - Starts with an empty cleared list; each migration PR adds its partial red-first and drives it green with the class swap - Self-tests the scanner against synthetic dirty/clean templates so an empty list can't false-green
- Adds a Puppeteer + pixelmatch suite under node:test (test:visual, own suite like test:lychee): element crops of fixture-site chrome regions, desktop+mobile x light+dark, against committed platform-keyed goldens; catches CSS-side regressions the markup goldens can't see - Keeps installs script-free: puppeteer's postinstall is denied via allowScripts (audit updated); install:chrome fetches the pinned browser on demand through the dep's own entry point (npm withholds bin links for script-denied packages) - Signal proven red-first: a divider recolor in _breadcrumb.scss fails all four shots at 16 differing pixels
- Adds a visual job to test.yaml (install:safe, on-demand browser, test:visual), uploading tmp/visual/ diffs as an artifact on failure - First run is deliberately red: no Linux goldens are committed yet; the run's actual-shot artifact seeds them, proving the pipeline end to end - Shields the fixture-site markup goldens from Prettier: they mirror Hugo output byte for byte
- Captures the authoritative golden set from the visual job's first (deliberately red) run, closing the red-to-green proof of the pipeline
…latforms - Pads region crops (an element shot stops at the border box, so margin regressions were invisible) and adds a full-viewport page shot per scheme/viewport as the coarse net for untracked regions - Skips the suite on platforms without a golden set (Windows: best-effort, opt in via a local uncommitted baseline; goldens/ gitignored except linux + darwin) - Adds update:visual-goldens:linux, a one-command Linux-golden refresh from the red visual job's artifact, for devs without a Linux machine
- First run of the update:visual-goldens:linux flow, seeded from the red visual job's artifact
- Populates the golden fixture's chrome (main-menu entries incl. an external link, footer links with FA icons, copyright, second docs section): a bare skeleton under-represented the regions the semantic-classes migration will touch - Makes the safety-net page shot full-page rather than viewport-clipped, so the below-the-fold footer is covered
- Framework-class scanner: action-aware attribute scan (quotes inside {{...}} truncated the old regex, hiding classes after them) that also tokenizes string literals inside class-attribute actions (delimit/cond-built lists) and matches case-insensitively; the defeating forms are now self-test cases, added red-first
- Visual suite: a missing golden set fails on Linux/CI instead of skipping (all-skipped exits 0, so goldens could vanish with CI green); a bijection test pins golden files to the shot list; wiring guard now covers the own-suite scripts' globs (test:visual, test:lychee)
- Clip geometry: clamps each crop edge against page bounds (origin-only clamping shifted crops near edges, and Puppeteer accepts overflow silently); adds mid-depth breadcrumb crops and a docs/ page shot so the __single variant has a visual home
- Hardens serveDir against dot-segment escapes; filters the goldens-refresh run query to completed runs; fixes README full-page wording and a stale script name in the audit comment
- Renames the browser-install script: "chrome" already names the shared UI regions in this repo (chrome build modes, tests/fixture-site/chrome-*) - Adds a CC BY 4.0 link to the fixture copyright, exercising the partial's Markdown-rendering path
- Framework-class scanner: flags edge-hyphenated literal fragments in cleared partials (concat-assembled names — print "d-" "flex", d-{{ .Bp }}-none — render whole but scan as fragments), self-tested red-first; attribute match anchored so data-class and kin are not scanned; whole-name evasion via replace/printf stays review's job, documented
- Visual compare: pixelmatch threshold 0 (0.1 silently swallowed per-channel shifts up to ~40/255 — theme-token drift territory); bit-exact determinism verified by consecutive runs; README claim aligned
- Visual suite refuses update mode under CI (writing goldens instead of comparing exits 0 — a silent bypass of the authoritative net)
- serveDir jail resolves symlinks (realpath) before the root check; goldens-refresh query narrowed to failed runs
- Pins term.html's class-coupled breadcrumb post-processing with a taxonomy-term fixture golden (replaceRE strips aria attributes and the active class by name, so a class swap in the partial silently no-ops the caller; the golden makes that break visible)
- Framework-class scanner, red-first: fragments are now inventory-anchored (flagged only when completable to a Bootstrap name, so ul-{{ $n }} stays clean), printf placeholder formats (d-%s-none) match against the inventory, and a lone "-" delimiter literal emits the joined name (delimit (slice "d" "flex") "-")
- Supply-chain audit pins install:browser and asserts its pre/post hook siblings stay absent (npm would run them unreviewed around the pinned command)
- Visual suite: update refusal also keys on GITHUB_ACTIONS (CI is workflow-mutable; GITHUB_-prefixed vars are not) and an after-hook asserts every shot comparison executed on the authoritative platform (skip-quarantined tests defeat the filename bijection)
- serveDir jail re-checks after directory-index resolution (the index itself may be a symlink); harness comment corrects the webfont claim (same-origin fonts load and are awaited)
- Framework-class scanner composes action literals the way Hugo would, red-first: printf verbs (indexed included) substituted with literal args in order — unresolved verbs stay as inventory-matched patterns, so td-prefixed formats stay clean — and piece literals joined with present delimiter literals, else with "-" and "" (a variable-held delimiter is invisible) - Supply-chain audit denies workflow run-steps and env entries touching the visual update lane (UPDATE_VISUAL_GOLDENS, update:visual-goldens, CI=/GITHUB_ACTIONS= shadowing): the in-suite refusal can't resist a step that rewrites its own environment, so the committed-workflow audit is the right boundary - Adds a taxonomy-term page shot: _taxonomy.scss couples article-teaser styling to breadcrumb classes, a mapping only a term-page visual can see regress - Visual suite builds its own fixture dir (sharing markup-golden's raced a concurrent test:repo run); README webfont claim and goldens-refresh comment corrected
- Framework-class scanner enumerates, red-first, instead of emulating Hugo: if/else/end structures in class attributes become branch alternatives and printf verbs (indexed, flagged, width/precision) substitute by enumeration over the action's literals, so whatever those literals can render is in the scanned set; enumeration is capped and an over-cap attribute throws — fail closed beats silent under-scan - Supply-chain audit positively pins the visual job's run sequence (install:safe, install:browser, test:visual, unconditional): the deny rules can't see a dropped or if-guarded comparison step - Visual suite acquires server then browser sequentially, retaining each as it settles: a browser-launch rejection stranded the already-listening server and hung the run past any exit
- Extracts the update-lane denies and the visual job's positive step pin from the supply-chain audit into tests/workflow-lint.test.mjs: the audit owns install/provenance invariants, this file owns check-execution integrity — different subjects over the same workflow files - Restores the audit's shared env screen to its pre-existing key set (it also feeds the Netlify screen, where the visual keys are meaningless)
- Documents the provenance guarantee reader-facing (it lived only in code comments): goldens are written by a real Hugo build of the checked-out theme and re-compared against a fresh build every run, so they can't silently drift from real output - Retitles the fixture-site README to cover the whole suite, with the CCR case registry as a section (all inbound references are file-level)
- Framework-class scanner composes the class attribute's whole literal pool (ordered pairs/triples joined by its delimiter literals), red-first: an over-approximation covering any value flow that rearranges quoted literals — range/with dot output, nested joins, cross-action assembly — without modeling each pipeline; pool capped, over-cap throws - Cleared partials now carry a static child-partial closure: each statically-called child must be cleared too or listed as visible staging debt (PENDING_CHILDREN) — a parent's rendered output includes its children's classes, so source-only scanning couldn't back the "emits no framework classes" claim (navbar → theme-toggler is the live example) - Workflow lint pins the visual job itself: no job-level if or continue-on-error (either skips or error-masks the whole job before any step runs), runs-on pinned to the authoritative platform - Anchors workflow-lint.test.mjs in the cross-root suite anchor, so deleting the guard file goes red from the scripts glob
- Moves goldens to goldens/PLATFORM/REGION/VIEWPORT-SCHEME.png (owner call): the set grows ~4 files per platform per migrated region, and region subfolders keep it browsable and group PR diffs naturally - Byte-identical git mv, so the compare stays green without a golden refresh; bijection walks recursively; the linux updater maps artifact shot names onto the nested paths
- DRY/lean review round: five facts (platform policy, all-skipped-exits-0, pool composition, review's-job scope, determinism mechanics) had accreted 3-5 statements each across the adversarial rounds; each keeps its one home (README for policy, the implementation site for mechanics) and the rest become pointers or go - Prunes change-log narration (round tags, dates) from self-test comments; homes the guard-family map at suite-anchor's pinned list - Adds the one missing why: the visual CI job sits apart from the OS matrix (browser install, Linux-only enforcement)
- Reworks em-dash glyphs in branch-added prose to construct-matched marks (colons for glosses and elaborations, sentence breaks); pre-existing CCR-registry text untouched - Trims restated-adjacent-text tells in the visual README (the visual job named twice, gitignored beside "uncommitted", the platform-keying double-statement)
chalin
force-pushed
the
chalin-m24-semantic-baseline-2026-0815
branch
from
August 16, 2026 15:58
4c15fce to
25c95e7
Compare
- Resolves the run's repository from the git remotes (upstream, else origin): PR workflows run in the base repo, and gh's autodetection needs an interactive set-default in fork clones, so the documented refresh silently queried the wrong repository for contributors - Selects the completed run matching HEAD instead of the branch's latest failure, which could predate an in-flight push and baseline stale shots
chalin
marked this pull request as ready for review
August 16, 2026 16:28
- Looks the HEAD run up directly (gh run list --commit) instead of scanning the branch's recent runs, which could miss it on a busy repo - Comments the deliberate 404 collapse in serveDir's error path and the job-id-as-contract coupling in the workflow lint
- Pool composition joins every in-order subsequence instead of capping at three pieces: Bootstrap has four- and five-segment names (text-decoration-line-through, link-underline-opacity-0-hover) that the depth cap silently under-scanned; the modeled flows preserve document order, so subsequences cover them (red-first) - partialCalls extracts static calls anywhere in an action (assignment, with/if expressions) and raw-string names, not just action-initial double-quoted forms (red-first) - Pins the fixture copyright to_year: the default tracks the wall clock, which would roll the footer and every committed page shot each January 1 (2027-clock build verified unchanged) - Repo regex keeps dotted repository names (docsy.dev) whole, stripping only a terminal .git; README's skip wording carries the CI exception
- Scanner fails closed on order-changing collection operations in class-attribute actions (append renders its element after the collection, maps range in key order): the in-order-subsequence premise doesn't hold there, red-first - partialCalls lexes real actions only (template comments and plain text can name partials without calling them), recognizes the namespaced partials.Include forms and parenthesized literal names, and keeps explicit non-html extensions, red-first - Supply-chain audit pins Puppeteer's config search surface empty (executable rc/config files, package.json puppeteer keys): the pinned install entry point loads them, so they can redirect the browser download or swap the launched executable; workflow lint denies PUPPETEER_* env and env-shadow prefixes the same way - Fixture builds strip all inherited HUGO_* env (a shell or workflow HUGO_PARAMS_* would silently change what the goldens represent); tests re-add overrides explicitly - Updater accepts trailing-slash, ssh-port, and case-varied GitHub remote forms
…swap - Adds a blog post to the fixture with a markup golden and page shots: blog/baseof wraps the same breadcrumb partial in a different layout context - Adds a print-media page shot (harness gains media emulation): _breadcrumb.scss hides breadcrumbs in print, a rule no screen shot can see regress
- Rewrites two comment asides that used literal em-dash glyphs - Trims golden-provenance restatements to the README home - States the duplicated-walk stance inline; the old pointer resolved to an unrelated comment - Documents that a size-mismatch failure writes only the actual PNG
- Action lexing is now string-aware in both the attribute lexer and partialCalls: a `}}` inside a Go-template string literal ended the action early, hiding classes and child calls behind it - The order-changing-op guard also matches the namespaced collections.* forms (collections.Reverse evaded the bare-alias regex) - The Puppeteer config-surface audit derives the search-place list from the installed loader, covering the six .config/ paths it missed - The comparator counts antialiasing-classified pixels (includeAA): pixelmatch's default AA detection passed non-identical images at threshold 0 - The workflow visual-job pin denies working-directory redirects, which could re-point the pinned npm commands at a no-op package.json All fixed red-first, with the round's attack templates as self-tests.
- The attribute regex admits `}` before class=, catching attributes
emitted whole from a branch ({{ if .X }}class="…"{{ end }} — live in
pager.html, whose btn/btn-primary/disabled were invisible)
- A stray else/end inside an attribute fails closed: control flow crossing
the attribute boundary silently dropped the remaining segments
- Comment actions are modeled as empty, not a space: comments fuse the
text around them (d{{/* … */}}-md-{{/* … */}}none renders d-md-none)
- Middle fragments (-md-, -decoration-) match the inventory by inclusion;
edge-anchored startsWith/endsWith both miss them
- Star width/precision printf verbs (%.*s) join the verb grammar; they
were treated as plain literals and skipped pattern checks
- The order-op guard also denies Scratch/Store sorted-map assembly
(SetInMap/GetSortedMapValues sort by key without any denied token)
All fixed red-first, with the round's attack templates as self-tests.
The built fixture's HTML is ground truth for what the theme emits, with no template semantics to re-model: rounds of scanner hardening all fixed divergences from what Hugo computes anyway. The new net asserts cleared regions render no Bootstrap classes (CLEARED_REGIONS, empty like the scanner's list) over a site-wide census anchor, red-proven with a planted breadcrumb region and a broken inventory. The two nets' blind spots are disjoint: the output net misses branches the fixture doesn't exercise (the scanner enumerates those, failing closed); the scanner misses what it can't lex (the output net renders it). Scanner hardening is frozen accordingly: new lexer-evasion findings are dispositioned as covered by the output net (note in the scanner header). The Bootstrap inventory moves to tests/lib/bootstrap-inventory.mjs, shared by both nets.
- Class extraction widens to any quoting/case (case-insensitive, single- quoted, unquoted), and the census asserts every emitted class= parses: quote drift in a migration PR can't silently blind the net - CLEARED_PARTIALS moves to a shared lib and the output net enforces the ratchet: every cleared partial maps to a region or a documented exemption, so scanner-invisible forms can't ship output-unchecked - Cleared regions check every regex instance across every listed page, not the first match on one page - The workflow lint pins the visual job's reachability: no needs gating, and the pull_request trigger stays present and unfiltered - The scanner freeze note names the known-invisible form (class strings hoisted into assignments) with a live in-tree example; the visual README documents the JS-driven-state limit and its interaction-shot remedy All red-proven: planted unmapped cleared partial, regressed extraction regex, needs edge, filtered trigger.
- The output net decodes character references before inventory matching: an encoded name (d&google#45;flex) styles as d-flex in the browser but passed raw-text comparison in both nets - The cross-net ratchet's partial→region binding is machine-checked: a second build stubs every cleared partial to a sentinel (site layouts override theme layouts), and each region must surface its partial's sentinel or stop matching — a stale or misbound regex can no longer satisfy the ratchet with an unrelated clean element - The workflow lint pins the visual job's full step sequence, action steps included, with each step's `with` keys: an inserted action or a checkout ref/repository/path override (which would test main instead of the PR) now fails; the diffs upload stays the one failure-gated step - The hook-sibling denial extends to the visual lane (test:visual, test:repo, is:clean, the update scripts): a smuggled pretest:visual could strip CI env and rewrite goldens before the pinned comparison - The visual job ends with is:clean, so any in-workspace golden rewrite is loud regardless of how it happened All red-proven: encoded-class self-test, misbound title region, checkout ref override, inserted action step, planted pretest:visual hook.
Round 12 parked them in the supply-chain audit because the hook loop lived there, but they are check-execution subjects, not install provenance: the audited split (workflow-lint header) puts script-layer execution integrity with the wiring guard. Install-path hooks stay in the audit. test:lychee joins the denied list (same lane, same hole).
- The scanner's review's-job caps never named unquoted attributes, though the freeze's disposition rule requires stated caps; named, with the output-net split - The step-pin assert message claimed the exact reviewed sequence while checking with-keys, not values; message now says what it checks
The header and test name claimed resistance to deliberate subversion, a stronger claim than the guard backs (or needs): it is a lint over careless workflow edits, and deliberate subversion is diff review's job. Wording only; every assertion is unchanged.
chalin
force-pushed
the
chalin-m24-semantic-baseline-2026-0815
branch
from
August 17, 2026 18:58
b5ca7c7 to
8d20511
Compare
This was referenced Aug 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
td-class migrationterm.html's class-coupled breadcrumb post-processing); refreshed via theupdate:markup-goldenslanebootstrap.css; catches literal tokens, class-name fragments assembled in template actions, and printf placeholder forms; the cleared list starts empty and each migration PR adds its partial red-first, with scanner self-tests preventing a false-green empty runtests/fixture-site/output-classes.test.mjs): cleared regions render no Bootstrap classes in the built fixture's HTML; complements the template scanner (blind-spot split stated in the file header)test:visual, new Linux CI job): Puppeteer + pixelmatch element crops and full-page safety-net shots against platform-keyed goldens; platform policy and refresh flows intests/visual/README.mdpuppeteer(postinstall denied viaallowScripts, audited; the pinned browser installs on demand throughinstall:browser),pixelmatch,pngjs