Night Watch Component Auditor

Night Watch — Component Auditor

Assigned to: Cindy's Navi (cixzhang)

Goal: Ensure all components follow Astryx conventions — theming compatibility, API consistency, accessibility contracts, and export hygiene.

Authority and process: Product behavior is governed only by committed repository records whose front matter says authority: current. Start with the component spec, then follow its applicable module, family, design, architecture, theme, and AST relationships. The wiki owns audit procedure, practical guidance, scoring, and the ledger lifecycle; it does not replace those product records.

⚠️ Keep in sync. When a durable product requirement changes, update its canonical repository owner first, then update the rubric procedure or examples when needed. Do not add a reusable product rule only to the wiki.

---

Why This Role Exists

Component quality regresses silently. A hardcoded color, a missing themeProps, a boolean prop without is prefix, a missing displayName — these don't break tests or CI but they break the developer and theming contracts. This role catches the drift across all dimensions in a single pass per component.

---

Scope

Activation status: spec:AST-029 is current and accepted. Its observational backfill, mode separation, TDD, storage, scoring, and eligibility-report decisions govern this procedure. FR11 still keeps unattended audit-eligibility auto-merge inactive until a separate atomic phase: shipped activation lands the canonical roster and package paths, shared prompt, trusted exact-head eligibility check, focused tests, and this procedure together. Current audit PRs instead clear trusted Review Loop and normal-merge immediately when every check is green.

Each nightly run takes two components sequentially from the canonical queue and runs a full audit of each against the Component Audit Rubric. Complete and record the first component before refreshing the queue for the second. Each component gets its own isolated worktree, branch, pull request, evidence, and ledger result. The run drives a real browser, fixes what it safely can, re-audits to prove the fixes, and records the result. If the first audit cannot be completed and recorded, do not start the second.

Treat each component as a complete audit, not a skim. A pass that only runs the mechanical checks finds drift a linter could have found; the expensive half is observable behavior, accessibility, RTL, and rendered evidence. Two components is a throughput target, never permission to reduce either audit's evidence quality.

The checks themselves are not restated here. The rubric is the single source for what is checked, how it is verified, and how it is scored. This page owns only the nightly mechanics: how the component is chosen, how fixes are made and proven, and how the result is recorded.

The score ledger — component-scores.json in this wiki — is the only record of progress. There is no separate state file, no queue kept on the side, no list of completed components. What has been audited, when, under which rubric version, and what is still open — all of it lives in the ledger, and the ledger is what the next night reads. If it is not in the ledger, it did not happen.

---

The nightly run

0. Prepare the isolated M5 worktree

Run Night Watch only on Cindy's M5. From each audit worktree, install dependencies exactly once with the repository's noninteractive fast path:

python3 "$HOME/astryx/tools/fast-install.py" "$PWD"

Do not run pnpm install first. The fast installer enforces noninteractive pnpm mode itself; callers do not select an interactive mode. If the fast path fails, report the exact failure before selecting a fallback. A successful identical-lockfile APFS clone plus reconciliation should take seconds, not minutes.

1. Pick the component

Read the ledger—component-scores.json in this wiki—and pick against the live roster generated from every component-bearing package registered in scripts/component-packages.cjs (currently Core, Lab, Charts, Rich Text, and Vega). The order is never-audited first, then oldest-audited, then lowest-scoring, so coverage grows before anything gets re-audited and the weakest components come back around soonest. Take the first one.

node scripts/score-ledger.mjs --queue --limit 1

Already audited today? The ledger's lastAudited says so. Stop; do not run twice.

2. Establish the observational component contract

Read the nearest current component contract and every applicable current public module contract, then follow their links to current family, design, architecture, and system authority plus objective standards. Apply the rubric procedure last. Discover the rubric's current version from the wiki at run time and record it in the scorecard; never copy a version from an old prompt, PR, ledger row, or prior audit. The version identifies procedure/scoring only. Every product finding must cite an applicable authority: current repository clause or objective external standard. A rubric check with no such owner cannot invent a requirement; record it as a rubric or authority gap and do not score it against the component.

Start with a bounded context set:

• the component source, consumer docs, public types/barrels, current contract, and focused tests;
• only the applicable excerpts from linked family/design/architecture/system records;
• the published declaration and package-export stanza when release status matters;
• changed-file metadata and relevant hunks from overlapping open pull requests.

Do not preload historical ledger examples, broad repository history, generic Storybook examples, RTL harness implementation, probe-kit internals, or whole package manifests. Read those only when a concrete requirement or evidence gap needs them. A broad search result is a lead to narrow, not context to ingest wholesale.

For Night Watch, always backfill a missing canonical component contract as a draft observational record in the audit PR, using the repository's component-specification template and current schema. When an existing contract is incomplete enough that the closed public-surface inventory cannot map to it, complete that observational draft in the same PR. Other audit modes may continue from applicable current authority and checkable evidence without preparing a draft, as spec:AST-029/FR3 permits.

The draft cannot settle policy or make the PR auto-merge eligible. Record only verified shipped behavior; do not add, improve, remove, or reinterpret behavior. Use the closed public-surface evidence inventory required by spec:AST-029/FR5 to make gaps visible for owner review.

The reviewer evaluates the backfill as an observational record, not a product proposal. They verify every normative claim against current implementation, tests, consumer docs, and applicable current authority; reject any new behavior, default, compatibility promise, ownership boundary, or subjective design decision; and grant exact-head spec approval only when the draft faithfully describes current shipped behavior.

Implementation defines component-local facts. Applicable current shared records and objective standards override conflicting implementation; those conflicts are audit findings to fix. A completed backfill becomes current only after exact-head owner approval. Any unresolved public API, compatibility, ownership, or subjective design decision remains recorded and keeps unattended audit eligibility false; it does not require another human after trusted Review Loop clears a non-breaking audit PR.

3. Audit it—the full rubric

Work Component Audit Rubric end to end, as a full component grading rather than a mechanical sweep. That means the automated checks and the judgement ones, and the screenshots: build Storybook, drive real Chromium, and capture every state the component has in light and dark. The rendered-design half is graded from pixels or it is reported not measured — never guessed, never scored zero.

Write the scorecard before changing a single line. Preserve the raw baseline test, a11y, RTL, browser, and sensor reports immediately; they are the before half of the night's evidence and must not be reconstructed after fixes.

Storybook fixture budget

Use Storybook as the reusable browser fixture surface, not as a permanent copy of the audit matrix. Start from existing stories. Drive one story through several states with Playwright, args, or deterministic setup when the underlying component composition is the same; multiple receipted screenshots may cite the same story id. Combine variants or sizes into one comparison story when comparison is the value.

Add a checked-in story only when a materially distinct rendered configuration, composition, interaction, viewport/theme setup, or recurring a11y/RTL/visual check is otherwise unreachable. Do not add one story per prop, input partition, unit test, audit row, screenshot, or sensor receipt. Do not add play merely because the audit drove an interaction; keep it only when that interaction should remain an ongoing Storybook regression check. Prefer the smallest fixture set that preserves truthful, repeatable evidence.

RTL applicability is mandatory

Run pnpm rtl:audit -- --filter <Name> for every component, new or existing. Do not call 0 pass / 0 fail / all N-A clean. Read the component source and every story for directional icons, contextual separators or connectors, positioned affordances, control order, scrolling or dragging, overlays, and horizontal keyboard behavior. Then leave the component in exactly one state:

• measured: an applicable automatic or curated dimension ran;
• verified N-A: add a specific core/Name or lab/Name reason to apps/storybook/rtl-audit/verified-not-applicable.json;
• coverage gap: the harness has no dimension for behavior you found.

A coverage gap is a system-level finding. Link #5364 rather than filing one issue per component, add or extend the reusable detector/target, and re-run the component. Never invent a target that measures unrelated behavior just to turn N-A into pass. A verified-N/A entry is not an allowlist: remove it if later behavior makes an RTL dimension applicable.

For text decorations, context decides applicability. A slash in prose is neutral; an aria-hidden slash between repeated breadcrumb items communicates progression and D6 measures it. Unicode-mirrored angle quotes must not also receive a CSS flip; arrows and contextual slashes require one mirror or a glyph swap.

Chrome's CDP accessibility tree does not reliably expose aria-current. Verify the DOM attribute and the element's computed or implicit role separately; absence from the AX tree alone is unverified, not a failure.

4. Fix what you can

Fix the findings that have a clear right answer—a raw value that should be a token, missing behavior coverage, a missing stable visual-regression snapshot, consumer-doc drift, an unguarded hover, or an implementation bug against a current shared contract or objective standard. The same audit PR may contain the draft observational worksheet and these objective fixes. Work in one branch: night-watch-component-audit/<Component>/YYYY-MM-DD.

For every audit-authored behavior test, follow Matt Pocock's public tdd skill at the commit pinned by spec:AST-029/DEC-4: derive the seam from current authority or an established public interface, prove the test fails before production changes, and implement one minimal observable vertical slice. Reject implementation-coupled, tautological, and horizontal test generation. Do not run an open-ended adversarial pass or turn a novel edge case into a product requirement; B12 was retired in rubric 1.16 for that reason.

Leave alone anything that needs a decision rather than a correction: a naming dispute, a new prop, a changed API shape, a visual judgement call, a system-level design question. Those are not yours to settle in a fix pass — carry them in the scorecard and say so in the PR's Needs Review section. The Component Lifecycle routing rules say where each kind goes.

If a finding is real but you cannot fix it safely, that is fine. Leave it in the ledger as an open blocking finding with a one-line explanation. An honest open finding is worth more than a rushed fix.

A pre-existing BLOCK describes component health; it does not automatically block the audit PR. Human-reviewed landing may proceed when the PR records the finding accurately, contains no breaking change, does not introduce or worsen the violation, keeps breaking, unsafe, or judgment-dependent remediation outside audit scope, and otherwise passes exact-head review and CI. An invariant violation remains a violation: do not relabel it as an exception, waiver, or accepted behavior merely because fixing it would be breaking. The BLOCK still caps the component grade and makes unattended auto-merge ineligible. A new or worsened BLOCK introduced by the audit PR must be fixed or reverted before landing.

5. Re-audit, freeze the source head, and prove it

Run the audit again on the fixed component. Before the first push, run the focused affected suites and the relevant package, Core, and Storybook typechecks; run knowledge validation when a contract changed; run strict lint and the relevant package/Storybook builds once. Fix those failures before committing.

Commit the source head, then freeze it while capturing final browser evidence. Bind every receipt and report to that real commit. If any source, spec, story, or test changes afterwards, create a new head and regenerate the affected exact-head evidence. Do not repeatedly run the repository-wide pnpm test on the managed Mac when known unrelated platform/baseline failures remain; focused local suites prove the changed surface and exact-head Linux CI supplies the repository-wide test signal.

• Capture the screenshots again, so you have a genuine before/after pair for anything visual.
• The score must move. If you fixed something blocking and the score is unchanged, then either the fix does not do what you think or the first audit was wrong. Find out which — do not record a number you cannot explain.
• If a fix made something else worse, revert it. A nightly pass must never leave a component worse than it found it.

6. Open the PR, clear Review Loop, and merge

Publish it ready for review, not as a draft. Use exactly this title format:

[Component Audit] <Component>: <short user-impact summary>

[Component Audit] is the single audit tag. Do not substitute [Night Watch], a Conventional Commit prefix such as fix(...), or a generic title such as "complete audit". The summary names the concrete improvement a component consumer receives; if the PR is observational only, say what current behavior/evidence becomes trustworthy.

The description carries:

• the component package and exact repository/spec heads;
• the before and after scores, and what moved;
• the observational draft and its known evidence gaps, when one was prepared;
• before/after screenshots for anything visual;
• the findings fixed, each citing its governing requirement; and
• a Needs Review section for everything deliberately left unresolved, and why.

After pushing the frozen source head, wait for exact-head CI and fix only failures caused by the audit change. If the head changes, invalidate and regenerate affected browser evidence before final reporting. Do not run redundant local full-suite cycles while the same exact-head Linux CI signal is already in flight. Three source-fix iterations, then stop and report the unresolved failure.

Route the frozen exact head through the repository's trusted Review Loop. Clearance requires all of the following, read back from GitHub on the same exact head:

• a guarded, publicly submitted APPROVED review with its URL and commit id;
• review-required=success whose status creator is the trusted GitHub Actions App 15368, not a user or another bot; and
• the status target/provenance binding that exact public review.

A private or settled review session, draft verdict, unprojected bot review, or user-authored success context is not clearance. Do not request or wait for another human after this public trusted clearance. A stale or request-changes verdict requires correction and a fresh exact-head guarded review.

As soon as trusted Review Loop clearance, exact-head spec approval, and every CI/status check are green, immediately perform the normal guarded squash merge under the standing landing authorization. A red check is never ignored merely because branch protection marks it non-required. First fix a PR-caused failure; for an inherited or shared-tool coverage failure, repair the owning reusable harness in an atomic linked PR and rerun the audit PR to green before landing. Do not pause for another review request, ask the user whether to merge, enable auto-merge, or use an administrator bypass. Verify the landed commit contains exactly the reviewed audit changes. An inherited pre-existing component BLOCK does not prevent this landing; an unexplained red check, a new BLOCK, or a worsened BLOCK does.

Do not write the final post-fix ledger/tracker row while the pull request is open. The scorecard and exact-head report in the PR are the pending result, not current main.

Do not enable unattended auto-merge while spec:AST-029 remains phase: accepted. Emit the versioned machine-readable eligibility report required by FR10 anyway: component/package, audit mode/rubric version, repository and contract heads, closed evidence inventory, unresolved objective and manual gaps, remediations with before/after evidence, approval states, check states, and a fail-closed verdict with reasons.

After the separate atomic FR11 activation changes the spec to phase: shipped and lands the trusted exact-head audit-eligibility check, unattended auto-merge additionally requires all of these on the current head:

• any component-spec backfill describes only verified shipped or remediated behavior and has exact-head spec-owner approval;
• every public concept and reachable state has a complete evidence-inventory row;
• the PR changes no public API, default, or compatibility/migration promise;
• no ownership conflict or conflict between current records remains;
• it makes no subjective representation, proportion, density, or interaction-feel decision;
• every remediation is an objective test, stable visual snapshot, doc-drift fix, or bug fix with before/after evidence;
• no unrelated cleanup is present; and
• every required approval/check and the trusted audit-eligibility status is green.

A missing, stale, inconsistent, or ineligible report stays in trusted Review Loop until corrected. Once trusted Review Loop clears the exact head, no additional human review is required; follow the immediate normal-merge sequence above.

7. Update the audit tracker after landing

Only after merge and landed-byte verification, record the post-fix result in component-scores.json. Set the scorecard's commit to the landed squash/merge commit, not the pre-merge pull-request head:

git clone https://github.com/meta/astryx.wiki.git /tmp/astryx-wiki

<landed scorecard JSON> | node scripts/score-ledger.mjs --record <Component> \
  --from - --ledger /tmp/astryx-wiki/component-scores.json --push

--push commits and pushes the wiki and prints the commit URL. Pipe the scorecard instead of leaving a second stored copy. If an interrupted or older run wrote a pre-merge row, replace it immediately with the landed commit; a PR head is never the final current-state tracker value.

Verify the pushed row contains the landed commit, post-fix score/grade, retained findings, rubric version, mode, PR/evidence links, and audit date. A section not verified remains not_measured, never guessed or scored zero.

The audit is complete only after this tracker write succeeds. Refresh the canonical queue—and start the second nightly audit—only after the first component is merged, landed bytes are verified, and its landed ledger row is confirmed. If review, merge, or tracker update is blocked, report that blocker and do not claim or start the next component.

Do not file ordinary per-finding issues. Findings not fixed live in the ledger row against the component. Only a genuinely system-level problem affecting many components gets one shared issue.

---

Does not do

• File hardening issues. Unfixed findings live in the ledger row, not in a ticket queue. Only a genuinely system-level problem gets an issue.
• Keep its own state. The ledger is the record of progress — there is no state file to update.
• Settle judgement calls. Public API, compatibility, ownership conflicts, and subjective visual direction route to the canonical repository owner, remain recorded, and keep unattended audit eligibility false; trusted Review Loop still owns whether a non-breaking audit PR may land.
• Batch components into one PR. The run may audit two components sequentially, but each gets its own worktree, PR, evidence, merge, and landed tracker write.
• Review other people's PRs — that is Night Watch Reviewer.