Night Watch Component Auditor

Night Watch — Component Auditor

Assigned to: Cindy's Navi (cixzhang)

Goal: Ensure all components follow Astryx conventions — theming compatibility, API consistency, accessibility contracts, and export hygiene.

Source of truth documents:

• API Conventions — API rules (naming, structure, composition, a11y, exports)
• Theming Infrastructure — Theming rules (tokens, themeProps, variant maps, sub-element targeting)
• Design Conventions — Visual/interaction rules. Audit only the objectively checkable conventions (the 🟢 rows in its "Making Conventions Evaluatable" table); leave subjective visual judgment to Component Hardening Protocol Layer 3.

⚠️ Keep in sync. When updating conventions in those pages, update the corresponding check in this auditor. When the auditor finds a pattern not covered by those pages, add it there first, then update the check here.

---

Why This Role Exists

Component quality regresses silently. A hardcoded color, a missing themeProps, a boolean prop without is prefix, a missing displayName — these don't break tests or CI but they break the developer and theming contracts. This role catches the drift across all dimensions in a single pass per component.

---

Scope

This role runs the Component Audit Rubric in mode N — its auto and semi items only. manual items are skipped, not failed, and §5b (rendered design judgment) is reported not measured because a nightly pass captures no screenshots.

That means the checks themselves are not restated here. The rubric is the single source for what is checked, how it is verified, and how it is scored; this page owns only the nightly mechanics below — selection, batching, the PR conventions, the refinement loop, and the state file.

Rubric section	What the nightly pass covers
Component Audit Rubric §1 Accessibility	the semi items — ARIA wiring, busy-vs-disabled, hover-only paths, target-size greps
§2 Theming & token integrity	tokens, themeProps placement and coverage, component var naming, primitive reuse, 'use client'
§3 Public API contract	prop and type naming, BaseProps and ...rest, input field surface, export hygiene
§5a Design conventions — objective	the 🟢 rows: spacing grid, concentric radius, control heights, type hierarchy, overlay tokens, motion safety, stacking, the reorder contract
§6 / §8 Testing and docs	the auto items — story and doc presence, doc↔source parity, the theming guards

Everything a nightly pass cannot judge — the manual rows, and all of §5b — goes to the PR's Needs Review section rather than being scored.

Reporting. Publish the objective subscore as objective: x/y checks. Do not synthesize a letter grade from an automated pass; only modes P, O and R produce a grade.

---

Nightly Checklist

This role runs once per night and produces one PR covering all findings.

Step 1: Check if Already Run Today

Check memory/xds-night-watch-state.json for componentAuditor.lastRunAt. If it's from today (PST), skip and NO_REPLY.

Step 2: Two-Pass Component Selection

The auditor runs two passes per night with separate PRs to keep hardening-issue work distinct from regular queue progress.

Pass 1 — Hardening Issues (priority)

1. List open hardening issues:

   gh issue list --repo facebook/astryx --state open --label hardening \
     --json number,title,body --limit 50

2. Extract component names from single-component issues (e.g. "Hardening: Toolbar" → Toolbar)
3. Already audited? If the component is in completedComponents (state file), close the issue immediately — no re-audit needed:

   gh issue close <number> --repo facebook/astryx --reason completed \
     --comment "Component previously audited by Night Watch. See audit PRs in component history."

4. For remaining components: audit them (Step 3) and create a separate PR (branch: navi/hardening-issues/YYYY-MM-DD)
5. Add all audited components to completedComponents in state.
6. Do NOT close hardening issues that have a new PR — the PR itself is the resolution. The issue closes when the PR merges (via GitHub's linked issues or the PM's gardening pass).
7. For batch hardening issues (e.g. "Hardening: Navigation Components"): read the issue body, extract all component names listed. If ALL are now in completedComponents, close with a comment listing which audit PRs covered them. Otherwise skip — it'll close once the last component is done.

Pass 2 — Queue (regular rounds)

1. Check memory/xds-night-watch-state.json for componentAuditor.auditQueue
2. If the queue is empty, rebuild it from the ledger: node scripts/score-ledger.mjs --queue --limit 5 orders unaudited components first, then oldest-audited, then lowest-scoring — so coverage grows before re-auditing, and the weakest components come back around soonest
3. Pick the next 5 components from the queue
4. Dedupe: skip any that were already audited in Pass 1 this run
5. Audit them (Step 3) and create a separate PR (branch: navi/component-audit/YYYY-MM-DD)
6. Update state: move completed to completedComponents, advance queue

Step 3: Audit the Components

For each component, run all checks (sections 1-10 above) in a single pass:

1. Read the source files — component, sub-components, types, styles, index.ts
2. Check theming: CSS variables, themeProps, component reuse
3. Check API: prop naming, type naming, structure, composition, input consistency, a11y, exports

Step 4: Report Findings

For each issue found, log to memory/xds-night-watch/{date}.md under a ## Component Audit section:

• Categorize as: hardcoded-value, missing-classname, classname-wrong-element, naming-violation, missing-type-prefix, structure-issue, composition-violation, input-inconsistency, a11y-gap, export-gap
• Include file path, line number, current code, and what the fix should be

Step 4b: Record the score in the ledger — required, every audit

An audit that isn't recorded didn't happen. After grading a component, write its score to Component Scores — the ledger is the wiki-side source of truth, so this is a wiki push, not a repo PR.

node scripts/score-ledger.mjs --record <Component> --from <scorecard.json>
# then commit and push the wiki working copy

Record, per the Component Audit Rubric's scorecard format:

• the overall score and grade, and every section score with its state (scored / limited / not_measured / na);
• the open BLOCK list — rule id, one line, and the issue number you filed for it (see Step 4c). FIXes and NITs live in the row only;
• distinct_defects, fixes, nits;
• lastAudited (today), mode (N), the commit audited, and the rubricVersion you graded under — copy it from the rubric's version block. Scores under different versions are not comparable, and the ratchet relies on this field to say so.

Only ever record what you measured. A section you could not verify is not_measured, never a guess and never a zero — the nightly pass captures no screenshots, so design_rendered is always not_measured in mode N.

The ledger is the ratchet's baseline. A later PR that lowers a recorded score, or introduces a new BLOCK, fails the check. So a score you record too generously becomes a gate nobody can pass, and one recorded too harshly blocks the next honest PR. Record what you found.

Step 4c: File an issue per BLOCK, and link it both ways

Every open BLOCK gets one GitHub issue — titled [audit] <Component>: <one-line finding>, labeled hardening — and its number goes on the BLOCK entry in the ledger, so the table links straight to it. FIXes and NITs are not issues; they live in the ledger row. A cross-cutting or system-level finding gets a single issue that every affected row links, never one per component.

Each issue must stand on its own, because whoever picks it up will not have the audit open:

• a link to the component's Component Scores row;
• a link to the Component Audit Rubric naming the rule id and section violated (A8, §1), and the rubric version it was graded under;
• the evidence — file:line, and the screenshot for a visual finding;
• the concrete fix, in the rubric's terms (the token to use, the sibling API to match, the accessible path that must work).

And state the closing protocol in the issue body, so the loop closes without you:

Resolving this requires: re-run the audit for this component (mode O) after the fix; put the visual results in the PR description — before/after screenshots of the affected states; update the Component Scores row with the new score and clear this BLOCK; then close this issue linking the PR. If the recorded score doesn't move, either the fix or the audit is wrong.

Step 5: Fix Findings — Two PRs Per Night (max)

Create separate PRs for each pass:

• Pass 1 branch: navi/hardening-issues/YYYY-MM-DD
• Pass 2 branch: navi/component-audit/YYYY-MM-DD

For each PR:

• Run pnpm test and pnpm build before creating the PR. Fix any failures.
• Publish PRs ready for review (not draft) — label with hardening
• When a finding needs human judgment, include it under a Needs Review section in the PR description
• If no issues found for a pass, do not create a PR — just update state and close issues

Step 5b: Refinement Loop — Drive PR to Green

After creating the PR, don't wait for the daytime PR Review job. Run the refinement loop immediately:

1. Record the HEAD sha you just pushed
2. Schedule a one-shot recheck 5 minutes out (to let CI start)
3. On recheck: a. SHA collision check: git fetch origin <branch> && git log -1 --format=%H origin/<branch>. If HEAD sha differs from what you recorded → STOP (someone else pushed, don't clobber their work) b. Check CI: gh pr checks <number>. If all green → done, log success c. If failing: Read the CI failure logs (gh run view <run_id> --job <job_id> --log-failed). Fix the failures in the worktree, run pnpm test && pnpm build locally, commit, push. Record the new HEAD sha. d. Schedule another recheck 5 minutes out
4. Iteration limit: max 3 fix attempts per PR. If still failing after 3 → log it and move on. The daytime PR Review job will pick it up.

Key rules:

• Always check SHA before acting — if it changed, someone else is working on it
• Only fix CI failures (test, lint, build) — don't re-audit or expand scope during refinement
• Each recheck is a one-shot scheduled job, not a busy-wait
• Log each iteration to memory/xds-night-watch/{date}.md

Step 6: Update State

Update memory/xds-night-watch-state.json:

{
  "componentAuditor": {
    "lastAuditedComponent": "Button",
    "auditQueue": ["Card", "Dialog", ...],
    "prsFiled": ["#650", "#800"],
    "lastRunAt": "2026-03-22T04:00:00Z",
    "completedComponents": ["Button", "TextInput"]
  }
}

When the queue is empty, clear completedComponents and rebuild from scratch.

The state file tracks the run; Component Scores tracks the components. Scores never live here — they belong in the ledger, where the ratchet, the sandbox view and the next auditor can all read them.

---

Does NOT Do

• Subjective design decisions — only objective checks (the objectively-checkable Design Conventions in dimension 11 are in scope; visual judgment — proportions, state treatment, density, composition — stays in Hardening L3)
• Review PRs (that's Reviewer)
• Fix CI beyond the refinement loop (that's QA)
• General issue triage (that's PM) — only closes hardening-labeled issues after auditing

---

State Schema

{
  "componentAuditor": {
    "lastAuditedComponent": "string",
    "auditQueue": ["string[]"],
    "prsFiled": ["string[]"],
    "lastRunAt": "ISO timestamp",
    "completedComponents": ["string[]"]
  }
}