[Golden Test] End-to-end dry-run verification before go-live

[PeterJhongLinksys]

Part of the daily golden-CI triage agent buildout.

Validate the whole pipeline on a real failing date (or synthetic DATA) before enabling live writes.

• Run collector → attribution → grouping → screen-overlap dedup → AI intended/regression verdict, in a DRY-RUN mode that does NOT actually open issues or trigger workflows.
• Verify: correct suspect PRs, correct grouping (incl. shared-widget blast radius), dedup matches an existing open issue, circuit breaker fires on a synthetic >5% failure, intended-but-broken veto works.
• Only after output is correct, remove dry-run and enable.

Tracking issue for automation infra (no PrivacyGUI app code changes).

[PeterJhongLinksys]

✅ Verification done in two phases, both with a context-free agent (given only the playbook path, no conversation history) so that the playbook's self-sufficiency was tested alongside its logic. No writes reached this repo during either phase.

Phase 1 — green-day path (2026-07-31). Lock → collector → crosscheck → anomaly check → all-green fast path → Telegram heartbeat (ok:true) → lock release. Ran with zero guessing. Doubled as a self-sufficiency audit and surfaced 12 defects, all fixed the same day (see #1186).

Phase 2 — judge-only replay on a real failing report (2026-07-31). A deliberately slim baseline (locale en only, 357 rows vs the normal 26-locale 9282) produced 24 failing screens across firewall / dmz / static_routing. The agent ran the full playbook with every write swapped for a printed draft.

Results:

• Grouping: all 24 rows collapsed into ONE cluster — correctly not split into three issues by area, since a single cause with a single failure mode spanning several areas is exactly the shared-component blast radius the design must not fragment.
• Verdict: downgraded to regression by the strict intended bar (no PR in the attribution window, confidence not High, broken layout detected).
• Vision judging (the only layer never previously exercised) proved genuinely useful. From 3 diff images it diagnosed a cumulative vertical layout shift — ghosted labels offset progressively further down the page (≈40 px near the top to ≈200 px at the bottom), toggle knobs merged into a figure-eight, rows collapsed until text interleaved unreadably — and independently ranked "baseline may be incomplete: only 357 rows / 1 locale" as the most likely cause, which was exactly the truth. broken_flag fired purely from pixels (hasOverflow was false on all 24 rows), confirming that veto works without the overflow axis.
• Dedup: all 6 open [Golden Test] issues lack a fingerprint line, so they were skipped entirely rather than treated as non-overlapping.
• Baseline: correctly not triggered (an unattributed regression cluster exists).
• Quota: 3 of 10 samples, 3 images total — far under the cap.

Facts finally proven against real failure data (previously only inferred from report-generator source):

• A golden mismatch surfaces as result == "error" (distribution {success: 333, error: 24}). An earlier suspicion that it might be "failure" — which would have meant the CI's failure counting was broken — is now dead.
• The crosscheck holds: the report's own fail: 24 equals the collector's extracted count.
• failureImages.{expected,actual,diff} really is emitted on failing rows, and all image URLs (including goldenPath) return HTTP 200 when appended verbatim to the report base.
• The report carries no pixel-difference magnitude. Alchemist reports only "does not match", so there is no cheap pre-screening signal; difference size can only be assessed by reading the isolated-diff image. Guidance that assumed otherwise was removed.

One near-miss worth recording. Retuning the systemic breaker in the collector had left the playbook body still describing the previous single 5 % threshold. The day's measured 6.72 % straddled the old and new rules, so the two paths would have produced entirely different issues (env-drift stub vs. full diagnosis). The dry run caught it. Lesson: when collector semantics change, grep the playbook in the same pass.

Breaker retuned as a result: it now requires both a high failing-row fraction and failures spread across most areas. Volume alone is a poor signal — editing one shared string across 26 locales is a large count yet perfectly attributable, and a partial baseline shrinks the denominator. Genuine SDK/font drift is wide because it changes rendering itself.