[Golden Test] Collector hardening: schedule-run filter, missing-prev-report guard, dynamic attribution window

[PeterJhongLinksys]

Part of the daily golden-CI triage agent buildout (judgment/fix layer on top of the golden-ci GitHub Actions).

Harden the existing read-only collector ~/.hermes/automation/export-kit/pipelines/golden_verify/collector.py with three fixes agreed during design review:

• Schedule-run filter — when polling Daily Golden Verify runs, only accept the scheduled run whose rendered branch == TARGET_BRANCH; ignore manual workflow_dispatch runs that may have rendered a different branch.
• Missing previous-report guard — if the previous day's verify report can't be fetched, HOLD the overflow diff and report it explicitly. NEVER treat all persistent overflows (~361) as "new" (that single-day blast is exactly what dedup exists to prevent).
• Dynamic attribution window — switch merged-PR fetch from a fixed "yesterday" span to a dynamic last-green-run → this-run span, so an unfixed cause from several days ago stays inside the window until its screens go green.

Context: collector is already written and verified (offline unit tests + live parse of the real report). These are incremental edits. TARGET_BRANCH is empirically dev-2.7.0, read live — do not hardcode.

Tracking issue for automation infra (no PrivacyGUI app code changes).

[PeterJhongLinksys]

✅ Dev done. Three collector hardenings landed and verified (offline unit tests + live --config run):

• Schedule-run filter: find_run now only trusts event==schedule runs; manual dispatch flagged manual_only.
• Missing-prev-report guard: overflow_diff_held — if yesterday's report is unfetchable, the overflow diff is HELD (no 361-issue blast).
• Dynamic attribution window: merged-PR fetch now spans last-green-run → this-run (attribution_window), replacing fixed-yesterday. Verified window on 2026-07-29: 2026-07-27T20:34 → 2026-07-28T20:30.

[PeterJhongLinksys]

🔄 Follow-up: the collector was changed substantially after the comment above, so that summary is now partly out of date. Current state of pipelines/golden_verify/collector.py:

Corrections to the earlier hardening

• verified_branch no longer falls back to usp. The original code returned the workflow default usp when gh variable get TARGET_BRANCH failed — indistinguishable from a genuine usp, so a lookup failure would have silently driven judging (and a baseline trigger) against the wrong branch. It now returns (None, 'unresolved') and the output carries verified_branch_source; only repo_var is trusted, anything else is an anomaly early-exit.
• Merged PRs are now fetched on systemic days too. They were suppressed, which made the playbook's requirement to list them (marked "not attributed") impossible to satisfy. Suppressing attribution is the playbook's job; suppressing the data was wrong.
• The systemic breaker was retuned to two conditions — a high failing-row fraction AND failures spread across most areas — replacing the single 5 % threshold. Volume alone misfires: one shared-string edit across 26 locales is a large count yet perfectly attributable, and a partial baseline shrinks the denominator (a real 3-area, 24-screen failure measured 6.7 % and would have been misclassified as environment drift). Output now separates systemic_thresholds from systemic_measured so a threshold can't be misread as a measurement.

New output fields

• Row caps: errors[] / new_overflow[] are capped, with *_total, *_truncated, and all_*_keys alongside. The full key list is always emitted because dedup fingerprinting needs keys, not rows — thousands of ~1 KB rows would otherwise flood the agent's context on exactly the day it matters most. clean is computed from the totals so a capped payload can never read as clean.
• image_base — the base URL that goldenPath and failureImages.* are relative to, so the agent never assembles URLs by hand.
• goldenPath and failureImages are now passed through; the report-internal filePath is dropped because it 404s on gh-pages.
• area per row (derived from testCaseFilePath), used by both attribution and the breaker's spread measure.
• messages was added and then removed: measured against real failing rows it holds only per-test debug output plus a ~1400-char stack trace whose sole verdict is "does not match". Alchemist reports no pixel-difference magnitude, so it offered no signal and only cost context.

Failure-detection authority (important, and a correction of earlier thinking). clean no longer consults the run conclusion. The report is the sole authority on image-comparison failures and overflow — a red run only means the round hit an obstacle, and a bad clone or broken environment turns it red too. Run status is used solely to establish whether a usable report exists at all.

--run-id <id> added for manual verification / re-judging a past day: it uses the named run and bypasses the schedule-only filter. The playbook never passes it, so the scheduled path keeps the filter intact. Chosen over a vague "allow manual runs" flag because it forces the operator to name the run explicitly.

[PeterJhongLinksys]

Collector gap found by the daily triage agent (2026-08-01 run)

Symptom: today's collector output reported new_overflow_total = 355 (and persistent_overflow_count = 6) even though the day was otherwise fully green (report_counting.fail = 0, errors_total = 0).

Verified root cause — false positive, not a real regression:

• Today's (2026-08-01) overflow key set is byte-identical to 2026-07-30's: today - 0730 = 0, 0730 - today = 0, both 361 keys.
• prev_date_with_report() picked 2026-07-31 as the diff baseline. That report contains only 357 real tests, locale en only (vs. 9282 / 25 locales in a full run) — it was produced by a manual workflow_dispatch render, not by the scheduled run.
• Because that partial baseline only carried 6 overflow keys, today_ovf - prev_ovf yielded 355 keys that are in fact long-standing, already-baked-in overflows.
• Confirmed: overflow_keys(0731) is a strict subset of today's set.

Why the existing guard did not catch it: overflow_diff_held only triggers when the previous report is missing (prev_data is None). It does not consider a report that exists but is a partial / non-scheduled render, which is an equally defective diff baseline.

Suggested hardening (for whoever picks this up):

1. When selecting the previous report, skip reports whose real_tests count or locale count is drastically smaller than today's (e.g. < 50% of today's real test count) and fall back further back in the lookback window; if no comparable report is found, set overflow_diff_held = true.
2. Alternatively/additionally, record which run id produced each report and prefer reports from event == schedule runs as diff baselines.
3. Consider emitting the baseline's real_tests / locale count in the collector output so the agent can sanity-check the diff basis itself.

No issue was opened for the 355 "new" overflows and no baseline update was triggered — the day was judged green on the report contents.

Report: https://linksys.github.io/PrivacyGUI-golden-ci/verify/dev/2026-08-01/golden_verify_report.html
Run: https://github.com/linksys/PrivacyGUI-golden-ci/actions/runs/30663234283