Harden canary-rollout _run_json: surface real error + backoff + cut call volume (fixes empty gates blocking promotion)

[don-petry]

Problem

The Canary Rollout gate (scripts/canary-rollout.sh, _run_json, ~L378-403)
fetches per-workflow run history with gh run list --repo <r> --workflow <wf> in a
tight per-agent × per-tier loop. Under the fleet-wide burst these calls hit
transient failures ("_run_json: failed to fetch run list … after 3 attempts",
~7/run across both repos), which yield empty gates (gate= for ring '' [] (triage=)) → agents don't promote. This is the actual blocker on dev-lead's
next→ring0 promotion (its gate shows cum_fail=0, triage=- — no health data, NOT
a real regression). It also breaks sync-issues (regression auto-tracking:
issue sync failed (non-fatal)), so real regressions like feature-ideation's go
un-tracked.

Root cause is almost certainly secondary rate-limiting from the per-workflow
call volume — but we can't confirm because _run_json swallows the underlying
error (the retry wrapper prints only "transient failure", never the HTTP
status/Retry-After). That observability gap must be closed first.

Scope (all in scripts/canary-rollout.sh)

1. Surface the real error (do first). In _run_json, capture gh's stderr and
   include the HTTP status / message in the ::warning::/::error:: lines
   (distinguish 403 secondary-rate-limit vs 5xx vs auth). No more blind "transient
   failure". Keep fail-closed: a persistent failure must remain a hard error, never
   an empty-but-green gate.
2. Backoff. Replace the fixed CANARY_GH_RETRY_SLEEP linear retry with
   exponential backoff + jitter; honor Retry-After / x-ratelimit-reset when
   present; raise default attempts. (Keep the CANARY_GH_RETRIES /
   CANARY_GH_RETRY_SLEEP env overrides.)
3. Cut call volume (the real fix). Replace per-workflow gh run list
   enumeration with far fewer calls — e.g. one paginated gh run list --repo <repo>
   (or a single GraphQL query) per repo, filtered to the registered workflows
   locally. This removes the burst that triggers the secondary limit.
4. Make sync-issues resilient. A run-history fetch failure must not silently
   drop the REGRESSION→needs-human issue upsert; on partial data, still upsert what
   it can and annotate the gap (so regressions are never silently un-tracked).

Acceptance criteria

• A forced fetch failure prints the real HTTP status/reason, not just "transient".
• Backoff is exponential+jittered; Retry-After honored.
• Per-evaluation gh run list call count is materially reduced (state the before/after in the PR).
• sync-issues still opens/updates the regression issue when some fetches fail.
• Fails closed: no empty-data gate is ever reported as promotable.
• Existing bats for canary-rollout stay green; add a case for _run_json error surfacing.

[don-petry]

Dev-Lead Implementation Plan

Issue: #803 — Harden canary-rollout _run_json: surface real error + backoff + cut call volume (fixes empty gates blocking promotion)

Scope

Harden scripts/canary-rollout.sh run-history fetching so a fleet-wide burst no longer yields empty gates: surface the real gh error, use exponential+jittered backoff honoring Retry-After, collapse per-workflow enumeration to one paginated per-repo fetch (cached), and make the gate + sync-issues fail closed and stay resilient on partial fetch failures.

Implementation Checklist

• _run_json (scope 1): capture gh stderr, classify (403 secondary-rate-limit vs 5xx vs auth), and include the real HTTP status/reason in the ::warning::/::error:: lines. Keep non-zero on sustained failure (fail closed).
• _run_json (scope 2): replace the linear retry with exponential backoff + jitter; honor Retry-After / x-ratelimit-reset parsed from stderr; raise default attempts (3→5). Keep CANARY_GH_RETRIES / CANARY_GH_RETRY_SLEEP overrides + the 30s self-backoff cap.
• Call-volume cut (scope 3): add a per-repo run-list cache (_repo_runs_json) that issues ONE paginated gh run list --repo <repo> per repo (dropping --workflow), widening the window as needed; _run_json filters workflowName + createdAt locally. Collapses repeated (repo,wf,since) and cross-agent shared-repo reads into a single call.
• Gate resilience (scope 5): gate-gathering helpers record a data gap (via a gap file) instead of aborting under set -e; _frontier_state forces state=BLOCKED + triage=INCOMPLETE on any gap so an empty-data gate is never promotable, while preserving a real REGRESSION/SUSPECT signal from readable repos.
• cmd_evaluate: surface triage=INCOMPLETE with an ::error:: naming the fetch gap.
• sync-issues resilience (scope 4): a fetch gap keeps the agent BLOCKED, so the blocker issue is still opened/updated (+needs-human) rather than closed; annotate the partial-data gap in the blocker body/evidence.

Tests to Write (tests/canary_rollout.bats)

• _run_json prints the real HTTP status/reason (403 secondary-rate-limit) on failure — not just "transient".
• Backoff honors a Retry-After value parsed from gh stderr (sleep called with it); jitter stays within the 30s cap.
• Per-repo call-volume reduction: two _run_json calls for the same repo (two workflows / two in-window sinces) issue a single gh run list (counter assertion); --workflow no longer passed.
• Local workflowName filtering returns only the queried workflow's runs from a mixed-workflow payload.
• Gate fails closed: on a sustained fetch failure _frontier_state yields BLOCKED/INCOMPLETE, never PROMOTE/COMPLETE.
• sync-issues still upserts the blocker issue (does not close it) when some fetches fail.

Test Command

bats tests/canary_rollout.bats (plus bash .dev-lead/scripts/dev-lead-lint.sh for shellcheck)

[don-petry]

Dev-Lead: cannot implement issue #803 — needs human attention

A stage timed out (exit 124) while implementing this issue. A timeout is treated as a first-class, non-retryable condition — retrying with the same budget would just time out again — so this escalates for human review on the first occurrence instead of wasting a same-budget retry.

• Cause: timeout
• Tier: action
• Elapsed: 2100s
• Budget: 2100s
• Run: https://github.com/petry-projects/.github/actions/runs/29700882105

This hit the max action budget. Split this issue into smaller, independently-implementable pieces, or raise vars.ACTION_TIMEOUT_SEC, then re-apply the dev-lead label.

Note: the engine had produced changes before the timeout. They were on branch dev-lead/issue-803-20260719-1935 on the runner and were not pushed, so they are not recoverable from this run (the runner is ephemeral). The redacted session snippet below shows what it attempted — treat it as context, not restorable output. Re-apply dev-lead after splitting the issue or raising the budget to regenerate the work.

[don-petry]

Dev-lead timed out at the 2100s action budget implementing this in one pass. Per the agreed approach, split into two independently-implementable stories (this issue is now the parent tracker):

• [#803 split 1/2] _run_json: surface real error + exponential jittered backoff #810 [split 1/2] — _run_json surface real error + exponential jittered backoff (small, high-impact).
• [#803 split 2/2] canary-rollout: collapse run-list enumeration to per-repo fetch + fail-closed sync-issues #811 [split 2/2] — collapse per-workflow enumeration to a per-repo fetch + fail-closed sync-issues (structural).

De-labeled dev-lead/dev-lead:needs-human here; the splits carry the dev-lead label. Closes when both merge.