[#803 split 1/2] _run_json: surface real error + exponential jittered backoff

[don-petry]

Split 1/2 of #803 (dev-lead timed out at the 2100s action budget doing all of #803 at once). This is the small, high-impact half — a single-function change.

Scope — scripts/canary-rollout.sh, _run_json (~L378-403) ONLY

1. Surface the real error. Capture gh's stderr and include the HTTP status / message in the ::warning:: and final ::error:: lines. Today the retry wrapper prints only "transient failure … after N attempts", hiding whether it's a 403 secondary-rate-limit vs 5xx vs auth. Distinguish these in the output.
2. Exponential + jittered backoff. Replace the fixed CANARY_GH_RETRY_SLEEP linear retry with exponential backoff + jitter; honor Retry-After / x-ratelimit-reset response headers when present; raise the default attempt count. Keep the CANARY_GH_RETRIES / CANARY_GH_RETRY_SLEEP env overrides working.
3. Fail closed. A persistent fetch failure must remain a hard error — never a silent empty-but-green gate.

Out of scope (that's #803b)

• Collapsing per-workflow enumeration to a per-repo fetch.
• sync-issues resilience.

Acceptance criteria

• A forced fetch failure prints the real HTTP status/reason, not just "transient".
• Backoff is exponential+jittered; Retry-After honored; attempts raised.
• Fails closed (no empty-data gate reported as promotable).
• Existing canary-rollout bats stay green; add a case asserting _run_json surfaces the real error on failure.

[don-petry]

Dev-Lead Implementation Plan

Issue: #810 — [#803 split 1/2] _run_json: surface real error + exponential jittered backoff

Scope

Rework _run_json in scripts/canary-rollout.sh to capture gh's stderr and surface the real HTTP status/class in its ::warning::/::error:: annotations, and replace the fixed linear retry with exponential + jittered backoff that honors Retry-After / x-ratelimit-reset hints — while preserving fail-closed behavior and the CANARY_GH_RETRIES / CANARY_GH_RETRY_SLEEP overrides.

Implementation Checklist

• Add _gh_err_summary helper: condense gh stderr into HTTP <status> <class>: <reason> (distinguishes secondary-rate-limit / auth / server / client / network)
• Add _gh_retry_after helper: parse Retry-After: / bare "retry after Ns" / x-ratelimit-reset: epoch from stderr
• Rewrite _run_json: capture gh stderr, include summary in warning + error lines, exponential+jittered backoff (cap 30s), honor retry-after hint, raise default attempts 3→6, keep env overrides + fail-closed non-zero return
• Update the _run_json header comment for the new default/behavior

Tests to Write (tests/canary_rollout.bats)

• _run_json surfaces the real HTTP 403 secondary-rate-limit status/class in the final ::error:: line on persistent failure
• _run_json surfaces the status/class in the ::warning:: retry line
• _run_json honors a Retry-After hint from gh stderr for the backoff delay
• (regression) existing transient-retry, fail-closed, env-override, and 30s-cap cases stay green

Test Command

bats tests/canary_rollout.bats (plus bash .dev-lead/scripts/dev-lead-lint.sh and shellcheck scripts/canary-rollout.sh)

[don-petry]

Dev-Lead: Implementation Complete

Plan Execution

• Added _gh_err_summary helper — condenses gh stderr into HTTP <status> <class>: <reason>, distinguishing secondary-rate-limit / rate-limit / auth / server-error / forbidden / client-error / network-or-unknown
• Added _gh_retry_after helper — parses Retry-After: / bare "retry after Ns" / x-ratelimit-reset: epoch from gh stderr
• Rewrote _run_json — captures gh stderr (no longer 2>/dev/null), surfaces the real status/class in both ::warning:: and final ::error:: lines, exponential + full-jitter backoff (cap 30s) that honors a server retry-after hint (cap 120s), raised default attempts 3→6, kept CANARY_GH_RETRIES / CANARY_GH_RETRY_SLEEP overrides, and preserved the fail-closed non-zero return
• Updated the _run_json header comment for the new behavior
• Added 4 new bats cases (403 secondary-rate-limit in ::error::, 502 server-error in ::warning::, 401 auth class, Retry-After honored)

Test Results (237 passing, 0 failing; shellcheck + dev-lead-lint clean)

ok 66 _run_json: retries a transient gh failure, then returns the payload (#738)
ok 67 _run_json: fails CLOSED (non-zero) when gh fails on every attempt (#738)
ok 68 _run_json: empty CANARY_GH_RETRIES falls back to safe default (3)
ok 69 _run_json: non-integer CANARY_GH_RETRIES falls back to safe default (3)
ok 70 _run_json: empty CANARY_GH_RETRY_SLEEP falls back to safe default (2)
ok 71 _run_json: non-integer CANARY_GH_RETRY_SLEEP falls back to safe default (2)
ok 72 _run_json: exponential backoff delay is capped at 30 seconds
ok 73 _run_json: empty repo returns [] without calling gh
ok 74 _run_json: wildcard repo returns [] without calling gh
ok 75 _run_json: final ::error:: surfaces the real HTTP 403 secondary-rate-limit status/class (#810)
ok 76 _run_json: retry ::warning:: surfaces the real HTTP status/class before retrying (#810)
ok 77 _run_json: an auth failure is classed as auth, not a bare transient (#810)
ok 78 _run_json: honors a Retry-After hint from gh stderr for the backoff delay (#810)

Files Changed

• scripts/canary-rollout.sh: added _gh_err_summary + _gh_retry_after helpers; rewrote _run_json to surface the real error and use exponential+jittered backoff honoring Retry-After, raised default attempt count, preserved fail-closed + env overrides
• tests/canary_rollout.bats: added 4 _run_json cases asserting the real HTTP status/class is surfaced and the Retry-After hint is honored

Notes

• Out-of-scope items (per-repo fetch collapse, sync-issues resilience) deferred to #803b — untouched.
• Exponential path keeps the existing 30s cap (existing cap test stays green); explicit Retry-After hints use a separate 120s cap so a genuine throttle window is honored without walling for minutes.
• Default attempt count raised 3→6; the two existing "falls back to safe default" cases assert fallback behavior (not the literal number) and remain green.