Skip to content

ci(fleet): harden dispatch-suite run recovery against transient API blips - #642

Merged
joshua-temple merged 1 commit into
mainfrom
fix/fleet-seed-trigger-fallback
Jul 20, 2026
Merged

ci(fleet): harden dispatch-suite run recovery against transient API blips#642
joshua-temple merged 1 commit into
mainfrom
fix/fleet-seed-trigger-fallback

Conversation

@joshua-temple

Copy link
Copy Markdown
Collaborator

Problem

The rc.2 fleet run went 11/12 green on real content. The single hard failure was cascade-example-4env (the heaviest lane), and it was not a cascade defect. In its scenario suite, "Step 1 - reset and seed dev" merges a src/** marker commit to trunk and then waits for the orchestrate run that push should trigger. On the failing run GitHub never created an orchestrate run for that merge SHA (verified: zero runs for that head_sha, a dropped best-effort push event), so the wait timed out with "timed out waiting for orchestrate run". 4env is most prone because it is the heaviest lane with the most concurrent trunk writers; the reset push itself logged "push rejected; re-applying reset onto updated trunk". A plain rerun passed. This transient trigger-loss is what forces reruns and blocks a clean whole-fleet pass.

Existing machinery (mapped before changing anything)

  • .github/workflows/fleet-e2e.yaml fans out per-lane with retry-attempts: '2' on every dispatch-suite call (whole-suite re-dispatch on failure).
  • .github/actions/dispatch-suite/action.yaml does dispatch -> recover -> watch. The watch loop already has a transient classifier plus a consecutive-error budget. The recovery loop did not, and neither loop had jitter.
  • The seed-and-wait logic lives in the example repos' scenario-suite.yaml (not in this repo). The wait_for_orchestrate_sha helper is where the dropped-trigger has no fallback. That is fixed in a companion PR against stablekernel/cascade-example-4env.

This PR (cascade-side, complementary hardening)

dispatch-suite/action.yaml:

  1. Recovery loop was assigning gh run list output with a plain command substitution under set -euo pipefail, so one transient Actions-API error (secondary rate limit / 5xx / auth blip) aborted the whole action and skipped the outer retry, failing a lane closed prematurely. It now captures the list status and treats a failure as "not yet visible" and keeps polling, matching the watch loop's existing tolerance.
  2. Added a shared sleep_with_jitter used by both the recovery and watch poll sleeps so many concurrently watched suites do not re-poll in lockstep and re-trip the shared secondary rate limit. The watch loop's existing transient classifier is unchanged.

Verification

  • actionlint clean on the composite action (only pre-existing SC2129 style findings remain in generated orchestrate.yaml/promote.yaml).
  • bash -n and shellcheck clean on the modified script.
  • Guardrail scan clean.

…lips

The cross-repo run-recovery loop assigned the gh run list result with a
plain command substitution under set -euo pipefail, so a single transient
Actions-API error (secondary rate limit, 5xx, auth blip) aborted the whole
action and skipped the outer retry, failing a lane closed prematurely. The
watch loop already guarded against this; the recovery loop did not.

Capture the list status and treat a failure as not-yet-visible so the loop
keeps polling. Add shared sleep_with_jitter backoff on both the recovery
and watch poll sleeps so many concurrently watched suites do not re-poll in
lockstep and re-trip the shared secondary rate limit.

Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
@joshua-temple
joshua-temple merged commit 8d862d2 into main Jul 20, 2026
17 checks passed
@joshua-temple
joshua-temple deleted the fix/fleet-seed-trigger-fallback branch July 20, 2026 04:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant