ci(fleet): harden dispatch-suite run recovery against transient API blips - #642
Merged
Merged
Conversation
…lips The cross-repo run-recovery loop assigned the gh run list result with a plain command substitution under set -euo pipefail, so a single transient Actions-API error (secondary rate limit, 5xx, auth blip) aborted the whole action and skipped the outer retry, failing a lane closed prematurely. The watch loop already guarded against this; the recovery loop did not. Capture the list status and treat a failure as not-yet-visible so the loop keeps polling. Add shared sleep_with_jitter backoff on both the recovery and watch poll sleeps so many concurrently watched suites do not re-poll in lockstep and re-trip the shared secondary rate limit. Signed-off-by: Joshua Temple <joshua.temple@stablekernel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The rc.2 fleet run went 11/12 green on real content. The single hard failure was
cascade-example-4env(the heaviest lane), and it was not a cascade defect. In its scenario suite, "Step 1 - reset and seed dev" merges asrc/**marker commit to trunk and then waits for the orchestrate run that push should trigger. On the failing run GitHub never created an orchestrate run for that merge SHA (verified: zero runs for thathead_sha, a dropped best-effort push event), so the wait timed out with "timed out waiting for orchestrate run". 4env is most prone because it is the heaviest lane with the most concurrent trunk writers; the reset push itself logged "push rejected; re-applying reset onto updated trunk". A plain rerun passed. This transient trigger-loss is what forces reruns and blocks a clean whole-fleet pass.Existing machinery (mapped before changing anything)
.github/workflows/fleet-e2e.yamlfans out per-lane withretry-attempts: '2'on everydispatch-suitecall (whole-suite re-dispatch on failure)..github/actions/dispatch-suite/action.yamldoes dispatch -> recover -> watch. The watch loop already has a transient classifier plus a consecutive-error budget. The recovery loop did not, and neither loop had jitter.scenario-suite.yaml(not in this repo). Thewait_for_orchestrate_shahelper is where the dropped-trigger has no fallback. That is fixed in a companion PR againststablekernel/cascade-example-4env.This PR (cascade-side, complementary hardening)
dispatch-suite/action.yaml:gh run listoutput with a plain command substitution underset -euo pipefail, so one transient Actions-API error (secondary rate limit / 5xx / auth blip) aborted the whole action and skipped the outer retry, failing a lane closed prematurely. It now captures the list status and treats a failure as "not yet visible" and keeps polling, matching the watch loop's existing tolerance.sleep_with_jitterused by both the recovery and watch poll sleeps so many concurrently watched suites do not re-poll in lockstep and re-trip the shared secondary rate limit. The watch loop's existing transient classifier is unchanged.Verification
actionlintclean on the composite action (only pre-existing SC2129 style findings remain in generatedorchestrate.yaml/promote.yaml).bash -nandshellcheckclean on the modified script.