fix(review-reviewers): stop a skipped init-tracking from cancelling the whole matrix - #877
Conversation
…he whole matrix A hard `needs: init-tracking` meant a 15-minute runner-acquisition failure on the tracking-issue job skipped all five matrix legs. Four consecutive ticks hit this during the 2026-08-06 Actions incident, costing 8h15m of coverage. The skill carries its own find-or-create fallback, so init-tracking is an optimisation for the first tick of each month, not a precondition. Keep `needs:` so it still wins the race normally, but let the matrix proceed when it doesn't complete.
tend-agent
left a comment
There was a problem hiding this comment.
The mechanism checks out: in all four cited runs the matrix job's conclusion was skipped, not cancelled, and the run conclusion was failure — so the skip came from the default needs-success gate, not from a run-level cancellation. cancelled() reports workflow-run cancellation, so it stays false in that shape and !cancelled() lets the legs through. Keeping needs: so ordering survives is the right call.
One inline fix on the new comment's cost estimate, plus one doc that this change invalidates: plugins/tend-ci-runner/skills/review-reviewers/SKILL.md still says the init-tracking job runs first "so matrix legs always find an existing one", with find-or-create framed as being only "for ad-hoc invocations and as a safety net". After this change that invariant no longer holds on a first-tick-of-month failure — the fallback becomes a real code path, not a safety net. Since that sentence is the load-bearing premise in the PR description, worth reconciling; happy to fold the one-sentence edit into this branch.
|
Good catch on the skill doc — that sentence was the premise the PR description leaned on, and the change does invalidate it. Both fixes are in 27641c6, alongside the inline N-1 correction.
The PR description's "at most one duplicate tracking issue" is corrected to N-1 there as well, and the sentence quoting the old "safety net" wording now points at the updated text instead. |
review-reviewerslost 8 h 15 m of coverage today across all five target repos, and the cause was a two-line gate in its own workflow rather than anything in a skill.What happened
A critical GitHub Actions incident (status page, created 2026-08-06T15:22:49Z, still
investigating) made runner acquisition unreliable. Four consecutivereview-reviewersticks then failed identically —init-trackingsat unscheduled for ~15 minutes, GitHub marked itcancelled, and because the matrix job has a hardneeds: init-tracking, all five legs wereskippedwithout ever trying for a runner:init-trackingcancelledskipped×5cancelledskipped×5cancelledskipped×5cancelledskipped×5That's 20 leg-analyses discarded. The legs were not themselves unschedulable: in 31119653960 (16:23Z)
init-trackingdid succeed, and matrix legs went on to acquire runners and run —numbagg/numbaggcompleted successfully at 16:34Z during the same incident.Why the gate isn't worth its cost
init-trackingexists to win a find-or-create race that can only bite on the first cron tick of a month — before that tick, no tracking issue exists for the new month and N legs would each try to create one. Thereview-reviewersskill still carries its own find-or-create fallback, sorting lowest-numbered-first precisely so a lost race degrades to a duplicate rather than a crash. This change makes that path load-bearing on a first-tick failure rather than a pure safety net, so the skill's wording is updated in the same commit to say so.So the gate trades up to N-1 duplicate tracking issues (four, at today's five legs), once a month against the entire analysis window for every target repo, every time the job can't get a runner. This change keeps
needs:— soinit-trackingstill runs first and still wins the race on every normal tick — and only relaxes the failure propagation.!cancelled()rather thanalways()is deliberate: a user cancelling the run should still stop the matrix.Gate assessment
needs:gate always propagates; there is no decision point for the agent to get right. Confirmed against the last 14review-reviewersfailures: theinit=cancelled → matrix=skippedshape appears only on today's four, while every 2026-08-05 failure raninit=successwith the matrix reaching the agent. The gate is the amplifier; the incident is only the trigger.init-tracking. perf(review-reviewers): cut cadence to 3-hourly, and anchor the window to the cron period #845 (cadence) and fix(review-reviewers): give each matrix leg its own PR branch name #858 (per-leg branch names) touch this workflow but not this gate.Evidence log: https://gist.github.com/192514ea2c36586f9b7f842a482d62ab
Found while analysing
PRQL/prqlin run 31132205674 — the run that had to absorb all four lost windows.