Skip to content

fix(review-reviewers): stop a skipped init-tracking from cancelling the whole matrix - #877

Merged
max-sixty merged 2 commits into
mainfrom
hourly/review-31132205674
Aug 7, 2026
Merged

fix(review-reviewers): stop a skipped init-tracking from cancelling the whole matrix#877
max-sixty merged 2 commits into
mainfrom
hourly/review-31132205674

Conversation

@tend-agent

@tend-agent tend-agent commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

review-reviewers lost 8 h 15 m of coverage today across all five target repos, and the cause was a two-line gate in its own workflow rather than anything in a skill.

What happened

A critical GitHub Actions incident (status page, created 2026-08-06T15:22:49Z, still investigating) made runner acquisition unreliable. Four consecutive review-reviewers ticks then failed identically — init-tracking sat unscheduled for ~15 minutes, GitHub marked it cancelled, and because the matrix job has a hard needs: init-tracking, all five legs were skipped without ever trying for a runner:

Run Created init-tracking matrix
31122877870 17:21:15Z cancelled skipped ×5
31125649607 18:17:43Z cancelled skipped ×5
31126978811 19:34:17Z cancelled skipped ×5
31128259981 21:22:19Z cancelled skipped ×5

That's 20 leg-analyses discarded. The legs were not themselves unschedulable: in 31119653960 (16:23Z) init-tracking did succeed, and matrix legs went on to acquire runners and run — numbagg/numbagg completed successfully at 16:34Z during the same incident.

Why the gate isn't worth its cost

init-tracking exists to win a find-or-create race that can only bite on the first cron tick of a month — before that tick, no tracking issue exists for the new month and N legs would each try to create one. The review-reviewers skill still carries its own find-or-create fallback, sorting lowest-numbered-first precisely so a lost race degrades to a duplicate rather than a crash. This change makes that path load-bearing on a first-tick failure rather than a pure safety net, so the skill's wording is updated in the same commit to say so.

So the gate trades up to N-1 duplicate tracking issues (four, at today's five legs), once a month against the entire analysis window for every target repo, every time the job can't get a runner. This change keeps needs: — so init-tracking still runs first and still wins the race on every normal tick — and only relaxes the failure propagation.

!cancelled() rather than always() is deliberate: a user cancelling the run should still stop the matrix.

Gate assessment

  • Evidence level: High — 4 occurrences in a single window, all with byte-identical job shape.
  • Structural, not stochastic. A hard needs: gate always propagates; there is no decision point for the agent to get right. Confirmed against the last 14 review-reviewers failures: the init=cancelled → matrix=skipped shape appears only on today's four, while every 2026-08-05 failure ran init=success with the matrix reaching the agent. The gate is the amplifier; the incident is only the trigger.
  • Change type: removal of a gate — 2 functional lines. Under the magnitude gate a removal clears at 1 occurrence; this has 4.
  • Not an incident workaround. The skill's own rule is to record transient upstream incidents rather than commit code around them, and this run recorded the rest of the incident's damage without acting on it. This one is different in kind: it removes a permanent single point of failure that will amplify the next infrastructure hiccup the same way, and it stays correct after the incident resolves.
  • Dedup: no open or closed tend issue or PR mentions init-tracking. perf(review-reviewers): cut cadence to 3-hourly, and anchor the window to the cron period #845 (cadence) and fix(review-reviewers): give each matrix leg its own PR branch name #858 (per-leg branch names) touch this workflow but not this gate.

Evidence log: https://gist.github.com/192514ea2c36586f9b7f842a482d62ab

Found while analysing PRQL/prql in run 31132205674 — the run that had to absorb all four lost windows.

…he whole matrix

A hard `needs: init-tracking` meant a 15-minute runner-acquisition failure on
the tracking-issue job skipped all five matrix legs. Four consecutive ticks hit
this during the 2026-08-06 Actions incident, costing 8h15m of coverage.

The skill carries its own find-or-create fallback, so init-tracking is an
optimisation for the first tick of each month, not a precondition. Keep
`needs:` so it still wins the race normally, but let the matrix proceed when
it doesn't complete.
@tend-agent tend-agent added the claude-behavior Behavioral issues found by review-reviewers label Aug 6, 2026

@tend-agent tend-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mechanism checks out: in all four cited runs the matrix job's conclusion was skipped, not cancelled, and the run conclusion was failure — so the skip came from the default needs-success gate, not from a run-level cancellation. cancelled() reports workflow-run cancellation, so it stays false in that shape and !cancelled() lets the legs through. Keeping needs: so ordering survives is the right call.

One inline fix on the new comment's cost estimate, plus one doc that this change invalidates: plugins/tend-ci-runner/skills/review-reviewers/SKILL.md still says the init-tracking job runs first "so matrix legs always find an existing one", with find-or-create framed as being only "for ad-hoc invocations and as a safety net". After this change that invariant no longer holds on a first-tick-of-month failure — the fallback becomes a real code path, not a safety net. Since that sentence is the load-bearing premise in the PR description, worth reconciling; happy to fold the one-sentence edit into this branch.

Comment thread .github/workflows/review-reviewers.yaml
@tend-agent

Copy link
Copy Markdown
Collaborator Author

Good catch on the skill doc — that sentence was the premise the PR description leaned on, and the change does invalidate it. Both fixes are in 27641c6, alongside the inline N-1 correction.

review-reviewers/SKILL.md now reads: init-tracking makes legs find an existing issue on a normal tick, it is not a precondition, and the find-or-create block is a live code path on the first tick of a month rather than only an ad-hoc fallback. The lowest-numbered-first sort keeps its stated reason — a lost race degrades to a duplicate rather than a crash — which is exactly the property this change relies on.

The PR description's "at most one duplicate tracking issue" is corrected to N-1 there as well, and the sentence quoting the old "safety net" wording now points at the updated text instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

claude-behavior Behavioral issues found by review-reviewers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants