fix(report-failure): dedup outage comments per run across matrix legs by tend-agent · Pull Request #809 · max-sixty/tend

tend-agent · 2026-08-03T04:16:12Z

Problem

When a matrix workflow fails during a bot outage, the tend-outage tracker issue is flooded with one near-identical comment per matrix leg, all citing the same run.

review-reviewers has a 5-leg matrix. During an outage this window (claude -p exiting 1 uniformly, self-resolved by 03:57Z), two failed review-reviewers runs each posted ~5–6 comments to #808:

run 30776520686 → 5 comments, all linking .../runs/30776520686
run 30779959756 → 6 comments, all linking .../runs/30779959756

Each comment is a one-row table differing only in a jitter-spread timestamp — the same run recorded 5–6 times.

Root cause

shared/steps/report-failure.sh is invoked once per matrix leg, every leg sharing one GITHUB_RUN_ID. The script already handles the concurrent-leg race on the create path — jittered backoff (#586) plus a self-heal reconcile that closes duplicate issues (#744) — but the append path had no dedup: every leg unconditionally gh issue comments its own row.

So the create-create race (duplicate issues) was solved; the comment-append race (duplicate comments) was not. This is the same class of concurrent-matrix-leg noise the maintainer has fixed repeatedly (#586, #744, and #560 which batches enrichment into one comment per issue) — this closes the remaining gap.

Fix

Symmetric to the existing issue reconcile:

Guard — before appending, skip if this run's URL is already recorded on the issue (body or an existing comment). Collapses most of the flood, since jitter (0–30s) already spreads the legs.
Reconcile — the check-then-act still races across jittered legs, so after posting, keep the earliest comment citing this run and delete later duplicates. Convergent: every racing leg sorts identically and computes the same keeper, so deleting an already-deleted comment is a harmless 404.

Net effect: one row per run, regardless of matrix width. Non-matrix workflows (single leg) are unaffected — the guard finds nothing, posts once, reconcile is a no-op.

Comments deleted are the bot's own auto-generated outage rows; no human content is touched.

Gate assessment

Class: structural — a shell script with no per-run comment dedup. Replayed 10/10 times, every matrix-workflow failure floods. Not a stochastic model lapse; confirmed by reading the script source and the 11 duplicate comments on Bot temporarily unavailable #808.
Evidence: High (structural). 2 distinct failed review-reviewers matrix runs this window, both exhibiting the identical flood (~11 duplicate-run comments total), plus the defect is guaranteed to recur on any future matrix-workflow outage.
Magnitude: targeted fix to one shared script (add a guard + a reconcile mirroring the adjacent issue reconcile) — Normal bar, met.
Passes both gates. The transient outage that triggered the failures is not itself a code fix (it self-resolved by 03:57Z, tracked by Bot temporarily unavailable #808); this fixes the durable noise the outage exposed.

Window & evidence

Analysis run: 30783291243, window ~00:12Z–04:02Z 2026-08-03 (extended back to the last successful review-reviewers analysis, since the 01:17Z and 02:43Z analysis runs both failed in the same outage — leaving a coverage gap the normal window floor didn't reach).
Evidence log: https://gist.github.com/e08f6e62d6478163cb425a75648eb7e4
Other in-window bot output was clean and accepted: PR fix: isolate tend's uv from the adopter's #807 got a bot APPROVE and merged at 00:41Z; corruption scan clean.

A matrix workflow (e.g. review-reviewers, 5 legs) invokes report-failure.sh once per leg on an outage, every leg sharing one GITHUB_RUN_ID. The existing reconcile handles the create-create race (duplicate *issues*) but nothing dedups the append path, so each leg posted its own near-identical row — flooding the tend-outage issue with 5-6 comments all citing the same run. Guard the comment on the run not already being recorded (body or existing comment), then reconcile symmetrically to the issue path: keep the earliest comment citing this run, delete later duplicates. Convergent across concurrently-jittered legs.

tend-agent

The dedup logic is sound and the convergence argument holds (stable sort_by over an id-ascending API response makes every leg pick the same keeper). One robustness gap on the delete path: the reconcile selects comments by a bare contains("${RUN_URL}") substring, with no author or row-format scoping. Any comment that merely contains the run URL — a human quoting it while investigating, or a nightly-enrichment comment (#560) that lists this run among others — would be selected and, if it sorts after the earliest row, deleted. That contradicts the PR's "no human content is touched" invariant, since the code doesn't actually restrict deletion to the auto-generated rows.

Matching the full markdown anchor [workflow run](${RUN_URL}) instead scopes both the guard and the delete to the generated rows only, and as a bonus removes a latent prefix-collision false positive (a bare-URL contains also matches a longer run id that has this one as a prefix). The guard and the reconcile must stay consistent, so both suggestions below apply together.

…s human comments

tend-agent · 2026-08-05T00:18:21Z

New evidence, same defect, roughly 7x the volume. The 2026-08-04 outage (Claude weekly limit, 429 rate_limit, from ~08:51Z until the 00:00Z reset) left #831 with 77 comment rows across 17 distinct runs — 15 review-reviewers runs contributing exactly 5 rows each, one per matrix leg, plus 2 single-leg tend-triage rows. This PR's original evidence was ~11 rows from 2 runs.

Each of those 75 duplicate-leg rows also fired an issue_comment event, so the flood produced 66 tend-mention runs on top of the comments (all correctly gate-skipped, but they are 66 queued jobs). With this PR merged the same outage would have recorded 17 rows instead of 77 — one per run, which is the readable timeline the issue is for.

Evidence log: https://gist.github.com/e08f6e62d6478163cb425a75648eb7e4

tend-agent · 2026-08-05T00:36:04Z

Coordination note from #836, which is now touching the neighbouring branch of this same script.

Review there caught that #836's reconcile carries its row onto the keeper unguarded, so a same-matrix race would post a row duplicating the keeper's seed row — the same flood this PR removes from the EXISTING branch, arriving by a path that bypasses this PR's guard. Fixed in 1c41d39 with the identical four-line check (gh issue view --json body,comments piped to grep -qF "[workflow run](${RUN_URL})").

That means the anchor check now exists twice in the file: once here on the append path, once on the create path. Leaving both copies rather than pre-factoring a helper — the two diffs are textually disjoint as they stand and a helper introduced on either branch would conflict with the other for no benefit until one lands. Whichever of the two merges second should fold them into one helper.

…#823) ## Problem The `Trigger` column of a `tend-outage` row is the only pointer back to the work a failed run stranded. It goes blank for the one trigger where that pointer matters most, and prints `#null` when a field is missing. **`repository_dispatch` is unhandled.** `tend-mention` relays review events through a secretless job that re-posts them as a `repository_dispatch`, so the handle job runs on that event and the PR number arrives as `client_payload.pr` rather than in a `pull_request` object. The `if`/`elif` chain has no branch for it, so every failure on the relay path records `Trigger: N/A` — and a relayed review is exactly the case a maintainer can't recover from the run alone, since `tend-review` fires only on `pull_request_target` and never retries. This path is in constant use: `gh api "repos/max-sixty/tend/actions/runs?event=repository_dispatch"` returns a steady stream of `tend-mention` runs. **`workflow_run` names no run.** The ci-fix path hardcodes `REF="CI fix for workflow run"`, discarding `workflow_run.id` — the id of the CI failure the job was dispatched to fix. **Missing fields render as `null`.** `jq -r '.issue.number'` prints the literal string `null` when the field is absent, so the cell reads `#null` rather than falling back to `N/A`. ## Solution Add a `repository_dispatch` branch reading `client_payload.pr`, link the upstream run id in the `workflow_run` branch, and give every extraction `// empty` plus a `${VAR:+…}` guard so an absent field leaves the cell blank (rendered as `N/A` by the existing `${REF:-N/A}`) instead of `#null`. ## Testing `shared/steps/` has no shell test harness — shellcheck via pre-commit is the only automated gate, and it passes on the changed file. So the block was exercised directly: the REF logic was sliced out of the script (between the `# Build a one-line reference` comment and `TIMESTAMP=`) and sourced under crafted `GITHUB_EVENT_PATH` payloads, before and after. Before, on `origin/main`: ``` repository_dispatch -> N/A workflow_run -> CI fix for workflow run issues (no number) -> #null ``` After: ``` relayed review -> #815 relay w/o pr -> N/A ci-fix -> CI fix for [run 30795510450](https://github.com/max-sixty/tend/actions/runs/30795510450) ci-fix w/o id -> CI fix for workflow run PR event -> #821 PR event w/o number -> N/A issue comment -> #808 issue w/o number -> N/A schedule -> N/A ``` Adding a real harness for `shared/steps/` is worth considering separately — three of the recent outage-path fixes have all landed in scripts nothing can test — but that is a bigger change than this fix warrants, so it is not bundled here. ## Scope Separate from the other two open changes on this path, and textually disjoint from both. #818 names the *cause* of a failure (in `claude/action.yaml`); #809 dedups *rows* across matrix legs (in the `EXISTING` branch of this same script, lines 49+). This one fixes what the row *points at*, in the REF block at lines 19–29. Co-authored-by: tend-agent <270458913+tend-agent@users.noreply.github.com>

…1243 # Conflicts: # shared/steps/report-failure.sh

tend-agent

The guard is correct now — both earlier threads are addressed by 86f39cc, and hoisting the anchor into run_issue_anchor is the right shape: the row and the two dedup matchers can no longer drift apart.

One remaining defect on the reconcile, in the original commit rather than the incremental. The delete path lists comments with ?per_page=100 and no pagination, and GitHub returns issue comments oldest-first — so once a tracker passes 100 comments, the rows this run just posted are not in the response at all and the reconcile silently no-ops. That is the flooded issue the PR exists for: #831 reached 77 rows in a single outage, so crossing 100 on a longer one or a wider matrix is the expected case rather than the edge. The guard is unaffected — gh issue view --json comments does paginate — so only the second line of defence is capped.

--paginate alone will not fix it: gh applies --jq per page, which breaks the cross-page sort_by(.created_at) | .[1:] (each page would keep its own earliest). sort=created&direction=desc is ignored by this endpoint. --paginate --slurp refuses --jq, so the working form is --slurp piped to a downstream jq 'add | …' — which is also what rate-limit-preflight.sh already does one file over for /issues/$PAUSE/events?per_page=100.

Separately, this ships ~40 lines of racy dedup with no test, in a script every adopter runs. generator/tests/test_shared_steps.py already drives rate-limit-preflight.sh and mark-notification-read.sh against a fake gh, and #836 adds a report_failure_env fixture for this exact script — the guard (skip when the anchor is already present) and the reconcile (keep the earliest, delete the rest) both look cheap to cover once whichever of the two lands first.

How the pagination behaviour was verified

Against cli/cli#13840, which has 139 comments:

$ gh issue view 13840 -R cli/cli --json comments --jq '.comments | length'
139
$ gh api "repos/cli/cli/issues/13840/comments?per_page=100" --jq 'length'
100
$ gh api "repos/cli/cli/issues/13840/comments?per_page=100" --jq '[.[0].created_at, .[-1].created_at]'
["2026-07-10T13:54:16Z","2026-08-03T21:08:08Z"]     # oldest-first; newest 39 absent
$ gh api "repos/cli/cli/issues/13840/comments?per_page=100&sort=created&direction=desc" --jq '[.[0].created_at, .[-1].created_at]'
["2026-07-10T13:54:16Z","2026-08-03T21:08:08Z"]     # sort/direction ignored
$ gh api --paginate "repos/cli/cli/issues/13840/comments?per_page=100" --jq 'length'
100
39                                                   # --jq runs per page
$ gh api --paginate --slurp "repos/cli/cli/issues/13840/comments?per_page=100" --jq 'length'
the `--slurp` option is not supported with `--jq` or `--template`
$ gh api --paginate --slurp "repos/cli/cli/issues/13840/comments?per_page=100" | jq -r 'add | length'
139

The empty case needs no extra guard: on a zero-comment issue --paginate --slurp yields [[]], so add gives [] and the filter emits nothing, exit 0.

Issue comments come back oldest-first, so the unpaginated `?per_page=100` read returned only the oldest page. Past 100 comments the rows this run and its racing sibling just posted are not in the response at all and the reconcile silently no-ops — on exactly the flooded trackers it exists for (#831 reached 77 rows in one outage). `--paginate` alone doesn't fix it: `gh` applies `--jq` per page, so each page would keep its own earliest comment and `sort_by | .[1:]` would delete the keeper. `--slurp` refuses `--jq`, so the filter moves downstream with `add` flattening the pages. Tests cover both halves of the dedup against a fake `gh`, alongside the existing `rate-limit-preflight.sh` and `mark-notification-read.sh` suites: the guard skipping a run already recorded, the reconcile converging two racing legs to one row, the anchor scoping that keeps a human comment naming the run out of the delete set, and the pagination case — which is the only one of the five that fails against the pre-fix script.

tend-agent · 2026-08-07T07:24:56Z

Both points addressed in 834734e.

Pagination — replied inline; took the suggestion as written.

Tests — added five against a fake gh, in the same shape as the existing rate-limit-preflight.sh suite (fixtures in, the script's own filters doing the work). Since neither this nor #836 has landed, this one carries its own report_failure_env rather than waiting on the other:

Test	Covers
`skips_a_run_already_recorded`	the guard — a leg whose sibling already posted this run's row posts nothing
`appends_a_row_for_an_unrecorded_run`	the guard doesn't swallow a genuinely new run
`reconciles_a_racing_leg`	two legs that both missed the guard converge to one row
`reconciles_past_the_first_page`	the pagination defect: 139 comments, the racing row at the end
`leaves_a_human_comment_naming_the_run`	the anchor scoping from `86f39cc` — a person linking the run isn't a delete candidate

Only reconciles_past_the_first_page fails against the pre-fix script; the other four pass on both, which is the intended split — one regression test for the defect just fixed, four characterising what the original commit already did.

The fake pages the comment list the way the endpoint does (--slurp gets the array of pages, a plain read gets the oldest 100 alone) rather than special-casing the flag the script happens to pass, so that counterfactual is honest rather than a fake built to agree with the fix.

Coordination with #836 — both PRs now add a report_failure_env fixture and a FAKE_GH_REPORT_FAILURE to the end of generator/tests/test_shared_steps.py, so whichever lands second will conflict there. I kept every shared name and env var identical to #836's (REPORT_FAILURE, OUTAGE_TITLE, OUTAGE_LABEL, RUN_LINK, OPEN_ISSUES_JSON, KEEPER_JSON, COMMENT_BODIES, FAKE_NEW_ISSUE) so the resolution is a union rather than a rewrite: take both sets of case arms in the fake — #836's repos/*/issues/N primary-key probe and this one's /comments?per_page=100 pager — and both sets of tests. The two test sets don't overlap: #836's exercise the create path, these the append path.

356 passed  (generator/)
pre-commit: shellcheck, ruff, typos — all pass

tend-agent

The pagination fix is right and the regression test is honest — I re-ran test_report_failure_reconciles_past_the_first_page against 4e93d87's script and it fails there (deleted [], expected ['140']), passes here. gh api --paginate --slurp on a zero-comment issue does return [[]] as the comment claims (checked against #879 in this repo), so add → [] → no output, exit 0.

Two gaps in what the new suite pins down, both verified by mutating the script and re-running:

The fake accepts --slurp with --jq; real gh exits 1 on it. That combination is the exact trap the script's comment exists to warn about, and the fake currently blesses it.
The guard's issue-body half is never exercised — _seen_by_the_guard hardcodes body: "", so every case reaches the guard through the comments list only. That's the minority path: on the first failed run of an outage one leg creates the issue with its row in the body, and its four siblings arrive at a tracker with no comments at all, matchable only on the body.

Suggestions inline. The second and third apply together — the keyword arg alone is inert, the parametrize alone is a TypeError.

How each gap was confirmed

Two mutations of shared/steps/report-failure.sh, each run against the suite as it stands and against the suite with the suggestions applied:

Mutation	As it stands	With suggestions
`\| jq -r "add \| …"` → `--jq "add \| …"` (fold the filter back into `gh`)	5 passed	4 failed — `returncode=1`, `stderr='the --slurp option is not supported with --jq or --template'`
guard's `--jq` loses `.body + "\n" +`	5 passed	1 failed — `skips_a_run_already_recorded[in-the-issue-body]`

And the underlying gh behaviour, on 2.97.0:

$ gh api --paginate --slurp "repos/max-sixty/tend/issues/879/comments?per_page=100"
[[]]
$ gh api --paginate --slurp "repos/max-sixty/tend/issues/879/comments?per_page=100" --jq 'length'
the `--slurp` option is not supported with `--jq` or `--template`
$ echo $?
1

Also checked: --slurp is new to this repo — rate-limit-preflight.sh:114 pairs --paginate with a streaming .[] | select(…) filter, which is safe per-page and so isn't the precedent the earlier review cited. Nothing to fix there; the filter has no cross-page reduction.

Not flagged, for the record: _nwise in the fake is a private jq builtin, but a removal would break the tests loudly rather than silently. #857 also touches this file, but only its header comment.

… guard Two gaps in the suite added a commit ago, both found by mutating the script: The fake applied `--jq` whatever flags came with it, so `--slurp --jq` was accepted in the test and rejected in production. That combination is the trap the reconcile's comment warns about — folding the filter back into `gh --jq` is the obvious simplification, and it left all five tests green while the script died under pipefail right after posting its row, never reconciling. The fake now refuses it the way `gh` does. The guard matches `.body + "\n" + comments`, but every case reached it through the comments list, so dropping the body half from the script's `--jq` left the suite green. That half carries the first failed run of an outage: one leg takes the `else` branch and seeds the issue with its row in the body, and its siblings then hit the `EXISTING` branch against a tracker with no comments at all. Parametrized, at the cost of one extra case. Confirmed by re-running each mutation: the first now fails 4 tests, the second fails `skips_a_run_already_recorded[in-the-issue-body]`.

tend-agent

Both gaps from the last review are closed in bb0a92d, re-derived here rather than taken on the commit message's word — I re-applied each mutation to shared/steps/report-failure.sh against this HEAD:

Mutation	Result on `bb0a92d`
`\| jq -r "add \| …"` → `--jq "add \| …"`	4 failed, `returncode=1`, `stderr='the --slurp option is not supported with --jq or --template'`
guard's `--jq` loses `.body + "\n" +`	1 failed — `skips_a_run_already_recorded[in-the-issue-body]` only

The fake's refusal also matches real gh on this runner (2.96.0): gh api --slurp --jq '.' … exits 1 with that message, and it is a parse-time rejection, so placing the check ahead of the subcommand dispatch is the right position for it. The body-half case is faithful too — the else branch seeds the issue through run_issue_create_and_reconcile with $ROW, which carries the anchor, so a sibling leg really does arrive at a tracker whose only match is the body.

357 passed in generator/, ci green on this commit. No further findings; not approving only because the PR is self-authored.

tend-agent added the claude-behavior Behavioral issues found by review-reviewers label Aug 3, 2026

tend-agent commented Aug 3, 2026

View reviewed changes

Comment thread shared/steps/report-failure.sh Outdated

Comment thread shared/steps/report-failure.sh Outdated

fix(report-failure): match generated-row anchor so dedup never touche…

86f39cc

…s human comments

This was referenced Aug 4, 2026

fix(report-failure): name the trigger a stranded outage row points at #823

Merged

review-runs-tracking: 2026-08 #801

Open

This was referenced Aug 5, 2026

fix(report-failure): reconcile duplicate outage issues by primary-key probe, not a lagging list #836

Open

fix(list-recent-runs): recover the window on the last successful run, and stop the fetch limit re-truncating it #838

Open

worktrunk-bot mentioned this pull request Aug 5, 2026

skills(review-runs): drain triggers stranded by a failed session #851

Open

This was referenced Aug 6, 2026

fix(actions): report an outage when the run fails ahead of the agent #857

Open

report-failure: a persistent pre-agent failure appends a row to the outage issue on every trigger #859

Open

Merge remote-tracking branch 'origin/main' into hourly/review-3078329…

4e93d87

…1243 # Conflicts: # shared/steps/report-failure.sh

tend-agent commented Aug 7, 2026

View reviewed changes

Comment thread shared/steps/report-failure.sh Outdated

tend-agent commented Aug 7, 2026

View reviewed changes

Comment thread generator/tests/test_shared_steps.py

Comment thread generator/tests/test_shared_steps.py Outdated

Comment thread generator/tests/test_shared_steps.py Outdated

tend-agent commented Aug 7, 2026

View reviewed changes

This was referenced Aug 7, 2026

skills(review-reviewers): sweep the window repo-wide before trusting a silent survey #885

Open

fix(review-runs,review-reviewers): dedup against merged PRs, and against tend before filing upstream #891

Open

Navigation Menu

fix(report-failure): dedup outage comments per run across matrix legs - #809

fix(report-failure): dedup outage comments per run across matrix legs#809
tend-agent wants to merge 5 commits into
mainfrom
hourly/review-30783291243

tend-agent commented Aug 3, 2026

Uh oh!

tend-agent left a comment

Uh oh!

Uh oh!

Uh oh!

tend-agent commented Aug 5, 2026

Uh oh!

tend-agent commented Aug 5, 2026

Uh oh!

tend-agent left a comment

Uh oh!

Uh oh!

tend-agent commented Aug 7, 2026

Uh oh!

tend-agent left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tend-agent left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

tend-agent commented Aug 3, 2026

Problem

Root cause

Fix

Gate assessment

Uh oh!

tend-agent left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tend-agent commented Aug 5, 2026

Uh oh!

tend-agent commented Aug 5, 2026

Uh oh!

tend-agent left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tend-agent commented Aug 7, 2026

Uh oh!

tend-agent left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tend-agent left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant