[safe-output-health] Safe Output Health Report — 2026-05-27

[github-actions[bot]]

Executive Summary

• Period: Last 24 hours (2026-05-26 ~05:30Z → 2026-05-27 ~05:40Z)
• Runs Analyzed: 81
• Runs With Safe-Output Job Executed: 56
• Safe-Output Jobs Failed (≥1 message): 3
• Total Failed Messages: 5
• Safe-Output Success Rate: 94.6% (runs) / ~96% (messages)
• Soft Recoveries: 2 (approve-on-own-PR → event=COMMENT)
• Headline Event: review_path_unresolved_422 REGRESSION — body-only fallback predicate matches only "Line could not be resolved", not today's "Path could not be resolved" variant. Three messages failed hard on §26491875017 (Smoke Copilot).

The three-day clean streak (2026-05-23 / 24 / 26) ended today with three independent safe-output failures. The most consequential is a partial regression of the previously-validated review_path_unresolved_422 remediation — the soft-recovery branch fires only on one of the two possible 422 error-message variants. Two of today's failures share a root-cause family with cluster target_star_review_comment_no_pr_number_fallback (2026-05-22), now exhibited on the add_comment handler.

Safe-Output Job Statistics

Job Type	Executions (msgs)	Failed	Notes
create_pull_request_review_comment	14+	2	Both rolled-back when atomic review submission 422'd on PR #35117
submit_pull_request_review	12+	1	422 "Path could not be resolved" — fallback didn't fire
add_comment	~20	1	target="*" without item_number on Contribution Check
push_to_pull_request_branch	2	1	Size-limit enforcement: 6100 KB > 1024 KB
create_issue / create_discussion / create_pull_request / add_labels / set_issue_type / update_pull_request / noop / missing_data / others	50+	0	All paths healthy

Critical Issue (Always Visible)

🚨 Cluster review_path_unresolved_422 — Body-only fallback predicate too narrow

• Cluster occurrence: 4 across 2026-05-21 → 2026-05-27 (1 hard / 2 soft / 1 hard)
• Today's run: §26491875017 — Smoke Copilot, PR Bump AWF Firewall to v0.25.56 and MCP Gateway to v0.3.20 #35117
• Error: Unprocessable Entity: "Path could not be resolved and Path could not be resolved"
• Outcome: Three messages failed (2× create_pull_request_review_comment + 1× submit_pull_request_review). Body-only fallback did not fire.
• Root cause (verified in code): actions/setup/js/pr_review_buffer.cjs:554 — predicate is errorMessage.includes("Line could not be resolved") && comments.length > 0. GitHub's review API returns both "Line could not be resolved" and "Path could not be resolved". Today's was the Path variant; the fallback retry block at lines 555–583 was never entered.
• Fix (one-line): change the predicate to

  if ((errorMessage.includes("Line could not be resolved") || errorMessage.includes("Path could not be resolved")) && comments.length > 0) {

• Test gap: pr_review_buffer.test.cjs:785 covers the Line variant; a mirror test for the Path variant should be added to lock in the dual match.

Why the existing pre-validation (Sub-pattern B) didn't catch it

pr_review_buffer.cjs:324 validates comment paths against pulls.listFiles before submission and drops invalid paths. Today's two buffered comments survived pre-validation (one on pkg/constants/version_constants.go:111), so the paths were in the diff listing — but GitHub still rejected them at submit time with "Path could not be resolved". This indicates the API rejects reviews when the resolved position is non-reviewable (e.g. binary diff, renamed/generated file, line outside the hunk range) even when listFiles reports the path. Hence the body-only retry remains the correct safety net; the predicate just needs to match both error variants.

Error Clusters This Audit

Cluster 1 — review_path_unresolved_422 (regression, see Critical above)

• Count: 1 occurrence (4 cumulative)
• Affected Job: submit_pull_request_review (cascades to rolled-back create_pull_request_review_comment items)
• Affected Workflow: Smoke Copilot
• Run: §26491875017
• Sample log:

  POST /repos/github/gh-aw/pulls/35117/reviews - 422 ...
  ##[error]Failed to submit PR review: Unprocessable Entity: "Path could not be resolved and Path could not be resolved"
  ##[error]3 safe output(s) failed:
    - create_pull_request_review_comment: Review finalization failed: ...
    - create_pull_request_review_comment: Review finalization failed: ...
    - submit_pull_request_review: Review finalization failed: ...
  

• Impact: High — atomic review submission drops all buffered comments; the body-only safety net exists but is bypassed for the Path variant.

Cluster 2 — target_star_add_comment_no_item_number_fallback (new)

• Count: 1 occurrence (newly identified)
• Affected Job: add_comment
• Affected Workflow: Contribution Check (schedule-triggered)
• Run: §26485276389
• Sample log:

  ##[warning]Target is "*" but no item_number/issue_number/pull_request_number/pr_number/pr/pull_number specified in add_comment item
  ##[error]✗ Message 1 (add_comment) failed: Target is "*" but no item_number/issue_number/pull_request_number/pr_number/pr/pull_number specified in add_comment item
  

• Root cause: The add_comment handler does not resolve a fallback target the way update_pull_request / submit_pull_request_review do. For schedule-triggered runs there is no triggering issue/PR (so a true fallback is impossible), but the error message should steer the agent toward create_issue for periodic reports, or require an explicit item_number.
• Sibling of: latent cluster target_star_review_comment_no_pr_number_fallback (2026-05-22)
• Impact: Per-message; the other 2 messages (create_issue, add_labels) succeeded.

Cluster 3 — push_to_pull_request_branch size-limit enforcement

• Count: 1 occurrence
• Affected Job: push_to_pull_request_branch
• Affected Workflow: Smoke Claude
• Run: §26491874960
• Error: Incremental diff size (6100 KB) exceeds maximum allowed size (1024 KB). Bundle size: 1 KB.
• Behavior: Working as designed — size enforcement fired; the warning text correctly identified this as a code-push failure (no WTD3 conflation). The remaining 12 messages all processed.
• Worth flagging: 6100 KB is anomalously large for a smoke run. Either the agent intentionally synthesized an oversized patch or a stray artifact was included.

Soft Recoveries (working as designed)

2 occurrences of approve-on-own-PR → event=COMMENT fallback

• §26491903660 — Test Quality Sentinel: Cannot submit APPROVE review on own PR. Retrying with event=COMMENT. → review #4369550373 created.
• §26492335167 — Matt Pocock Skills Reviewer: same fallback → review #4369603428 created.

Recommendations

Critical (Immediate)

1. Patch pr_review_buffer.cjs:554 to match both 422 error-message variants
   • Priority: Critical
   • Root cause: predicate matches only "Line could not be resolved"; misses "Path could not be resolved"
   • Action: One-line change (see Critical Issue section above) + matching unit test in pr_review_buffer.test.cjs
   • Affected: all workflows submitting buffered PR reviews (Smoke Copilot/Claude, PR Code Quality Reviewer, Matt Pocock Skills Reviewer, etc.)

High

2. Unify target resolution in add_comment handler (and tighten error message)
   • Priority: High
   • File: actions/setup/js/safe_output_handler_manager.cjs — add_comment handler
   • Action: When target: "*" and no item_number is provided, fall back to triggering issue/PR from GITHUB_REF / GITHUB_EVENT (same as update_pull_request). For schedule-triggered contexts (no triggering issue/PR), upgrade the error to suggest create_issue or an explicit item_number.
   • Affected: Contribution Check (and any other schedule-triggered workflow using add_comment with target: "*")

Medium

3. Investigate the 6100 KB Smoke Claude patch
   • Priority: Medium
   • Action: Determine whether §26491874960 intentionally exercised the size-limit branch or whether an agent-side artifact was unexpectedly included in the patch. If intentional, consider raising the max for the smoke workflow only; if unintentional, fix the agent's patch construction.

Work Item Plans

Work Item 1 — Fix review_path_unresolved_422 body-only fallback predicate

• Type: Bug Fix
• Priority: Critical
• Description: The body-only retry branch in pr_review_buffer.cjs:554 matches only one of GitHub's two 422 error-message variants for unresolvable review comments. Today's §26491875017 regressed three messages because the "Path could not be resolved" variant slipped past the predicate.
• Acceptance Criteria:
  • pr_review_buffer.cjs:554 predicate matches "Path could not be resolved" in addition to "Line could not be resolved"
  • New unit test in pr_review_buffer.test.cjs exercises the Path variant and asserts body-only retry + success
  • Existing Line-variant test still passes (pr_review_buffer.test.cjs:785)
  • CI run on the patched branch reproduces a body-only fallback when a Path-422 is mocked
• Technical Approach: One-line || insertion in the existing condition; mirror existing test case substituting the error message
• Estimated Effort: Small (≤ 30 min)
• Dependencies: None

Work Item 2 — add_comment target="*" fallback to triggering context (or clearer error)

• Type: Enhancement
• Priority: High
• Description: add_comment handler does not fall back to the triggering issue/PR when target: "*" and no item_number is provided, unlike sibling handlers. For schedule-triggered runs there is no triggering target, but the error message should redirect the agent rather than leaving it stuck.
• Acceptance Criteria:
  • When triggering context is a PR/issue, the add_comment handler resolves target: "*" to that PR/issue automatically
  • When triggering context is schedule/workflow_dispatch, error message explicitly suggests create_issue or an explicit item_number
  • Unit test covers the schedule-triggered failure path with the new error message
• Technical Approach: Mirror the resolution logic from update_pull_request / submit_pull_request_review handlers
• Estimated Effort: Small–Medium
• Dependencies: None

Work Item 3 — Investigate Smoke Claude oversized patch

• Type: Investigation
• Priority: Medium
• Description: Determine whether §26491874960's 6100 KB incremental diff is expected smoke coverage of the size-limit branch or an agent-side construction bug
• Acceptance Criteria:
  • Identify whether the smoke workflow intentionally exercises the >1024 KB scenario
  • If yes, document it in the workflow; if no, fix the agent's patch construction
• Estimated Effort: Small
• Dependencies: Smoke Claude workflow author input

Historical Context

Date	Runs (safe-out exec)	Failed Runs	Success %	Headline
2026-05-21	48	1	97.9	review_path_unresolved_422 (hard, Line)
2026-05-22	24	1	95.8	target_star_review_comment cluster identified; body-only fix landed
2026-05-23	8	0	100.0	clean (low volume)
2026-05-24	6	0	100.0	clean
2026-05-26	47	0	100.0	review_path_unresolved_422 soft-recovered (Line); cancellation counter mislabel identified
2026-05-27	56	3	94.6	review_path_unresolved_422 regression (Path); target_star sibling on add_comment

Trends

• Error rate trend: Spiked today after three clean days. Two of three failures map to known root-cause families (review 422 fallback gap; target="*" resolution gap on add_comment).
• Recurring patterns: review_path_unresolved_422 now at 4 occurrences with one regression — the remediation needs widening.
• Coverage: 56 runs with safe-outputs executed is the highest in the 6-day baseline.

Metrics & KPIs

• Overall Safe-Output Success Rate (24h): ~96% messages / 94.6% runs
• Most Reliable Job Types: create_issue, create_discussion, create_pull_request, add_labels, update_pull_request, noop — 0 failures observed today
• Most Problematic Job Type: review submission family (submit_pull_request_review + create_pull_request_review_comment rollback) — 3 of 5 failed messages
• Soft-recovery utility: approve-on-own-PR → event=COMMENT fired twice cleanly

Next Steps

• Open a bug-fix PR for Work Item 1 (one-line predicate fix + test)
• Open an enhancement issue for Work Item 2 (add_comment target="*" fallback / error message)
• Triage the Smoke Claude 6100 KB patch (Work Item 3)
• On next audit, confirm target_star_add_comment_no_item_number_fallback does not silently recur in other schedule-triggered workflows
• On next audit, watch for cancellation_counter_mislabeled_code_push_failed — not exercised today; remediation still pending

References:

• §26491875017 — Smoke Copilot (review 422 Path variant — regression)
• §26485276389 — Contribution Check (add_comment target="*" sibling cluster)
• §26491874960 — Smoke Claude (push_to_pull_request_branch size limit)

Generated by 🔒 Safe Output Health Monitor · opus47 34.1M · ◷

• expires on May 28, 2026, 5:56 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-05-28T05:56:59.658Z.

Closed by Workflow