[safe-output-health] 🏥 Safe Output Health Report - 2026-05-21

[github-actions[bot]]

Executive Summary

• Period: Last 24 hours (ending 2026-05-21)
• Runs Analyzed: 73
• Workflows Active: 21
• Safe Output Jobs Executed: 48
• Safe Output Jobs With Failures: 1
• Total Safe-Output Messages Processed: 88
• Failed Safe-Output Messages: 5
• Overall Safe Output Success Rate: 97.9%
• Error Clusters Identified: 1

Safe-output health is strong: 47 of 48 dispatched jobs completed all messages successfully. One run (§26206536101 — Matt Pocock Skills Reviewer) hit a single root cause that failed all 5 of its messages because PR review submission is atomic.

Safe Output Job Statistics

Job Type	Total Calls	Failures	Success Rate
create_pull_request_review_comment	~14	4	~71%
submit_pull_request_review	~3	1	~67%
add_comment	~28	0	100%
create_issue	~5	0	100%
update_issue	~5	0	100%
add_labels	~6	0	100%
create_pull_request	~1	0	100%
missing_tool / missing_data / noop / report_incomplete	~26	0	100%

Counts are derived from per-run "Total messages" lines in the safe_outputs job logs; types are approximated from handler-config and per-message traces. Only the create_pull_request_review_comment / submit_pull_request_review pair shows any failures, all within a single run.

Error Clusters

Cluster 1: review_path_unresolved_422 — PR review submission atomic-failure

• Count: 5 message failures across 1 run

• Affected Jobs: create_pull_request_review_comment (4), submit_pull_request_review (1)

• Affected Workflows: Matt Pocock Skills Reviewer

• Affected Run: §26206536101 on PR refactor(parser): break up oversized functions in pkg/parser to satisfy 60-line limit #33683

• Sample Error:

  POST /repos/github/gh-aw/pulls/33683/reviews - 422
  ##[error]Failed to submit PR review: Unprocessable Entity: "Path could not be resolved"
  

• Root Cause: The handler buffers each create_pull_request_review_comment individually (each "Message N completed successfully"), then submits all 4 as a single PR review at the end. GitHub's POST /pulls/{n}/reviews is atomic — if any one comment references a path/line not present in the diff, the entire review is rejected and every buffered comment is reported as failed. Candidate offending paths in this run: pkg/parser/import_bfs.go:35, :97, :118, and docs/adr/33683-extract-helpers-to-satisfy-function-line-limit.md:48. The .md file was likely a new file the agent invented or commented on a line outside the diff hunks.

• Impact: High for this run (all skill-review feedback dropped). Low repo-wide (1/48 jobs).

Root Cause Analysis

API-Related Issues

The only error category in the window is a GitHub REST 422 from POST /pulls/{n}/reviews. No rate-limit hits (the run's pre-check showed 13390/15000 remaining headroom — 89%), no auth failures, no network timeouts.

Data Validation Issues

The failure is effectively a missing pre-flight diff validation step in the safe-output handler. The handler trusts the agent's (path, line) coordinates until submission time. The error message returned by GitHub does not identify which comment was offending, which forces the whole batch to fail without diagnostic info.

Permission Issues

None observed. The same job had Resource not accessible by integration warnings in adjacent Design Decision Gate runs for branch protection reads, but those are warnings, not safe-output failures, and they did not affect output processing.

Other Issues

No parsing, JSON, or handler-manager bugs were observed. All 88 messages were parsed and dispatched to handlers correctly. The 5 failures occurred purely at the GitHub API submit stage.

Recommendations

Critical Issues (Immediate Action Required)

None. Failure is isolated to one run and the affected workflow's other invocations succeeded.

Bug Fixes Required

1. Atomic PR review failure swallows all comments
   • File/Location: safe_output_handler_manager.cjs — PR review finalization block (=== Finalizing PR Review ===)
   • Problem: A single bad (path, line) causes POST /pulls/{n}/reviews to 422; manager records all buffered comments as failed without identifying which one is at fault.
   • Fix: Before submitting, fetch GET /repos/{owner}/{repo}/pulls/{n}/files and verify each buffered comment's path exists in the response and its line/position falls inside a diff hunk. Drop invalid ones (logging which) and submit the rest. On 422 from GitHub, fall back to submitting comments individually so the offending entry is isolated.
   • Affected Jobs: create_pull_request_review_comment, submit_pull_request_review

Configuration Changes

None recommended. The Matt Pocock Skills Reviewer config (max=10, side=RIGHT) is reasonable.

Process Improvements

1. Improve agent-side path discipline

   • Current State: The agent emits review comments referencing files (including a newly authored ADR doc at docs/adr/33683-extract-helpers-to-satisfy-function-line-limit.md) that are not present as modified lines in the PR diff.
   • Proposed: When emitting create_pull_request_review_comment, the agent prompt for Matt Pocock Skills Reviewer should include the explicit constraint: "Only comment on lines that appear in the PR diff. If you want to suggest a new file, use add_comment (issue comment) instead."
   • Benefits: Eliminates the most common cause of Path could not be resolved 422s without code changes.

2. Better error surfacing in handler manager

   • Current State: 5 identical "Path could not be resolved" errors with no path attribution.
   • Proposed: Wrap each comment in a try-individual fallback on review-level 422, so the offending path is named in the manifest and step summary.

Work Item Plans

Work Item 1: Pre-validate PR review comment paths against PR diff

• Type: Bug Fix
• Priority: Medium (isolated to PR-review safe outputs; PR-review handler is in heavy use)
• Description: Add a pre-submission validation pass in the PR review handler that fetches the PR's changed files + diff hunks once, then verifies each buffered review comment's (path, line/position) lands within a hunk. Invalid comments are dropped (with manifest entries) so they don't take down the whole review.
• Acceptance Criteria:
  • Handler fetches GET /repos/{owner}/{repo}/pulls/{n}/files once per review submission
  • Each buffered comment is validated against returned paths and hunk ranges before POST /pulls/{n}/reviews
  • Invalid comments produce a clear per-comment error message naming the offending path/line
  • Valid comments still submit successfully when others are dropped
  • Unit test reproduces the 422 case from run 26206536101
• Technical Approach: Extend create_pull_request_review_comment handler buffer flush in safe_output_handler_manager.cjs. Use existing getOctokit to call pulls/files; build a Set<path> and a Map<path, Array<{start,end}>> of hunk ranges; filter buffered comments through this before the review POST.
• Estimated Effort: Small (~1 day)
• Dependencies: None

Work Item 2: Individual-comment fallback when atomic review POST 422s

• Type: Enhancement
• Priority: Low (covered by Work Item 1, but useful as defense-in-depth)
• Description: When POST /pulls/{n}/reviews returns 422 "Path could not be resolved", fall back to submitting each buffered comment individually so the offending entry is isolated and the rest survive.
• Acceptance Criteria:
  • On 422, handler retries comments one-by-one
  • Step summary lists which comments succeeded and which failed with the exact path
  • Submit_pull_request_review still attaches if at least one comment lands
• Technical Approach: Wrap the bulk submit in try/catch; on 422 with the specific message, iterate pulls/{n}/comments calls.
• Estimated Effort: Medium
• Dependencies: Coordinate with Work Item 1

Work Item 3: Agent prompt constraint for Matt Pocock Skills Reviewer

• Type: Process Improvement
• Priority: Low
• Description: Update the workflow's prompt to instruct the agent to restrict review comments to lines that appear in the PR diff; route file-level suggestions for new files through add_comment instead.
• Acceptance Criteria:
  • Prompt includes explicit constraint about diff-line coverage
  • No further 422 "Path could not be resolved" from this workflow in next audit window
• Estimated Effort: Small
• Dependencies: None

Historical Context

This is the first Safe Output Health audit recorded in cache memory (/tmp/gh-aw/cache-memory/safe-output-health/). No prior baseline to compare. Pattern review_path_unresolved_422 added to error-patterns.json for future trend tracking.

Trends

• Error rate trend: N/A (no baseline)
• Most common recurring issue: N/A (first audit)
• Improvement since last audit: N/A

Metrics and KPIs

• Overall Safe Output Success Rate (run-level): 97.9% (47/48 jobs with zero failed messages)
• Overall Safe Output Success Rate (message-level): 94.3% (83/88 messages succeeded)
• Most Reliable Job Types: add_comment, create_issue, update_issue, add_labels, noop — 100% in this window
• Most Problematic Job Type: create_pull_request_review_comment (atomic submission risk)
• Average Safe-Output Job Duration: <2s after dispatch (Process Safe Outputs step typically completed in ~300ms–1s)

Next Steps

• File Work Item 1 (pre-validate paths) as a tracked issue/PR in github/gh-aw
• Update Matt Pocock Skills Reviewer prompt with diff-line discipline note (Work Item 3)
• Confirm the review_path_unresolved_422 cluster does not recur in the next 24h audit
• Cross-link this audit with cache memory entry safe-output-health/2026-05-21.json

References:

• §26206536101 — Matt Pocock Skills Reviewer (sole failing safe-output job)
• §26206535616 — Design Decision Gate (safe outputs succeeded; agent error noted but out of scope)
• §26207620211 — this Safe Output Health Monitor run

Generated by 🔒 Safe Output Health Monitor · ● 17.6M · ◷

• expires on May 22, 2026, 5:48 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Safe Output Health Monitor.

A newer discussion is available at Discussion #33948.