[safe-output-health] Safe Output Health Report - 2026-05-23 (clean day, 100% success)

[github-actions[bot]]

Executive Summary

• Period: Last 24 hours (2026-05-22 ~04:00 UTC → 2026-05-23 ~05:30 UTC)
• Repository: github/gh-aw
• Runs with a safe_outputs job slot: 21
• safe_outputs jobs actually executed: 8 (13 gated off by safe_outputs.if)
• safe_outputs jobs failed: 0
• Safe-output messages processed: 7 successful, 0 failed (100% success rate)
• Error clusters identified: 0 (clean day)

This is a clean audit day. All 8 safe_outputs jobs that ran completed with Failed: 0 and emitted no ##[error] / ##[warning] entries. Three handler types were exercised in production beyond signaling: create_issue, create_discussion, and create_pull_request (PR #34165 was created and merged end-to-end). Caveat: today is Saturday with materially lower workflow volume (8 executions vs. 24 yesterday, 48 the day before), and the Smoke Claude/Copilot workflows that produced the prior two days' issue clusters did not run, so the two open patterns are dormant-but-untested rather than confirmed fixed.

Safe Output Job Statistics

Job Type	Executions	Failures	Success Rate
noop	4	0	100%
create_issue	1	0	100%
create_discussion	1	0	100%
create_pull_request	1	0	100%
(jobs with 0 messages)	2	0	n/a
Total	9 messages across 8 jobs	0	100%

Per-run breakdown (8 executed safe_outputs jobs)

Run	Workflow	Msgs	OK	Fail	Types	Notes
§26323913381	sergo (Serena Go Expert)	2	2	0	create_issue, create_discussion	Created #34171 (issue) + #34172 (discussion)
§26323431910	go-logger	1	1	0	create_pull_request	Created PR #34165 — merged to main (commit 5288def)
§26324183191	smoke-ci	1	1	0	noop	Push event, no PR context
§26323805965	pr-sous-chef	1	1	0	noop	Run summary recorded
§26323418142	daily-astrostylelite-markdown-spellcheck	1	1	0	noop	No valid fixes after analysis
§26323201498	design-decision-gate	1	1	0	noop	—
§26323916376	ai-moderator	0	0	0	(none)	Agent emitted no messages — handler exited cleanly
§26323643312	release	0	0	0	(none)	Agent emitted no messages — handler exited cleanly

Why 13 of 21 safe_outputs slots were skipped

These runs are not failures — they are working-as-intended gating. The safe_outputs.if condition is:

((!cancelled()) && (needs.agent.result != 'skipped') && (needs.detection.result == 'success'))

In all 13 cases, either:

• needs.agent.result == 'skipped' (agent never ran), or
• needs.detection.result != 'success' (detection job failed or was skipped — these are out of scope for safe-output health and are tracked by a separate monitor).

Examples: runs 26324232375, 26324236738, 26324231633, 26324146164, 26323929451, 26323916405/390/361, 26323914272, 26324183213, 26323901865, 26323390667, 26323206790. All exited at the if-evaluation step with Result: false and no further work.

Error Clusters

None identified for this audit window.

A repository-wide grep across all 8 executed safe_outputs job logs for ##[error], ##[warning], ✗ Message, Failed: [1-9], and exited with code [1-9] returned zero matches. The 12_/13_/8_ Process Safe Outputs step in every executing run ended with the === Processing Summary === block reporting Failed: 0 and "Safe Output Handler Manager completed".

Status of Previously-Identified Clusters

review_path_unresolved_422 (first seen 2026-05-21)

• Today's count: 0 messages exercised this code path (no submit_pull_request_review messages emitted in the last 24h).
• Status: Dormant, undertested today. Remediation (body-only retry fallback) demonstrated working on 2026-05-22 (run 26269860137). No regression evidence because the path was not exercised.
• Recommendation: Continue monitoring. Consider the pre-validation enhancement from the cache memory's solutions.json as a Medium-priority follow-up — would save the recovery round-trip and surface clearer errors.

target_star_review_comment_no_pr_number_fallback (first seen 2026-05-22)

• Today's count: 0 messages exercised this code path (no create_pull_request_review_comment messages emitted in the last 24h; Smoke Claude did not run).
• Status: Latent / open / unfixed. The handler gap identified yesterday in run 26269897290 was not exercised today, so we cannot say it is fixed — only that it did not surface.
• Recommendation: This remains the highest-priority open work from yesterday's audit. See "Work Items" below.

Root Cause Analysis

No new root causes to analyze today. The two open patterns from prior audits are preserved verbatim in /tmp/gh-aw/cache-memory/safe-output-health/error-patterns.json and remain candidates for upstream remediation regardless of today's clean reading.

Recommendations

Critical Issues (Immediate Action Required)

None. No active incidents in the audit window.

Bug Fixes Required (Carried Forward from 2026-05-22)

1. Unify target:"*" resolution across safe-output handlers (highest priority carryover)

   • File / location: actions/safe_output_handler_manager.cjs, create_pull_request_review_comment handler branch that triggers when item explicitly carries target:"*".
   • Problem: When an item carries target:"*" and no pull_request_number, sibling handlers (update_pull_request, submit_pull_request_review) auto-resolve to the triggering PR; create_pull_request_review_comment rejects. Sibling code paths inside the same handler do auto-resolve correctly when target is omitted, so the gap is the explicit-"*" branch.
   • Fix: Extract a shared resolveTargetPRNumber(item, ctx) helper used by all three handlers. When target === "*" and item.pull_request_number is absent, fall back to ctx.triggering_pr_number (from GITHUB_REF: refs/pull/{n}/merge).
   • Affected jobs: create_pull_request_review_comment.
   • Effort: Small. Priority: High (single-source-of-truth prevents this whole class of inconsistency).

2. Improve error-message granularity for handler rejections (carried)

   • When rejecting an item, include the message index and the problematic field (e.g., Message 2 (create_pull_request_review_comment): target is "*" but item has no pull_request_number).
   • Effort: XS. Priority: Medium.

Configuration / Process Improvements

3. Pre-validate review-comment paths against PR diff (carried, prevention layer for review_path_unresolved_422)
   • Before buffering review comments, fetch PR files once and verify each (path, position/line) maps to a diff hunk. Reject invalid items individually with precise errors rather than letting the bulk POST /pulls/{n}/reviews fail.
   • Effort: Small. Priority: Medium (handler already recovers via body-only fallback; this saves a round trip and gives agents clearer feedback for self-correction).

Work Item Plans

Work Item 1: Unify target:"*" resolution in safe-output handlers

• Type: Bug Fix
• Priority: High
• Description: Extract a single resolveTargetPRNumber(item, ctx) helper used by create_pull_request_review_comment, update_pull_request, and submit_pull_request_review. When target === "*" and no pull_request_number is provided, fall back to the triggering PR (from GITHUB_REF).
• Acceptance criteria:
  • All three handlers reach resolveTargetPRNumber via one code path.
  • A test reproducing the 2026-05-22 scenario in actions/setup/js/awf_reflect.test.cjs (lines :25 / :214 mentioned in cache) passes without per-item rejection.
  • Smoke Claude run with target:"*" review-comment items completes with Failed: 0.
• Technical approach: Refactor in actions/safe_output_handler_manager.cjs. Add unit test covering the explicit-"*"-with-no-PR-number case.
• Estimated effort: Small.
• Dependencies: None.

Work Item 2: Per-item validation for review-comment paths/lines

• Type: Enhancement (prevention layer)
• Priority: Medium
• Description: Before buffering each create_pull_request_review_comment, validate (path, position/line) against the PR's files list. Drop invalid items individually with a clear error message rather than failing the entire review submission and falling back to body-only.
• Acceptance criteria:
  • Handler fetches PR files once per run and caches the validity map.
  • Invalid items are rejected with a message naming the offending path/line.
  • Valid items submit successfully even when invalid siblings are present in the same batch.
  • Existing body-only fallback remains as a safety net for unexpected 422s.
• Estimated effort: Small–Medium.
• Dependencies: None.

Work Item 3: Improve handler rejection error messages

• Type: Enhancement (developer experience)
• Priority: Medium
• Description: When rejecting a safe-output message, include the message index, handler type, and the specific field that caused rejection in the error string.
• Acceptance criteria:
  • All handler rejection paths emit messages matching pattern Message N (handler_type): <field> <reason>.
  • Logs allow an agent reading them post-hoc to identify exactly which item was rejected without diffing against its own output.
• Estimated effort: XS.

Historical Context

Date	Runs	safe_outputs ran	Failed	Success	Headline
2026-05-21	73	48	1	97.9%	review_path_unresolved_422
2026-05-22	65	24	1	95.8%	target_star_review_comment_no_pr_number_fallback
2026-05-23 (today)	21 slots / 8 ran	0	100%	— (no new clusters)

Trends:

• Failure-count trend: Decreasing (1 → 1 → 0), but volume also decreasing (Saturday effect).
• Success-rate trend: 97.9% → 95.8% → 100%, again influenced by today's smaller denominator.
• Open clusters: Both prior clusters are dormant today because the surfacing workflow (Smoke Claude/Copilot) did not run. Neither has been remediated upstream; the review_path_unresolved_422 recovery path is in place but the target_star_review_comment fix is still open.
• Validated handlers this window: noop (×4), create_issue (×1), create_discussion (×1), create_pull_request (×1, PR merged). End-to-end success on a non-trivial code path (create_pull_request → merged commit 5288def).

Metrics and KPIs

• Overall safe-output success rate (24h): 100%
• Most-exercised handler: noop (4 of 7 successful messages)
• Highest-value handler validated: create_pull_request (1/1, downstream merge confirmed)
• Most-problematic handler historically: create_pull_request_review_comment (the only handler with a still-open root-cause cluster) — but not exercised in this window.

Next Steps

• Open / refresh issue for Work Item 1 (unify target:"*" resolution) — High priority, still open from 2026-05-22.
• Tomorrow's audit: specifically check whether Smoke Claude reproduces the target_star_review_comment_no_pr_number_fallback cluster.
• Tomorrow's audit: specifically check whether submit_pull_request_review 422 → body-only fallback continues to recover cleanly.
• Cache memory updated: 2026-05-23.json written; index, recurring-failures, error-patterns reflect the dormant-but-open status of both prior clusters.

References:

• §26323913381 — sergo run (2 successful safe outputs)
• §26323431910 — go-logger run (PR #34165 created & merged)
• §26324183191 — smoke-ci noop

Generated by 🔒 Safe Output Health Monitor · ● 31.9M · ◷

• expires on May 24, 2026, 5:42 AM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Safe Output Health Monitor.

A newer discussion is available at Discussion #34375.