[safe-output-health] 🏥 Safe Output Health Report - 2026-06-24

[github-actions[bot]]

Executive Summary

• Period: Last 24h (run window ~01:28Z → 05:32Z, 56 run directories)
• Runs Analyzed: 56
• safe_outputs Jobs (completed, non-skipped): 55
• safe_outputs Jobs Failed (in scope): 4 — all smoke-only, 0 production
• safe_outputs Job Success Rate: 92.7% (51/55)
• Error Clusters Identified: 1 family (re-reproduction, no new clusters)

Headline: The production safe_outputs streak holds — zero production safe-output job failures today. All 4 hard failures are smoke-only (production_impact=false), agent=SUCCESS in every case, and all on a single PR branch copilot/bump-firewall-v0279-and-mcpg-v0329. This is the 4th reproduction window for the long-tracked smoke target/context-resolution family (prior: 06-11, 06-14, 06-15). Yesterday's production Changeset-bundle offender did not recur (workflow absent).

safe_outputs Job Statistics

gh-aw runs all safe outputs through a single safe_outputs job per run (no per-handler jobs), so failures are reported at the job level. Per-message breakdown is not observable today (step logs not pre-bundled; the audit MCP tool timed out at 120s on run 28068855925).

Metric	Count	Rate
safe_outputs jobs completed	55	—
safe_outputs jobs success	51	92.7%
safe_outputs jobs hard failure	4	7.3%
safe_outputs jobs skipped	0	—
Production safe_outputs failures	0	—
Agent-job failures (out of scope)	7	all handed off to safe_outputs=SUCCESS

Error Cluster: Smoke target/context-resolution hard-fail family (smoke-only) — REPRODUCED

• Count: 4 runs, all branch copilot/bump-firewall-v0279-and-mcpg-v0329
• Pattern: agent=SUCCESS + detection=SUCCESS, failure isolated to downstream safe_outputs job
• Production impact: none (smoke workflows only)

Run	Workflow	Event	Inferred sub-cluster
§28068855925	Smoke Copilot	workflow_dispatch	add_labels/remove_labels no-item-number hard-fail (no triggering context)
§28070822297	Smoke Copilot	workflow_dispatch	same (2nd repro ~1h later, 02:37Z)
§28068825851	Smoke Claude	pull_request	add_comment→discussion Resource not accessible by integration / target:* review-comment
28068825831	Smoke Codex	pull_request	same discussions:write / target family

Root-cause analysis (inferred — message logs not recoverable today)

The smoke workflows deliberately exercise every safe-output handler in one run. Two known, still-open defects in this family explain the failures:

1. Soft-skip vs hard-fail inconsistency (target_star_add_labels_no_item_number_hardfail). On workflow_dispatch there is no triggering PR/issue. create_pull_request_review_comment / comment_memory soft-skip (⏭ Not in pull request context), but add_labels / remove_labels / add_comment hard-fail (✗ No issue/PR number available), failing the whole job. The two Smoke Copilot workflow_dispatch runs match the 06-11 (run-27320477001) and 06-14 (run-27481382799) signature exactly. Family window count now 4 (06-11, 06-14, 06-24×2).

2. Discussions write-scope gap (add_comment_discussion_resource_not_accessible_by_integration). The smoke safe_outputs job token lacks discussions:write, so add_comment falling through to the discussion-comment path hard-fails Resource not accessible by integration. Smoke Claude (pull_request) is the exact 06-14/06-15 offender — it was JOB-clean on 06-23 and regressed today; Smoke Codex (also clean 06-23) failed similarly.

Why inference, not certainty: Process Safe Outputs step logs were not pre-bundled, and the audit MCP tool timed out (120s). Attribution comes from the smoke prompt + the documented 06-11/06-14/06-15 family history, consistent with prior-audit methodology.

Recommendations

Bug fixes (in scope, still OPEN):

1. Unify missing-trigger-context handling across add_labels/remove_labels/add_comment and the review-comment/comment_memory handlers — soft-skip (⏭) with a clear message when no issue/PR number is resolvable, instead of ##[error] that fails the job. This is the cleanest, now 4-times-reproduced demonstration of the inconsistency. Severity: low (smoke-only) but cheap and high-signal.
2. Grant the smoke safe_outputs job token discussions:write (or make add_comment→discussion soft-skip when the scope is absent). Reproduced across 06-14/06-15 and again 06-24.
3. (Carried) pr_review_buffer.cjs:554 Path-variant predicate fix — UNVALIDATED for the 27th consecutive audit (no PR-reviewers exercised the path today).

Process / observability:

4. Pre-bundle the "Process Safe Outputs" step log (or fix the audit MCP tool timeout) so per-message failure strings are recoverable. Today's 120s timeout + absent step logs blocked exact root-causing of all 4 failures — a recurring limitation across audits.

Work Item Plan

WI-1: Unify soft-skip vs hard-fail for missing trigger context

• Type: Bug Fix · Priority: Medium · Effort: Small
• Acceptance: add_labels/remove_labels/add_comment soft-skip (not hard-fail) when no resolvable issue/PR target exists; a smoke workflow_dispatch run no longer fails its safe_outputs job for this reason.

WI-2: discussions:write for smoke safe_outputs token

• Type: Config/Permission · Priority: Low-Medium · Effort: Small
• Acceptance: Smoke Claude/Codex pull_request add_comment→discussion no longer 403s; or it soft-skips cleanly.

WI-3: Recover per-message safe-output logs in the monitor

• Type: Observability · Priority: Medium · Effort: Small
• Acceptance: Audit can read failed-message error strings without depending on the audit MCP tool (which timed out today).

Historical Context & Trends

• 06-22: 0 failures · 06-23: 1 (production Changeset bundle) · 06-24: 4 (all smoke).
• Count rose 1→4 but composition shifted: production blast radius dropped to zero today; the 06-23 production bundle-transport failure neither recurred nor was retested (Changeset Generator absent).
• The smoke target/context family is firmly open — reproduced on both workflow_dispatch (add_labels-no-context) and pull_request (add_comment→discussion / target:*) variants in one window.
• Most reliable: all production handlers (create_issue, create_discussion, create_pull_request, add_labels-with-context, push_to_pull_request_branch default patch-format) — 0 failures.
• Most problematic: smoke matrix safe-output exercising under missing-context / discussions-scope conditions.

Next Steps

• Land WI-1 (soft-skip unification) — closes 4-window-reproduced family
• Land WI-2 (discussions:write or soft-skip)
• Improve monitor log recovery (WI-3) so future audits read exact messages
• Watch for Changeset Generator to retest the 06-23 patch-format:bundle production path
• Re-exercise pr_review_buffer.cjs:554 Path-variant (UNVALIDATED 27 audits)

References:

• §28068855925 — Smoke Copilot (workflow_dispatch)
• §28068825851 — Smoke Claude (pull_request)
• §28070822297 — Smoke Copilot (2nd repro)

Generated by 🔒 Safe Output Health Monitor · 415.8 AIC · ⌖ 16.6 AIC · ⊞ 9.3K · ◷

• expires on Jun 24, 2026, 9:46 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Safe Output Health Monitor.

A newer discussion is available at Discussion #41386.