[safe-output-health] 🏥 Safe Output Health Report - 2026-07-20

[github-actions[bot]]

Executive Summary

• Period: Last 24 hours (window ~03:46Z–05:18Z, 2026-07-20)
• Runs Analyzed: 37 (all run_summary.json triaged directly from job_details[])
• Workflows Active: 26 distinct workflows
• Safe Output Jobs Executed: 37
• Safe Output Jobs Failed (in-scope): 2
• In-Scope Safe-Output Job Success Rate: 94.6% (35/37)
• Error Clusters Identified: 2

Clean streak broken. After two fully-clean read-only days (07-18, 07-19), this is a write-rich day with two in-scope Process Safe Outputs hard failures. The headline is the first create_pull_request_review_comment actuation hard-failure since the 2026-05-27 Path regression.

Safe Output Job Statistics

Job outcome	Count	Notes
safe_outputs job = success	35	includes ~15 reviewer/write runs
safe_outputs job = failure	2	both at step Process Safe Outputs
Total	37	94.6% success

Observable item mix is limited to the two failing runs (only they carried summary.json on disk). See the metrics caveat below.

Error Clusters

Cluster 1 — Review-comment actuation hard failure (HIGH) 🔴

• Count: 1 run
• Run: PR Code Quality Reviewer — §29716501145
• Event: pull_request (branch fix/yamllint-trailing-spaces-custom-steps)
• Item mix: exactly 1× create_pull_request_review_comment
• Fingerprint: agent = success, detection = success, firewall = 0 blocked, ErrorCount/Noop/MissingTool = 0, safe_outputs job = failure at Process Safe Outputs
• Root cause (high-confidence family, variant unconfirmed): This is the first in-scope hard failure of the review-comment actuation path since the 2026-05-27 Path regression. A single line/path-anchored review comment that fails the whole job is the exact shape of the long-tracked review_path_unresolved_422 family — GitHub returns 422 "Path could not be resolved" / "Line could not be resolved" and the body-only fallback at pr_review_buffer.cjs:554 fails to rescue it. The sibling target_star_review_comment_no_pr_number_fallback is also plausible.
• Impact: PR review dropped; production run shows red. Item/position-specific, not a blanket reviewer regression — the other two PR Code Quality runs plus all Matt Pocock / Test Quality / Impeccable reviewer runs this window succeeded.

Why the exact variant can't be confirmed (observability gap)

The Process Safe Outputs step stdout was not pre-bundled, and the sandbox blocked jq/bash/cat on /tmp while the audit MCP tool surfaces only run diffs — not the step log. This is the 4th distinct in-scope JOB-layer failure class with an unreadable exact error (after 06-23 & 06-26 Changeset Generator and 07-11 Firewall Collector). Without the step log, we cannot distinguish "fallback fired and still failed" from "predicate never matched."

Cluster 2 — Smoke all-handler job failure (MEDIUM, smoke/diagnostic) 🟡

• Count: 1 run
• Run: Smoke Copilot — §29718026927
• Event: workflow_dispatch (issue_comment on PR feat(copilot): support model: auto/model: none to bypass model pinning on automatic-selection plans #46556, branch copilot/fix-copilot-engine-issue)
• Item mix: 9 distinct types in one run — send_slack_message, upload_artifact, add_comment, create_issue, create_discussion, create_check_run, set_issue_type, comment_memory, submit_pull_request_review
• Fingerprint: agent = success, 433K tokens, firewall = 0 blocked, safe_outputs job = failure at Process Safe Outputs
• Root cause (likely family): An all-handler smoke that deliberately exercises the full matrix; at least one handler hard-failed. Given prior smoke JOB failures (06-14 / 06-24), the most probable culprits are the discussions-scope path (add_comment→discussion or create_discussion needing discussions:write the smoke safe_outputs token lacks — see add_comment_discussion_resource_not_accessible_by_integration) or an edge handler (set_issue_type / create_check_run / comment_memory / send_slack_message).
• Impact: Low — validation/smoke workflow, not production. Message-level attribution not observable (step log not pre-bundled).

Root Cause Analysis

API / Actuation Issues

Both failures are at the actuation layer (Process Safe Outputs), not the agent or network layer — firewall was clean (0 blocked) on both. Cluster 1 is a single-item review-comment submission failing atomically; Cluster 2 is one-of-nine handlers in an all-handler smoke.

Permission Issues

Cluster 2's likely discussions:write scope gap on the smoke safe_outputs token is a recurring, documented smoke-only defect (add_comment_discussion_resource_not_accessible_by_integration, occurrences on 06-14/06-15/06-24).

Observability (cross-cutting)

Neither exact error is recoverable — the Process Safe Outputs step log is not pre-bundled. This is now the 4th such opaque JOB-layer failure class in audit history.

Recommendations

Critical / High

1. Confirm and (if applicable) fix the review-comment 422 variant — Priority: High
   • Action: Pre-bundle the Process Safe Outputs step stdout on failure, then re-read run §29716501145. If the error is Path could not be resolved, apply the known one-line predicate fix at actions/setup/js/pr_review_buffer.cjs:554 to also match "Path could not be resolved" (the retry block already handles both cases), and add the mirrored Path-variant unit test.
   • Affected: create_pull_request_review_comment, submit_pull_request_review (all line-anchored reviewers).

Observability (recurring — raise once, applies to all JOB-layer failures)

2. Pre-bundle the Process Safe Outputs step stdout/stderr on failure — Priority: High
   • Problem: 4 distinct in-scope JOB-layer failures now have unreadable exact errors.
   • Fix: Upload the step log as an artifact whenever the step conclusion is failure. This unblocks root-causing Cluster 1 and every future opaque JOB failure.

Configuration / Defensive

3. Smoke safe_outputs token discussions: write — Priority: Medium (smoke-only)

   • If the smoke intends to comment on / create discussions, grant discussions: write in the compiled lock; otherwise point the smoke target at a resolvable issue/PR or expect a graceful skip.

4. Soft-skip parity across handlers — Priority: Medium

   • Handlers that can't resolve context should soft-skip (⏭) rather than hard-fail the whole job — matching create_pull_request_review_comment's existing behavior. This blunts the blast radius of both clusters.

Work Item Plans

Work Item 1: Root-cause & fix review-comment 422 recurrence

• Type: Investigation → Bug Fix
• Priority: High
• Description: PR Code Quality Reviewer run-29716501145 hard-failed on a single create_pull_request_review_comment. Confirm whether it is the Path-variant of review_path_unresolved_422 and fix.
• Acceptance Criteria:
  • Process Safe Outputs step log captured for the failing run
  • Variant confirmed (Path / Line / target:* / other)
  • If Path-variant: pr_review_buffer.cjs:554 predicate matches both Line/Path; regression test added
• Effort: Small–Medium

Work Item 2: Pre-bundle Process Safe Outputs step log on failure

• Type: Enhancement (observability)
• Priority: High
• Description: Upload the Process Safe Outputs stdout/stderr artifact when the step fails, so JOB-layer failures are root-causable.
• Acceptance Criteria:
  • On step failure, log is uploaded to run artifacts
  • Audit tooling can read it
• Effort: Small

Work Item 3: Smoke discussions-scope / handler-failure isolation

• Type: Investigation (smoke)
• Priority: Medium
• Description: Identify which of the 9 handlers failed in Smoke Copilot run-29718026927; grant discussions: write or adjust the smoke target as appropriate.
• Acceptance Criteria:
  • Failing handler identified from step log
  • Fix applied (scope grant or target change) or documented as expected
• Effort: Small

Historical Context

Date	Runs	SO jobs fail	Success %	Headline
2026-07-20	37	2	94.6%	review-comment actuation hardfail + smoke all-handler
2026-07-19	12	0	100%	fully clean read-only
2026-07-18	14	0	100%	fully clean read-only
2026-07-17	131	0	100%	write-rich, in-scope clean
2026-07-16	14	0	100%	clean read-only (4th consec)

Trends

• Error-rate trend: up from a two-day 100% clean streak to 94.6% — driven by a genuine reviewer-path hard failure, not smoke noise alone.
• Notable: the review_path_unresolved_422 family — happy-path healthy but the Path-variant fallback unvalidated for 52 consecutive audits — now has its first plausible in-production hard-fail data point since 2026-05-27. This elevates the long-deferred observability fix from "nice-to-have" to "blocking root cause."
• Standing-open (unchanged): Changeset Generator patch-format:bundle bundle-transport hardfail — ABSENT ~24 days, still the highest-priority unvalidated production signature.

Metrics and KPIs

• Overall Safe-Output Job Success Rate: 94.6% (35/37)
• Most Problematic Job Type: create_pull_request_review_comment (1 hard failure this window)
• Metrics caveat: behavior_fingerprint.actuation_style reported read_only on all 37 runs, including the two that demonstrably actuated (1 and 9 items). The actuation fingerprint / SafeItemsCount undercounts bash-safeoutputs CLI-wrapper writes (recurring blind spot). The day is not read-only; the full write-mix for the ~15 successful reviewer/write runs is not enumerable because only the two failing runs carried summary.json on disk.
• Coverage: All 37 safe_outputs job conclusions read directly from job_details[] — this metric is authoritative and complete for the window.

Next Steps

• Capture the Process Safe Outputs step log for run-29716501145 and confirm the review-comment 422 variant
• Ship the pre-bundle-step-log-on-failure observability change
• Isolate the failing handler in Smoke Copilot run-29718026927 (likely discussions-scope)
• Continue watching for a Changeset Generator patch-format:bundle re-exercise (unvalidated ~24d)

References:

• §29716501145 — PR Code Quality Reviewer (safe_outputs FAILURE)
• §29718026927 — Smoke Copilot (safe_outputs FAILURE)
• §29718622115 — PR Code Quality Reviewer (success, same workflow)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by 🔒 Safe Output Health Monitor · 492.6 AIC · ⌖ 17.9 AIC · ⊞ 9.4K · ◷

• expires on Jul 20, 2026, 9:38 PM UTC-08:00

[github-actions[bot]]

Cave bot tap wall. Smoke run 29720659363 say hello.

📰 BREAKING: Report filed by Smoke Copilot · 117.8 AIC · ⌖ 8.24 AIC · ⊞ 19K · ◷
_{Comment /smoke-copilot to run again
Add label smoke to run again}

[github-actions[bot]]

This discussion has been marked as outdated by Safe Output Health Monitor.

A newer discussion is available at Discussion #46981.