[safe-output-health] 🏥 Safe Output Health Report — 2026-07-23

[github-actions[bot]]

Executive Summary

• Period: Last 24 hours (window ~2026-07-22 12:58Z → 2026-07-23 05:18Z)
• Runs analyzed: 265 (every run's safe_outputs job conclusion read directly from job_details[] in the pre-bundled run_summary.json)
• Safe-output jobs executed: 264 (+1 skipped, activation-gated)
• Safe-output jobs failed (in-scope): 2
• In-scope success rate: 99.24% (262 success / 264 executed)
• Error clusters identified: 2 — both map to long-tracked standing-open signatures; no new failure class

The clean streak of 07-21 / 07-22 broke again on this write-rich window — the same 2-in-scope-failure shape as 07-20. Both failures are isolated and family-consistent; the vast majority of the fleet actuated cleanly.

Safe Output Job Statistics

Conclusion	Count	Notes
success	262	—
failure (in-scope)	2	both at the Process Safe Outputs step
skipped	1	run-29923473208 — gated at activation (Check daily workflow token guardrail), job never executed → out of scope, excluded from denominator
total	265

Safe-output item types in the two failed runs: create_pull_request_review_comment ×3, update_pull_request ×2, add_comment ×2, create_issue ×1.

Error Clusters

Cluster 1 — Review-comment actuation hard-fail (HIGH) · review_path_unresolved_422

• Count: 1 run · 2nd in-production hard-fail in 4 days (after 2026-07-20 run-29716501145)
• Run: PR Code Quality Reviewer — §29946005059 (📚 Documentation Noob Test Report - November 26, 2025 #4826), pull_request, branch linter-miner/stringbytesroundtrip-c93b0c9cf4ee201b
• Shape: agent job = success, 0 firewall blocks, 3× create_pull_request_review_comment, safe_outputs job = failure at Process Safe Outputs
• Root cause (inferred): strongest single-signal match to the review-path 422 signature — one or more line/path-anchored review comments hit a GitHub 422 "Path/Line could not be resolved" where the body-only fallback at pr_review_buffer.cjs:554 did not rescue the submission. Sibling target_star_review_comment_no_pr_number_fallback is also possible.
• Impact: the intended code-review comments were not posted; the run shows red. Item/position-specific, not a blanket reviewer regression (262/265 jobs succeeded, including other reviewer runs).

Why the exact variant is unconfirmed

The Process Safe Outputs step stdout/stderr is not pre-bundled in the run artifacts, jq is sandbox-blocked from reading /tmp, and the agenticworkflows logs CLI timed out at 300s. Root cause is therefore inferred from the safe-output item type + trigger context + family history, not read directly. This is the recurring observability gap flagged below.

Cluster 2 — Scheduled-run target resolution hard-fail (MEDIUM, production) · target-resolution family

• Count: 1 run (new in-scope production failure this window)
• Run: PR Sous Chef — §29974524390 (Add jqschema shared workflow for analyzing large tool outputs #1942), schedule, branch main
• Shape: agent job = success, mixed 5-item payload — 2× update_pull_request, 2× add_comment, 1× create_issue, safe_outputs job = failure at Process Safe Outputs
• Root cause (inferred): PR Sous Chef is schedule-triggered, so there is no triggering PR/issue context. add_comment / update_pull_request items that must resolve a target PR hard-fail on scheduled runs unless each item carries an explicit resolvable pull_request_number — the same soft-skip-vs-hard-fail handler inconsistency tracked in target_star_add_comment_no_item_number_fallback and update_issue_target_triggering_default_hardfails_scheduled_run.
• Impact: the scheduled PR-maintenance actions (nudges / PR-body updates) did not actuate; run shows red. PR Sous Chef ran read-only/clean twice on 07-22 — this scheduled run actuated writes and failed.

Root Cause Analysis

• API / actuation errors: Cluster 1 is a GitHub 422 review-submission family; Cluster 2 is a target-resolution failure on a context-less scheduled run. No auth, rate-limit, or firewall issues (0 blocked requests in both).
• Permission errors: none observed.
• Data validation: not the cause; agents produced valid items and their agent jobs succeeded.
• Observability: the exact step error is unrecoverable for both failures — the single most important cross-cutting gap (see recommendations).

Recommendations

Critical / High

1. Pre-bundle the Process Safe Outputs step stdout/stderr on failure — Priority: High. This is now the 5th+ class of in-scope JOB-layer failure whose exact error cannot be read from the pre-bundled artifacts, and the logs CLI timed out. Without it, both of today's failures (and the 07-20 pair) can only be family-attributed, not root-caused. This unblocks every other fix below.
2. Apply the pr_review_buffer.cjs:554 Path-variant predicate fix — Priority: High. Change the fallback predicate to (errorMessage.includes("Line could not be resolved") || errorMessage.includes("Path could not be resolved")) && comments.length > 0, and add a pr_review_buffer.test.cjs case mirroring the existing "Line" test with "Path". The review-comment path has now hard-failed in production twice in 4 days (07-20, 07-23); the one-line fix has been the rejig docs #1 outstanding remediation for 55 audits and should be applied pre-emptively rather than waiting for a confirmable 422.

Medium

3. Unify missing-trigger-context handling for add_comment / update_pull_request on scheduled runs — when no PR/issue number is resolvable, soft-skip (⏭) with a clear message rather than a ##[error] that fails the whole job, matching create_pull_request_review_comment behavior. Alternatively, PR Sous Chef should attach explicit pull_request_number to each add_comment/update_pull_request item (it operates cross-PR on a schedule).
4. Surface the offending item index + path/line in the review-submission error so agents can self-correct on retry.

Work Item Plans

Work Item 1 — Pre-bundle Process Safe Outputs step log on failure

• Type: Enhancement (observability) · Priority: High · Effort: Small–Medium
• Acceptance: on a safe_outputs job failure, the Process Safe Outputs step stdout/stderr is uploaded as a run artifact and readable by the audit tooling.
• Rationale: 5th+ class of unreadable in-scope failure; blocks confirmation of clusters 1 & 2.

Work Item 2 — Path-variant review-comment fallback fix + regression test

• Type: Bug fix · Priority: High · Effort: Small
• Location: actions/setup/js/pr_review_buffer.cjs:554 (+ pr_review_buffer.test.cjs)
• Acceptance: both "Line could not be resolved" and "Path could not be resolved" 422s enter the body-only fallback; new unit test locks in dual-variant matching.

Work Item 3 — Scheduled-run target-resolution: soft-skip parity

• Type: Bug fix (handler consistency) · Priority: Medium · Effort: Medium
• Acceptance: add_comment / update_pull_request with unresolvable target on schedule/dispatch runs soft-skip (⏭) instead of hard-failing the job; OR PR Sous Chef supplies explicit pull_request_number per item.

Historical Context & Trends

• 07-23 vs recent: 07-18/07-19 clean → 07-20 2 in-scope fails (94.6%) → 07-21/07-22 clean (100%) → 07-23 2 in-scope fails (99.24%). Clean streaks keep breaking on write-rich windows; read-only windows are uniformly clean.
• Review-path signature: now 2 in-production hard-fail data points in 4 days (07-20, 07-23). The pr_review_buffer.cjs:554 Path-variant fix remains UNVALIDATED for the 55th consecutive audit.
• Standing-open, not exercised today: Changeset Generator patch-format:bundle bundle-transport (push_to_pull_request_branch) — absent/not-actuating ~27 days since 2026-06-26; still the highest-priority open production signature, unvalidated.

KPIs

• Overall in-scope safe-output success rate: 99.24%
• Most problematic job type this window: create_pull_request_review_comment (review submission 422 family)
• Failure mode: 100% of in-scope failures were at the Process Safe Outputs step with agent job success (resilient handoff held everywhere else)

Next Steps

• Land Work Item 1 (pre-bundle step log) to make future review-path/target-resolution failures confirmable
• Land Work Item 2 (pr_review_buffer.cjs:554 predicate fix + test) — 2 in-prod data points now justify pre-emptive fix
• Land Work Item 3 (scheduled-run soft-skip parity for add_comment/update_pull_request)
• Watch the next PR Code Quality Reviewer and PR Sous Chef write-runs for recurrence
• Continue watching for a Changeset Generator patch-format:bundle actuation to finally exercise that signature

References:

• §29946005059 — PR Code Quality Reviewer (review-comment hard-fail)
• §29974524390 — PR Sous Chef (scheduled target-resolution hard-fail)
• §29716501145 — 2026-07-20 review-comment hard-fail (prior data point)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• awmgmcpg

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "awmgmcpg"

See Network Configuration for more information.

Generated by 🔒 Safe Output Health Monitor · age00 376.2 AIC · ⌖ 17.3 AIC · ⊞ 9.4K · ◷

• expires on Jul 23, 2026, 9:34 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Safe Output Health Monitor.

A newer discussion is available at Discussion #47712.