[safe-output-health] Safe Output Health Monitor: Daily Audit (2026-08-04) - 2 job failures found, root causes identified

[github-actions[bot]]

Executive Summary

Audited all 200 agentic workflow runs from the last 24 hours (2026-08-03 → 2026-08-04) for safe-output job health (the unified safe_outputs job / "Process Safe Outputs" step). Found 2 safe-output job failures, both root-caused to the actual raw step logs (process-safe-outputs.stderr.log / .stdout.log) rather than just structured summaries. Both failures are item-level validation errors inside an otherwise-mostly-successful batch — no framework crash, no API outage, no infrastructure issue. This is the first run of this monitor, so there is no historical baseline to compare against; today's data becomes that baseline going forward.

This differs from the assumed architecture of separate per-type jobs (create_discussion, create_issue, etc.) — gh-aw actually runs one unified safe_outputs job with a single "Process Safe Outputs" step that batches all configured safe-output types together.

Safe Output Job Statistics

Metric	Value
Total workflow runs analyzed	200
Runs with a safe_outputs job failure	2
Failure rate (of all runs)	~1%
Total safe-output items attempted across failing runs	15
Items successfully processed & persisted	10
Items that failed validation	3
Items skipped (policy, not error)	2

Note: both failing runs still persisted the majority of their safe-output items (PRs updated, comments posted, issue created) — the job-level "failure" status reflects a strict all-or-nothing exit code even on partial batch success. See Recommendations.

Error Clusters

Cluster 1 — resolve_pull_request_review_thread wrong node-ID type (Validation)

• Run: §30838483274 — "PR Sous Chef" (pi/gpt-5.4, schedule trigger)
• Occurrences: 2 of 13 items in this run's batch
• Raw error: thread_id must reference a PullRequestReviewThread node ID (PRRT_...); received PullRequestReviewComment for PRRC_kwDOPc1QR87c6owv (and ...owq)
• Root cause: The agent supplied a GitHub GraphQL comment node ID (PRRC_...) where the handler requires the parent thread node ID (PRRT_...). The other 9 items in the same batch (4 update_pull_request, 4 add_comment, 1 create_issue) succeeded and are live.
• Secondary non-fatal warnings (did not fail the job): a 403 on PUT .../pulls/49991/update-branch because the PR touches a .lock.yml workflow file and the GitHub App lacks workflows permission to merge base into it; a 422 on another PR's update-branch call ("no new commits on the base branch"). Both are pre-existing, non-blocking warnings — the PR body updates for those PRs still succeeded.

Cluster 2 — push_to_pull_request_branch outside allowed-files scope (Validation/Config)

• Run: §30867235474 — "Design Decision Gate 🏗️" (claude-sonnet-4-6, PR Refactoring squad steps for activation #50127 trigger). Note: this run is also separately flagged with an unrelated agent_logic classification by the audit heuristic on the agent job — that is a distinct, out-of-scope concern for a different monitor; only the safe_outputs job failure is analyzed here.
• Occurrences: 1 of 2 items in this run's batch
• Raw error: Cannot push to pull request branch: bundle modifies files outside the allowed-files list (actions/setup/js/parse_token_usage.cjs, actions/setup/js/parse_token_usage.test.cjs, actions/setup/sh/audit_pre_agent_workspace.sh, actions/setup/sh/audit_pre_agent_workspace_test.sh). Add the files to the allowed-files configuration field or remove them from the bundle.
• Root cause: design-decision-gate is configured with allowed-files: ["docs/adr/**"] — the agent's diff/bundle for PR Refactoring squad steps for activation #50127 touched 4 unrelated source files in addition to the intended ADR doc. The handler correctly blocked the push as an out-of-scope-file guardrail. The remaining add_comment item still succeeded, with a failure note appended.

Root Cause Analysis

Both failures are the safe-output validation guardrails functioning as designed, rejecting agent-produced content that didn't meet handler-level constraints (wrong ID type; out-of-scope file changes) — not a defect in the safe-outputs framework itself. The shared symptom across both: a job marked "failure" even though most/some items in the batch succeeded and were persisted to GitHub. This is because the "Process Safe Outputs" step processes all items but exits non-zero if any item errors, making job-level pass/fail a blunt signal for what is really a partial-success outcome.

No infrastructure, API-outage, rate-limit, or parsing errors were found in the safe-outputs job across all 200 runs in this window.

Recommendations

Bug fixes / handler improvements:

• resolve_pull_request_review_thread: clarify the tool schema/description to state the required node-ID type explicitly (PRRT_..., obtained via the reviewThreads GraphQL connection — not from an individual review comment). Consider having the handler resolve a PRRC_... comment ID to its parent thread ID server-side instead of hard-failing, since that mapping is deterministic and would make this class of error self-healing.

Process improvements:

• Distinguish partial batch success from total failure in the safe_outputs job's exit/summary (e.g. a distinct step outcome or job-summary annotation when items_succeeded > 0). Today both audited failures show 9/13 and 1/2 items succeeded, yet the job — and by extension the whole run — shows a flat "failure," which is noisy for monitoring and obscures that most content did get delivered.
• For workflows using allowed-files restrictions on push_to_pull_request_branch (like design-decision-gate), consider surfacing that scope constraint explicitly in the agent's own prompt/instructions so it self-limits its diff before ever reaching the safe-outputs handler, reducing wasted agent turns whose output gets rejected downstream.

Config changes: none required — both handlers' existing validation behavior is correct and should not be loosened.

Work Item Plans

1. [Handler] Improve resolve_pull_request_review_thread node-ID handling — update tool description; evaluate server-side PRRC→PRRT resolution fallback. Owner: safe-outputs framework maintainers.
2. [Observability] Add partial-success signal to safe_outputs job — expose items_succeeded/items_failed counts in the job's GitHub Actions status/summary rather than only a binary conclusion. Owner: gh-aw core.
3. [Prompting] Surface allowed-files scope to agents at generation time for workflows like design-decision-gate that restrict push_to_pull_request_branch targets. Owner: workflow authors.

Historical Context

This is the first run of the Safe Output Health Monitor — /tmp/gh-aw/cache-memory/safe-output-health/ had no prior audit data. Today's findings (2 failures, both Validation-category, both partial-batch) are now stored as the baseline (2026-08-04.json, error-patterns.json, recurring-failures.json, solutions.json) for trend comparison in future daily audits.

Metrics / KPIs

• Safe-output job failure rate: 2/200 runs (~1%)
• Item-level success rate across failing runs: 10/15 (67%) items still succeeded despite job-level "failure"
• Error categories seen: Validation (2/2 clusters); 0 API/Network/Parsing/Permission-only failures caused a job failure (the 403/422 warnings observed were non-fatal and did not block the job)

Next Steps

• Track whether resolve-thread-wrong-node-id and push-branch-outside-allowed-files recur in tomorrow's audit; promote to recurring-failures.json if seen again.
• Revisit the partial-success-signaling recommendation with the gh-aw core team.

References:

• §30838483274 — PR Sous Chef
• §30867235474 — Design Decision Gate 🏗️

Generated by 🔒 Safe Output Health Monitor · agent · 273.9 AIC · ⌖ 16.7 AIC · ⊞ 9.7K · ◷

• expires on Aug 4, 2026, 9:33 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-05T05:33:11.644Z.

Closed by Workflow