[safe-output-health] Safe Output Health Monitor - Daily Report 2026-09-25

[github-actions[bot]]

Executive Summary

Daily safe-output health audit for 2026-09-25 (~24h window). Scope: safe_outputs job executions only (create_discussion, create_issue, add_comment, update_project, submit_pull_request_review, resolve_pull_request_review_thread, approve_workflow_run, etc.) — agent/detection/activation job failures are out of scope for this monitor.

• 202 safe_outputs job executions analyzed (199 success / 3 failure), plus 11 skipped (no safe outputs configured for those runs)
• Success rate: 98.51% ((202−3)/202)
• 3 failures, all fully root-caused with 100% (3/3) raw error-text retrieval — a first for this audit series, versus a historical ~50% retrieval rate
• Key discovery: 2 of the 3 failures (66%) share one confirmed underlying bug class across 2 different tools and 2 different workflows — a PR-scoped safe-output tool hard-failing when invoked outside PR context instead of soft-skipping. This generalizes a previously tool-specific fix into a single cross-cutting recommendation.

Safe Output Job Statistics

Metric	Value
Runs analyzed (safe_outputs job)	202
Successful	199
Failed	3
Skipped (no safe outputs configured)	11
Success rate	98.51%
Raw error-text retrieval rate	3/3 (100%)

Error Clusters

Cluster A — "No PR context" decline miscategorized as hard failure (2 of 3 failures, 66%)

• §36076891728 — Smoke Claude — submit_pull_request_review: "Target is "triggering" but not running in pull request context, skipping PR review" (3rd occurrence: 2026-08-29, 2026-08-31, 2026-09-25)
• §36090252562 — PR Sous Chef — resolve_pull_request_review_thread x8: "Cannot resolve review threads outside of a pull request context" (3rd occurrence: 2026-09-23 x2, 2026-09-25; root cause confirmed today via direct raw error text, previously only hypothesized as a stale-reference race)

Both runs otherwise succeeded on their remaining 8-9 safe-output items — job-level "failure" despite high per-item success, reconfirming that per-item success counts alone don't guarantee job success.

Cluster B — Credential/permission bug, genuine and unresolved (1 of 3 failures, 33%)

• §36076531511 — Smoke Project — update_project x2: "Bad credentials - https://docs.github.com/rest" (4th occurrence over nearly a month: 2026-08-31, 2026-09-05, 2026-09-23, 2026-09-25)

Root Cause Analysis

Cluster A: no-PR-context decline bug class (click to expand)

Two distinct safe-output tools — submit_pull_request_review (target: triggering) and resolve_pull_request_review_thread — both fail with an E099 hard error when a workflow run is not itself PR-triggered (e.g. a scheduled smoke-test run). The tools correctly detect there is no PR to act on, but instead of soft-skipping the way add_comment already does under the analogous "no context" condition, they surface the decline through the same hard-failure path used for genuine bugs.

This was previously tracked as two separate, tool-specific investigations. As of today, the resolve_pull_request_review_thread root cause is directly confirmed (not just structurally inferred), and its error text and mechanism are functionally identical to the already-confirmed submit_pull_request_review case. These are now treated as one shared bug class requiring a single fix in the shared no-context-detection path used by all PR-scoped safe-output tools.

This is structurally similar to (but distinct from) an already-tracked sibling family: protected-files/allowed-files declines on approve_workflow_run and push_to_pull_request_branch being miscategorized as hard failures. Together, these two "policy decline miscategorized as failure" families likely account for the majority of all safe_outputs failures tracked across this audit's history since 2026-08-22.

Cluster B: Smoke Project "Bad credentials" (click to expand)

update_project fails with a GraphQL 401 "Bad credentials" error against the target GitHub Project board. This is distinct from the more common "Resource not accessible by personal access token" (scope-limited) errors seen elsewhere — "Bad credentials" suggests an invalid or missing token rather than an under-scoped one. All 4 occurrences to date have been on the Copilot engine; engine-specificity remains untested (no comparison run on a different engine has recurred yet) but the credential problem itself is no longer in doubt after 4 occurrences spanning nearly a month.

Recommendations

Critical

• Escalate the Smoke Project "Bad credentials" investigation — 4th occurrence, nearly a month unresolved, 100% failure rate on affected runs. Needs owner assignment to check the update_project token/credential configuration for the Copilot-engine path.

Bug Fixes

• Implement a generalized "no PR context → soft skip" fix covering submit_pull_request_review, resolve_pull_request_review_thread, dismiss_pull_request_review, and any other PR-scoped safe-output tool with the same precondition, in the shared context-detection code path (not patched per-tool). This single fix would have prevented 2 of today's 3 failures and at least 6 confirmed occurrences since 2026-08-29.
• (Carried over, still open) Reclassify protected-files/allowed-files policy declines on approve_workflow_run / push_to_pull_request_branch as soft skips rather than hard failures.

Process

• Continue tracking raw-log retrieval reliability. Today's 3/3 (100%) successful safe-output-errors.json retrieval is a positive signal against the historical ~50% average — worth confirming over the next several audits before declaring the underlying reliability issue resolved.

Work Item Plans

1. Smoke Project credentials — reproduce against project [mcp-analysis] MCP Structural Analysis - 2026-04-02 #24068 with the Copilot engine; verify whether the configured token/secret is present, valid, and has project scope; compare against a non-Copilot-engine run if one becomes available to test engine-specificity.
2. No-PR-context reclassification — locate the shared context-detection helper used by PR-scoped safe-output tools; extend it to return a skip result (matching add_comment's existing behavior) instead of raising when no PR context is available; apply uniformly rather than tool-by-tool to prevent future recurrences in yet-untested tools.

Historical Context & Trends

• This audit's per-item isolation model continues to hold: individual safe-output items within a batch fail/succeed independently, but the job as a whole is still marked "failure" if any item fails — even when 8-9/9 items succeeded.
• The no-PR-context bug class has now been confirmed across 2 tools (submit_pull_request_review since 2026-08-29, resolve_pull_request_review_thread confirmed today) and is likely to be found in additional tools not yet observed under this condition.
• The Smoke Project credential bug remains the longest-unresolved genuine bug currently tracked (vs. miscategorization bugs), at 4 occurrences over ~26 days with no fix shipped.
• Success rate today (98.51%) is roughly in line with the recent range (97.17%–99.75% over the last 5 audited days), not an outlier in either direction.

Metrics & KPIs

• Success rate: 98.51% (202 runs, 3 failures)
• Raw-log retrieval rate: 100% (3/3), best in this audit's history
• Open recurring-bug backlog: 2 confirmed bug classes needing a fix (no-PR-context decline miscategorization; Smoke Project bad credentials), plus previously-tracked protected-files/allowed-files decline family (still unshipped) and 2026-09-22/23 add_labels-fails-beside-create_discussion pattern (2 occurrences, still open)

Next Steps

1. Assign an owner to the Smoke Project credential investigation given its 4th occurrence.
2. Scope and implement the generalized no-PR-context soft-skip fix across all affected safe-output tools.
3. Continue daily monitoring; watch for a 3rd consecutive high raw-log-retrieval-rate day to confirm the reliability improvement is durable.

References:

• §36076531511 — Smoke Project (Bad credentials)
• §36076891728 — Smoke Claude (no-PR-context decline)
• §36090252562 — PR Sous Chef (no-PR-context decline, root cause confirmed today)

Generated by 🔒 Safe Output Health Monitor · claude · agent · 498 AIC · ⌖ 11.6 AIC · ⊞ 11K · ◷

• expires on Sep 25, 2026, 9:07 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-09-26T05:07:56.277Z.

Closed by Workflow