[safe-output-health] Safe Output Health Report - 2026-10-01

[github-actions[bot]]

🏥 Safe Output Health Report - 2026-10-01

Executive Summary

• Period: Last 24 hours (approx. 2026-09-30T04:40Z – 2026-10-01T04:40Z)
• Runs Analyzed: 212 workflow runs (pre-downloaded log cache)
• Safe Output Jobs Executed: 210
• Safe Output Jobs Failed: 11
• Overall Safe Output Success Rate: 94.76% — the lowest recorded since this monitor began tracking on 2026-08-22 (previous worst: 97.17% on 2026-09-23)
• Error Clusters Identified: 2 (1 brand-new dominant cluster, 1 already-tracked recurrence)

Caveat on cadence: there is an 8-day gap since the last audit (2026-09-23 → 2026-10-01), so no day-over-day trend is available — only a comparison against that prior data point.

Safe Output Job Statistics

Job Type (failing item)	Occurrences	Workflows Affected
set_issue_type	10	Smoke Copilot, Smoke Copilot - AOAI (apikey), Smoke Copilot - AOAI (Entra)
update_project / create_project_status_update	1	Smoke Project
Total failures	11	out of 210 safe_outputs job executions (199 succeeded)

Error Clusters

Cluster 1 (NEW, DOMINANT): set_issue_type silently fails while every sibling item succeeds

• Count: 10 occurrences (91% of today's failures)

• Affected workflows: Smoke Copilot, Smoke Copilot - AOAI (apikey), Smoke Copilot - AOAI (Entra) — all 3 engine variants of the Smoke Copilot family

• Affected runs: 36745551409, 36745556440, 36745561562 (16:37Z, commit bc0233b0), 36762114132, 36762134104 (18:56Z, commit f19087b6), 36795117785, 36797299158 (00:13–00:39Z, main branch, scheduled trigger, commit 856e7fa3), 36812740323, 36812743202, 36812746071 (03:55Z, commit e4da7f04)

• Sample evidence (run 36812746071, Smoke Copilot - AOAI (Entra)):

  MCP gateway tool_calls: add_comment, add_comment, create_issue, set_issue_type, send_slack_message  (all invoked)
  safe_outputs.items result array: add_comment (#64461 comment), create_issue (#64683), add_comment (#64461 comment), send_slack_message
  → set_issue_type: called, but ABSENT from the result array. Job step "Process Safe Outputs" concluded "failure".
  

• Root cause: set_issue_type is invoked by the agent in every single occurrence (visible in usage/activity/summary.json → gateway.tool_calls), but never produces an entry in the same file's safe_outputs.items / items_by_type, while every other configured safe-output type in the same batch (add_comment, create_issue, send_slack_message, add_labels, remove_labels, create_check_run, comment_memory) succeeds and shows a real URL/number. This reproduced identically across:

  • 3 distinct commits on an in-development branch (copilot/fix-agentic-conversation-session-state, workflow_dispatch-triggered), and
  • main, via the normal schedule trigger (commit 856e7fa3)

  This rules out a branch-specific or trigger-specific explanation — it is a genuine, reproducible code-level defect in the set_issue_type safe-output handler itself. It is also not consistently correlated with the separate clients2.google.com firewall block noted in some of these runs (present in only ~4 of the 10 occurrences), so that network block is circumstantial, not causal.

• Impact: This is the single most consistent, 100%-reproducible signature in this monitor's entire history (10/10 checked occurrences share the exact shape) and is the direct cause of today's record-low success rate. No credential ambiguity, no policy-decline miscategorization, no "which item failed" evidence gap — just a clean, structural, repeatable bug.

• Evidence gap: raw safe-output-errors.json / per-item error/stack-trace text was not retrievable this audit — an agenticworkflows audit call with artifacts: ["all"] against run 36812746071 returned only the same structured run_summary.json shape, no raw console text. This is the same intermittent raw-log-retrieval gap flagged in several prior audits (tracked separately below).

Cluster 2 (recurring, already tracked): Smoke Project update_project / create_project_status_update — Bad credentials

• Count: 1 occurrence (run 36795322922, 2026-10-01T00:16:15Z)
• Affected job: safe_outputs / Process Safe Outputs
• Affected tools: update_project ×6, create_project_status_update ×1 — 100% of the run's 7 safe-output items failed
• Root cause: 4th confirmed occurrence of the long-tracked smoke-project-update-project-bad-credentials pattern (first seen 2026-08-31). The token used by the Copilot-engine "Smoke Project" workflow continues to be rejected for GraphQL project-management mutations against github/projects/24068.
• Impact: Low — isolated to a single smoke-test workflow, already understood, fix still pending.

Root Cause Analysis

Code/Logic Issues

The set_issue_type cluster above is a genuine, high-confidence, actionable bug — not a false positive or policy decline like most of this monitor's historically-tracked clusters. It warrants immediate engineering attention.

Permission/Credential Issues

Smoke Project's update_project/create_project_status_update 401 "Bad credentials" issue persists unresolved after 4 occurrences spanning a month (first seen 2026-08-31).

Notable Absence

Zero occurrences today of the previously-dominant PR Sous Chef / approve_workflow_run pattern (13+ prior occurrences across the monitor's history) — no evidence this was fixed, simply no PRs today triggered the relevant protected-files/fork-PR conditions.

Observability Gap (carried forward)

Raw per-item error text (safe-output-errors.json) was again not retrievable via agenticworkflows audit/logs with artifacts: ["all"] — this is now a recurring, ~50%-of-audits limitation that forces root-causing via structural cross-referencing (tool-call vs. result-array diffing) rather than direct error messages. See the carried-forward recommendation below.

Recommendations

Critical Issues (Immediate Action Required)

1. Fix set_issue_type silent failure in the safe_outputs handler
   • Priority: Critical
   • Root Cause: The set_issue_type safe-output tool is invoked but never returns/records a result, in 10/10 checked occurrences across 3 workflow variants, 4 distinct commits, both workflow_dispatch and schedule triggers.
   • Recommended Action: Investigate the set_issue_type runtime handler (compiler-side config in pkg/workflow/set_issue_type.go; runtime execution happens in the safe-outputs processing script invoked during the "Process Safe Outputs" step). Check for an unhandled exception in the issue-type GraphQL mutation (e.g., referencing an issue type name/ID that doesn't resolve in this repo), a recent regression, or a permissions gap specific to that mutation. Also consider making set_issue_type soft-skip-on-failure (consistent with how add_comment's no-context case already soft-skips) so one failing item doesn't flip an otherwise-healthy batch to a hard job failure.
   • Affected: Smoke Copilot, Smoke Copilot - AOAI (apikey), Smoke Copilot - AOAI (Entra)

Bug Fixes Required

1. set_issue_type handler — no result produced
   • File/Location: pkg/workflow/set_issue_type.go (compiler/config side) and its runtime safe-outputs processing counterpart
   • Problem: Handler is called but produces no item in safe_outputs.items, with job-level "Process Safe Outputs" step concluding failure
   • Fix: Add error propagation/logging so the failure surfaces a concrete reason in safe-output-errors.json; fix the underlying mutation/call so it succeeds or gracefully declines
   • Affected Jobs: safe_outputs (Smoke Copilot family)

Process Improvements

1. Raw log/error-text retrieval reliability
   • Current State: agenticworkflows audit/logs with artifacts: ["all"] intermittently fails to retrieve safe-output-errors.json/raw console text even when the targeted run and job are correctly specified (recurred again today against run 36812746071).
   • Proposed: Investigate why raw artifact retrieval is inconsistent (parameter combination, retention window, or a race with upload) so future audits don't have to fall back on structural cross-referencing to find root causes.
   • Benefits: Faster, higher-confidence root-causing for every future new signature.

Work Item Plans

Work Item 1: Fix set_issue_type safe-output handler silent failure

• Type: Bug Fix
• Priority: Critical
• Description: set_issue_type fails to produce a result in 100% of 10 occurrences today across 3 Smoke Copilot workflow variants, both on an in-development branch and on main, while every sibling safe-output item in the same batch succeeds. This is driving the safe_outputs job failure rate to a monitor-record low.
• Acceptance Criteria:
  • Root cause of the silent/no-result failure in set_issue_type identified (handler exception, GraphQL mutation failure, or permissions gap)
  • Fix applied so set_issue_type either succeeds or fails with a clear, logged error in safe-output-errors.json
  • A single set_issue_type failure no longer hard-fails the whole safe_outputs job when sibling items succeed (soft-skip or isolated failure reporting)
• Technical Approach: Reproduce locally against one of the Smoke Copilot workflows' configuration; add logging/error propagation to the handler; verify against a scheduled main-branch run
• Estimated Effort: Small–Medium
• Dependencies: None

Work Item 2: Investigate Smoke Project "Bad credentials" (carried forward)

• Type: Investigation
• Priority: Medium-High (4th occurrence over a month)
• Description: update_project/create_project_status_update continue to fail with GraphQL 401 "Bad credentials" against github/projects/24068 for the Copilot-engine "Smoke Project" workflow.
• Acceptance Criteria:
  • Confirm whether the token used is expired, missing the project scope, or not passed through correctly for this engine/runtime combination
• Technical Approach: Compare against a known-working update_project run if one exists; check secret/token configuration for this workflow
• Estimated Effort: Small
• Dependencies: None

Historical Context

Comparing against the last available data point (2026-09-23, since there is an 8-day audit gap):

Metric	2026-09-23	2026-10-01
Runs analyzed	212	212
Safe output failures	6	11
Success rate	97.17%	94.76%

Trends

• Error rate trend: Sharply increasing — today's 11 failures is the highest ever recorded (previous max was 7 on 2026-08-29), and the 94.76% success rate is the lowest ever recorded.
• Most common recurring issue (today): set_issue_type silent failure (brand new, 10 occurrences) — not a previously-tracked signature.
• Resolved/dormant since 2026-09-23: Smoke Issues jira/linear credential pattern, Auto-Triage Issues add_labels pattern, PR Sous Chef resolve_pull_request_review_thread/dismiss_pull_request_review pattern, the long-dominant PR Sous Chef approve_workflow_run pattern — none recurred today, though absence of PRs touching the relevant conditions is a more likely explanation than an actual fix for most of these.
• Still unresolved: Smoke Project "Bad credentials" (4th occurrence); none of the 14 previously-open recommendations in the solutions backlog have confirmed ship dates.

Metrics and KPIs

• Overall Safe Output Success Rate: 94.76% (199/210)
• Most Reliable Job Type: All job types other than set_issue_type and Smoke Project's update_project/create_project_status_update succeeded today (e.g., add_comment, create_issue, send_slack_message, add_labels, remove_labels, create_check_run, comment_memory all had 100% success in the runs examined)
• Most Problematic Job Type: set_issue_type (0% success in every occurrence it was invoked today)

Next Steps

• Prioritize fixing the set_issue_type handler — this is the highest-confidence, highest-impact, most actionable finding in this monitor's history to date
• Re-audit within 24–48h (not another 8-day gap) to confirm whether set_issue_type recurs at the same 100% rate or is fixed
• Continue monitoring Smoke Project credential issue (4th occurrence, still unresolved)
• Investigate the intermittent raw-log/artifact-retrieval gap in agenticworkflows audit/logs to improve future root-causing speed

Generated by 🔒 Safe Output Health Monitor · claude · agent · 513.1 AIC · ⌖ 9.38 AIC · ⊞ 6.7K · ◷

• expires on Oct 1, 2026, 8:54 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-10-02T04:54:53.494Z.

Closed by Workflow