[safe-output-health] Safe Output Health Report - 2026-10-14

[github-actions[bot]]

Executive Summary

• Period: Last ~24 hours (239 workflow run directories downloaded and analyzed; 236 contained a safe_outputs job entry)
• Runs Analyzed: 239
• Safe Output Jobs Executed: 236 (217 success, 17 skipped due to upstream job failure/no-op, 2 failed)
• Safe Output Jobs Failed: 2
• Success Rate (of executed jobs): 99.1% (217/219, excluding the 17 skipped)
• Error Clusters Identified: 2 (1 recurring, 1 new)

Monitoring cadence: healthy, but tight. This audit ran 47.9 hours after the previous successful publish (run 38204331412, 2026-10-12T04:34:23Z) — just under the 48h alert threshold. The prior audit itself followed a ~144h gap flagged as a recurring self-health issue. No gap this cycle, but the margin is thin; see Process Improvements below.

Safe Output Job Statistics

Job Type	Total Executions	Failures	Success Rate
safe_outputs (consolidated job — all safe-output types route through one job per run)	236	2	99.1%†
push_ledger_changes (adjacent ledger-reconciliation job)	1 observed this cycle	1	0%‡

† Calculated over the 219 executed (non-skipped) runs. 17 runs show safe_outputs as skipped because an upstream job (activation/agent) failed first, not because of a safe-output defect.
‡ Only 1 push_ledger_changes execution was visible in this window's logs, and it failed in the same run as Cluster 1 below — too small a sample to treat as a rate, but it's a direct cascade of that cluster.

Per-type breakdown (create_discussion, add_comment, create_issue, etc.) isn't separately measurable from local job logs: gh-aw consolidates all safe-output item types into a single safe_outputs job per run, so failures are tracked at the job+step level, not per output type.

Error Clusters

Cluster 1: ledger_append validation failure → push_ledger_changes reconcile failure (RECURRING)

• Count: 1 occurrence this cycle (2nd distinct workflow to hit this signature; 3rd total occurrence tracked since 2026-10-05)
• Affected Jobs: safe_outputs (step: Process Safe Outputs), push_ledger_changes (step: Reconcile and push ledger changes)
• Affected Workflow: Agentic Workflow Audit Agent — run 38469476854
• Sample evidence:

  safe_outputs job → "Process Safe Outputs" step: conclusion=failure
  safe_outputs job → "Upload validated ledger transactions" step: skipped
  Run-level ledger summary: { "transactions_added": 0 }
  push_ledger_changes job → "Reconcile and push ledger changes" step: conclusion=failure
  (meanwhile, in the same batch: create_discussion succeeded -> discussion #68537)
  

• Root Cause: This matches an already-tracked pattern (caveman-optimizer-ledger-append-invalid-builtin-fields, first confirmed 2026-10-05 for Daily Caveman Optimizer): a ledger_append safe-output item whose value fields don't match the safe-outputs ledger handler's expected built-in transaction schema gets rejected, failing Process Safe Outputs even though other items in the same batch (here, create_discussion) succeed independently — confirming per-item isolation still holds. The rejection leaves nothing for push_ledger_changes to reconcile, so that job fails too. This is now the second distinct, unrelated workflow to hit the same failure shape, which raises confidence this is a shared bug in the safe-outputs ledger handler's schema/allow-list rather than a one-off authoring mistake in a single workflow.
• Confidence caveat: the raw ERR_VALIDATION message text (available for the 2026-10-05 occurrence) was not retrievable for this run — agenticworkflows audit with artifacts:["all"] returned only the structured summary, not safe-output-errors.json. This occurrence is matched by step-failure + cascade shape only, so treat it as plausible, not confirmed, until raw error text is pulled for run 38469476854 specifically.
• Impact: Medium. The ledger entry and its downstream git reconciliation are lost for this run, though the discussion that was the workflow's primary output still published successfully. The 2026-10-12 audit found push_ledger_changes clean across 4 executions, so this bug is intermittent/payload-dependent, not an always-on break — but it has now reappeared after a clean cycle, confirming the underlying fix from 2026-10-05 is still unshipped.

Cluster 2: Copilot-engine sub-agent model routing failure → empty safe-output payload (NEW)

• Count: 1 occurrence (new signature, not previously tracked)
• Affected Job: safe_outputs (step: Process Safe Outputs)
• Affected Workflow: PR Code Quality Reviewer — run 38476997137
• Sample error (from the agent job's sub-agent invocation, which starved the safe-output payload):

  400 Cannot translate Copilot request feature 'include' between Responses and Chat
  Completions. Routing model "claude-haiku-5.5" to /chat/completions is incompatible:
  this request needs /responses to preserve 'include'. Choose a model supporting
  /responses for this request, or use function tools/feat...
  

  Sub-agent: grumpy-coder (declared model tier: small, resolved to claude-haiku-5.5). The safeoutputs.submit_pull_request_review MCP tool call is logged with status: "incomplete" and output_size: 0; the run shows 0 created items.
• Root Cause: This is an upstream agent-job defect, not a bug in the safe-outputs processor itself. A sub-agent requesting the "small" model tier resolves to claude-haiku-5.5 under the Copilot engine, which then tries to route the call through the /chat/completions API surface. That surface can't carry the include request feature the call needs — only /responses supports it — so the sub-agent invocation hard-fails with a 400 before producing output. Because this workflow's submit_pull_request_review safe-output depended on that sub-agent's result, the main agent session ended with no valid payload, and Process Safe Outputs then failed with nothing to process.
• Impact: Low-medium, single occurrence. But it's a generic engine/model-routing bug, not workflow-specific — any workflow that routes a sub-agent through the Copilot engine at the "small"/haiku tier while requesting the include feature would hit the same wall.

Root Cause Analysis

API-Related Issues

None this cycle. No GitHub API rate-limit, authentication, or permission errors observed in any of the 236 safe_outputs job executions (github_rate_limit_usage stayed well under quota across sampled runs).

Data Validation Issues

Cluster 1 (ledger_append schema mismatch) is the dominant validation issue this cycle, and it's a repeat offender first confirmed 9 days ago in a different workflow.

Permission Issues

None observed.

Other Issues

Cluster 2 is an engine/model-routing compatibility bug (Copilot engine + small-model sub-agents + the include request feature), not a safe-outputs-processor bug per se — flagged here because its visible symptom is a safe_outputs job failure, and because it directly caused a lost safe-output item (the PR review was never submitted).

Recommendations

Critical Issues (Immediate Action Required)

None. Both failures this cycle are isolated (1 occurrence each) with confirmed per-item isolation — no batch-wide or systemic outage.

Bug Fixes Required

1. Ledger handler built-in transaction field validation (recurring, 3rd occurrence since 2026-10-05)

   • Problem: ledger_append safe-output items with fields outside the handler's expected built-in schema are rejected, failing the whole safe_outputs job step and cascading into push_ledger_changes reconcile failures.
   • Fix: Either tighten the workflow-side ledger_append payload schemas (Daily Caveman Optimizer, Agentic Workflow Audit Agent) to only emit accepted built-in fields, or extend the handler's allow-list to accept workflow-specific custom fields if they're intentional. This was already recommended on 2026-10-05 and remains unshipped.
   • Affected: safe_outputs job, push_ledger_changes job.

2. Copilot engine model-routing for small/haiku-tier sub-agents

   • Problem: Sub-agents requesting the "small" model tier under the Copilot engine resolve to claude-haiku-5.5 and get routed to /chat/completions, which can't carry the include request feature some calls need — causing a hard 400 and an empty/incomplete safe-output payload downstream.
   • Fix: Route calls requiring include to /responses regardless of model tier, or ensure "small"/haiku-tier resolution only maps to models/endpoints that support the request's required features.
   • Affected: agent job (sub-agent invocation), safe_outputs job (starved of valid payload as a result).

Configuration Changes

None identified this cycle.

Process Improvements

1. Monitoring cadence margin

   • Current State: This audit ran at 47.9h against a 48h alert threshold — effectively right at the edge, following a prior audit that itself recovered from a ~144h self-health gap.
   • Proposed: Tighten the schedule trigger or lower the alert threshold (e.g., flag at 36h) so a single missed or delayed trigger doesn't silently erode the margin to zero again.
   • Benefits: Earlier warning before a full missed-cycle gap reopens, consistent with the self-health issue flagged critical on 2026-10-07 and partially recurring on 2026-10-12.

2. Raw error-text retrievability for ledger failures

   • Current State: For 2 of the 3 tracked ledger_append occurrences (2026-09-05-shape audits and now 2026-10-14), agenticworkflows audit/logs with artifacts:["all"] returned only the structured summary, not the raw safe-output-errors.json with the exact ERR_VALIDATION message.
   • Proposed: Investigate why raw safe-output error artifacts aren't consistently retrievable for ledger-related failures specifically (they were retrievable for the original 2026-10-05 confirmation), and whether retention/upload conditions differ for this artifact type.
   • Benefits: Faster, higher-confidence root-causing of future ledger_append failures instead of shape-matching against prior occurrences.

Work Item Plans

Work Item 1: Fix ledger_append built-in transaction field validation

• Type: Bug Fix
• Priority: Medium (recurring, cross-workflow, but low-frequency and non-blocking for the workflow's primary output)
• Description: The safe-outputs ledger handler rejects ledger_append items whose value fields don't match its built-in transaction schema, failing Process Safe Outputs and cascading into push_ledger_changes reconcile failures. Confirmed for Daily Caveman Optimizer (2026-10-05) and now matched by shape for Agentic Workflow Audit Agent (2026-10-14, run 38469476854).
• Acceptance Criteria:
  • Confirm whether the 2026-10-14 occurrence shares the exact ERR_VALIDATION cause via raw safe-output-errors.json for run 38469476854
  • Either constrain affected workflows' ledger_append payloads to the handler's accepted schema, or extend the handler's allow-list
  • Verify push_ledger_changes no longer fails when a ledger_append item is rejected (job should reconcile the remaining valid transactions, or clearly skip rather than fail, when there's nothing to reconcile)
• Technical Approach: Compare the Daily Caveman Optimizer and Agentic Workflow Audit Agent ledger_append payload shapes against the handler's built-in field schema; align or extend as needed.
• Estimated Effort: Small
• Dependencies: None

Work Item 2: Fix Copilot engine routing for small/haiku-tier sub-agents requiring include

• Type: Bug Fix
• Priority: Medium
• Description: Sub-agents requesting the "small" model tier under the Copilot engine resolve to claude-haiku-5.5, which gets routed to /chat/completions — incompatible with the include request feature some calls need, causing a 400 and starving the safe-output payload.
• Acceptance Criteria:
  • Calls requiring include are routed to /responses regardless of resolved model tier, or the "small" tier is mapped to a model/endpoint combination that supports the request's required features
  • A failed sub-agent invocation of this kind no longer silently produces an empty/incomplete safe-output payload — either the main agent retries without the failed sub-agent's contribution, or the failure is surfaced clearly rather than presenting as a generic Process Safe Outputs failure
• Technical Approach: Audit the Copilot engine's model-tier-to-endpoint routing logic for the include feature flag; add a compatibility check or endpoint override for small/haiku-tier requests that need it.
• Estimated Effort: Small–Medium
• Dependencies: None

Historical Context

Compared against the 2026-10-12 audit (0 failures across 178 executions, 100% success — the first fully clean cycle since 2026-09-11):

Trends

• Error rate trend: Slightly up from 0% (2026-10-12) to 0.9% of executed jobs (2026-10-14) — still very low, 2 isolated occurrences, no systemic regression.
• Most common recurring issue: The ledger_append validation / push_ledger_changes reconciliation bug family, first confirmed 2026-10-05, clean on 2026-10-12, now recurred in a new workflow on 2026-10-14. This continues the historical pattern (also seen with the approve_workflow_run protected-files miscategorization) where safe-output failures are low-frequency, payload-dependent, and tend to reappear in new workflows rather than being fully eliminated.
• Improvement since last audit: None of the 2026-10-12 audit's "absent recurring patterns" (PR Sous Chef approve_workflow_run, Design Decision Gate allowed-files decline, Smoke Issues/Project credential failures, hardcoded smoke temp-id, triggering-target hard-fail) recurred this cycle either — those remain either fixed or simply not exercised in this window.

Metrics and KPIs

• Overall Safe Output Success Rate: 99.1% (217/219 executed safe_outputs jobs; 91.95% if counting the 17 upstream-skipped runs as a flat denominator, though those skips are not safe-output defects)
• Most Reliable Job Type: safe_outputs — 217 clean executions out of 219, with both failures isolated to single runs
• Most Problematic Job Type: push_ledger_changes — only 1 execution observed this window, and it failed (100% failure rate on a sample of 1; not statistically meaningful but tracks the recurring ledger bug)
• Average Time to Failure: Both failures surfaced within the first ~4 seconds of their respective Process Safe Outputs steps (step "started_at"→"completed_at" spans of ~3-4s before conclusion=failure)

Next Steps

• Retrieve raw safe-output-errors.json for run 38469476854 to confirm (not just shape-match) the ledger_append validation cause
• Ship the ledger_append schema/allow-list fix tracked since 2026-10-05 (now affecting 2 workflows)
• Investigate and fix the Copilot engine's small/haiku-tier routing incompatibility with the include request feature
• Tighten this monitor's own cadence margin so future runs don't land within ~0.1h of the 48h alert threshold

Generated by 🔒 Safe Output Health Monitor · claude · agent · 471.4 AIC · ⊞ 6.7K · ◷

• expires on Oct 14, 2026, 8:42 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-10-15T04:42:53.808Z.

Closed by Workflow