[safe-output-health] Safe Output Health Report - 2026-10-04

[github-actions[bot]]

🏥 Safe Output Health Report - 2026-10-04

Executive Summary

• Period: Last 24 hours of downloaded logs (307 workflow run directories; 268 had a cached audit.json available for analysis)
• Runs Analyzed: 268
• Safe Output Job Executions: 209 (safe_outputs job across all runs)
• Safe Output Jobs Failed: 4
• Safe Output Jobs Skipped (benign, no items to process): 13
• New subsystem observed: push_ledger_changes job — 3 executions, 2 failed
• Error Clusters Identified: 2 (one brand-new, one a generalization of a long-standing known issue)
• Process note: ⚠️ This is the first safe-output-health audit since 2026-09-23. There is an 11-day monitoring gap (2026-09-24 → 2026-10-03) during which no audit ran. Trend comparisons to 2026-09-23 should be read as discontinuous, not a day-over-day series, and the new push_ledger_changes subsystem described below almost certainly shipped during this gap (it appears in zero prior audits).

Safe Output Job Statistics

Job / Item Type	Total Executions	Failures	Success Rate
safe_outputs (all item types, combined job)	196 (excl. 13 benign skips)	4	97.96%
push_ledger_changes (new subsystem)	3	2	33.3% (very small sample — see caveat)
— create_issue (item count, approx.)	~20	1 of these runs' job hard-failed despite item success	—
— create_discussion (item count, approx.)	~9	1 of these runs' job hard-failed despite item success	—
— add_comment (item count, approx.)	~3	0	100%
— create_pull_request (item count, approx.)	~2	0	100%
— add_labels (item count, approx.)	~2	0	100%
— push_to_pull_request_branch (item count, approx.)	~1	0	100%

Item-type counts are derived from created_items/safe_output_summary across all 268 cached runs and are approximate (some workflows configure multiple types per run).

MCP server health: clean across the board — 0 degraded/failed MCP servers, 0 missing_tools, 0 missing_data signals in any of the 268 cached runs.

Error Clusters

Cluster 1 (NEW): Ledger-reconciliation subsystem fails on empty/near-empty transaction sets

• Count: 2 of 3 observed executions of the new push_ledger_changes job
• Affected Jobs: push_ledger_changes ("Reconcile and push ledger changes" step), and in one case also the safe_outputs job's "Upload validated ledger transactions" step
• Affected Workflows: Daily Caveman Optimizer (run 37156048484), Agentic Workflow Audit Agent (run 37157293798)
• Sample Error:

  failure_kind: agent_logic (Daily Caveman Optimizer) / driver_exit (Agentic Workflow Audit Agent)
  "ledger": { "transactions_added": 0 }
  Step "Reconcile and push ledger changes": conclusion = failure
  

  (No raw stderr/console text was retrievable — see Evidence Gap below.)
• Root Cause: Not yet confirmed with raw error text, but both failures share a telling shape: both occurred when there was nothing (or effectively nothing) to reconcile. Daily Caveman Optimizer is a read-only run that produced 0 safe-output items. Agentic Workflow Audit Agent's agent job crashed before producing any output at all, because its "Start MCP Gateway" step failed with driver_exit (an agent-job infrastructure issue, out of scope for this monitor) — yet push_ledger_changes still ran afterward and still failed trying to reconcile an empty ledger. This points toward an edge case in the zero/near-zero-transaction code path of the ledger reconciliation script, rather than a merge-conflict or volume-related bug.
• Impact: This is a brand-new job type that appears in zero prior audits (the last one, 2026-09-23, has no mention of "ledger" anywhere) and only ran 3 times in this 24h window, so the 67% failure rate is a small-sample signal, not a confirmed rate — but it warrants close watching over the next few audit cycles since it's an unhardened new code path.

Cluster 2 (recurring, now spreading): "Process Safe Outputs" step hard-fails despite the sole intended item succeeding

• Count: 2 occurrences today, part of a 4-occurrence pattern first flagged 2026-09-10
• Affected Jobs: safe_outputs ("Process Safe Outputs" step)
• Affected Workflows: PR Triage Agent (run 37168157408), Auto-Triage Issues (run 37168204963)
• Sample Error:

  failure_kind: agent_logic, error_count: 1
  safe_output_summary.total_items: 1 (matches the single configured item)
  created_items: [ { type: "create_issue", url: ".../issues/65453" } ]   <- item WAS created
  job conclusion: failure                                               <- yet job still hard-fails
  

• Root Cause: Same class of bug previously confirmed for PR Sous Chef's approve_workflow_run and Design Decision Gate's push_to_pull_request_branch (a policy decline getting miscategorized as a hard job failure — see reclassify-protected-file-decline-as-non-failure in the recurring-failures database) — except here there is no policy decline at all. The GitHub write (issue [PR Triage Report] PR triage report - 2026-10-04 #65453, discussion [Auto-Triage] Auto-Triage Issues Report #65451) succeeded cleanly and is visible with a real URL, yet the job still reports failure. This is harder to explain by the existing root cause alone and may point to a distinct bug — plausibly a post-write validation or ledger-recording step failing silently and poisoning the overall step's exit code (notably, this is the same day the new ledger subsystem above was also observed failing).
• Impact: This generalizes a previously workflow-specific pattern (PR Sous Chef, Auto-Triage Issues' add_labels variant) to two more workflows (PR Triage Agent, and a new variant on Auto-Triage Issues), with no fix shipped in over 3 weeks since first confirmed.

Root Cause Analysis

API-Related Issues

None observed today — 0 rate-limit, 0 authentication, 0 permission-error signatures.

Data Validation / Parsing Issues

None observed — both Cluster 2 failures show the agent's output was parsed and processed correctly (the single configured item was created); the failure is downstream of successful processing.

Permission Issues

None confirmed today. The historically dominant approve_workflow_run fork-PR PAT-permission-gap pattern did not recur.

Other Issues — new ledger subsystem

See Cluster 1. This is infrastructure-level (a new git-based ledger reconciliation job), not a credential or permission issue as far as the available evidence shows.

Evidence Gap (process issue, not a code bug)

For all 4 failures, a fresh agenticworkflows logs pull with artifacts:["all"] returned only grader-inferred failure_kind/classification fields (agent_logic, driver_exit) — no raw stderr/console text was present in the exported bundle. This is the same gap flagged in roughly 5 of the last 8 historical audits, and it continues to block confident root-causing from this monitor alone.

Recommendations

Critical Issues (Immediate Action Required)

1. Harden push_ledger_changes for empty/near-empty transaction sets
   • Priority: Critical (new code path, unhardened, 2/3 observed failures)
   • Root Cause: Likely does not gracefully no-op when there are zero transactions to reconcile/push
   • Recommended Action: Add an explicit guard: if transactions_added == 0 (or the agent job never produced output), skip/no-op the reconcile-and-push step rather than attempting and failing it
   • Affected: Daily Caveman Optimizer, Agentic Workflow Audit Agent (likely any workflow with 0 safe-output items going forward)

Bug Fixes Required

1. "Process Safe Outputs" step hard-fails despite successful item creation
   • File/Location: Safe-outputs processing script (likely process_safe_outputs.cjs or equivalent, per historical naming) — same family as the already-tracked protected-files/allowed-files miscategorization, but this variant has no policy decline to explain it
   • Problem: Job conclusion reports failure even though created_items shows the sole configured item was created successfully with a real URL
   • Fix: Investigate whether a post-write step (plausibly the new ledger-transaction recording step) is failing silently and poisoning the parent step's exit code; if so, decouple ledger-write failures from the safe-output item's own success/failure status
   • Affected Jobs: safe_outputs across PR Triage Agent, Auto-Triage Issues (now 4 workflows total including PR Sous Chef)

Configuration Changes

None identified this cycle — no rate-limit, credential, or firewall-policy configuration issues observed.

Process Improvements

1. Restore the daily audit cadence

   • Current State: 11-day gap between this audit and the last one (2026-09-23 → 2026-10-04)
   • Proposed: Confirm the scheduled trigger for this monitor is still active and investigate why it did not fire for over a week
   • Benefits: Closes the blind window during which the new ledger subsystem shipped and started failing without anyone noticing

2. Capture raw stderr/console text in exported log bundles

   • Current State: agenticworkflows logs with artifacts:["all"] returns only structured grader metadata (failure_kind, classification) for job-level failures, never the actual error message
   • Proposed: Include the raw safe-outputs job console log text (or at minimum the safe-output-errors.json file, referenced in several historical audits) in the exported bundle
   • Benefits: Converts the recurring "root cause unconfirmed, inferred from grader shape only" caveat (present in ~5 of the last 8 audits, including this one) into a confirmable diagnosis

Work Item Plans

Work Item 1: Harden ledger reconciliation for empty transaction sets

• Type: Bug Fix
• Priority: Critical
• Description: push_ledger_changes's "Reconcile and push ledger changes" step fails when the run produced zero (or effectively zero, due to an upstream agent-job crash) safe-output transactions. Observed in 2 of 3 total executions of this new job in the last 24h.
• Acceptance Criteria:
  • Reproduce a zero-transaction run locally and confirm the current failure mode
  • Add a guard clause that no-ops the reconcile/push step when there is nothing to reconcile
  • Verify the fix against both observed failure shapes (0-item read-only run; agent job crashed upstream before producing output)
• Technical Approach: Add an early-exit check on transaction count before the git reconcile/push logic runs
• Estimated Effort: Small
• Dependencies: None

Work Item 2: Decouple safe-output item success from downstream ledger/validation step failures

• Type: Bug Fix / Investigation
• Priority: High
• Description: safe_outputs jobs for PR Triage Agent and Auto-Triage Issues reported conclusion: failure even though their sole configured item (an issue, a discussion) was created successfully and appears in created_items with a working URL. This generalizes a previously workflow-specific "partial-success-still-fails" pattern (PR Sous Chef, Auto-Triage Issues' earlier add_labels variant) to two more workflows.
• Acceptance Criteria:
  • Obtain raw error text for the next occurrence (requires Work Item 3 or manual log retrieval) to confirm whether the ledger-write step is the proximate cause
  • If confirmed, make ledger-write failures non-fatal to the parent safe-outputs job's conclusion when the actual GitHub write succeeded
  • Add a regression test covering "item write succeeds, ledger write fails" as an explicit non-failure case
• Technical Approach: Trace the "Process Safe Outputs" step's exit-code logic to confirm whether it aggregates the ledger-transaction step's result
• Estimated Effort: Medium
• Dependencies: Work Item 1 (shares code path)

Work Item 3: Add raw error text to exported log artifacts

• Type: Investigation / Tooling
• Priority: Medium
• Description: The agenticworkflows logs MCP tool's artifacts:["all"] option does not currently surface raw stderr/console text for failed jobs, only grader-inferred failure_kind/classification. This has blocked confident root-causing in ~5 of the last 8 safe-output-health audits, including all 4 failures analyzed today.
• Acceptance Criteria:
  • Confirm whether raw text is captured anywhere upstream (GitHub Actions job logs) but simply not included in the export
  • If so, add it to the exported bundle under a clearly-named field
• Technical Approach: Audit the log-export pipeline for where job console text is dropped
• Estimated Effort: Medium
• Dependencies: None

Historical Context

Compared with the last recorded audit (2026-09-23: 212 runs, 6 failures, 97.17% success — the worst day on record at the time), today's 268-run window shows a similar overall failure count (4) and a slightly better success rate (97.96%), but the comparison is not a clean trend line given the 11-day gap. More notably:

• None of 2026-09-23's specific failure signatures recurred today (Smoke Issues, Smoke Project, Auto-Triage add_labels-beside-create_discussion, Smoke Copilot partial-tool-failures, PR Sous Chef review-thread failures) — all those workflows ran clean this cycle.
• Instead, two different clusters appeared: one brand new (ledger reconciliation, likely shipped during the monitoring gap) and one generalization of a long-known-but-unfixed shape (hard failure despite successful item write) onto two additional workflows.
• The dominant historical pattern (PR Sous Chef approve_workflow_run protected-files decline, 13+ occurrences through 2026-09-22) did not occur even once today.

Trends

• Error rate trend: Roughly flat vs. the last data point (97.17% → 97.96%), but not directly comparable due to the gap
• Most common recurring issue: The "job hard-fails despite successful write" family remains the most persistent unresolved issue across the monitor's entire history, now spanning 4 different workflows
• Improvement since last audit: The specific 2026-09-23 failure modes (credential-related Smoke workflows, review-thread resolution) did not recur

Metrics and KPIs

• Overall Safe Output Success Rate (excluding benign skips): 97.96% (192/196)
• Most Reliable Job Type: add_comment, create_pull_request, add_labels, push_to_pull_request_branch — 100% success on all observed executions today
• Most Problematic Job Type: The new push_ledger_changes job (2 of 3 executions failed) — small sample, flagged for close watching rather than as a confirmed systemic failure rate
• Average Time to Failure: Not computed — insufficient failure count today for a meaningful distribution

Next Steps

• Ship a guard clause for push_ledger_changes on empty/zero-transaction runs (Work Item 1)
• Investigate whether ledger-write failures are poisoning the safe_outputs job's exit code for otherwise-successful item creation (Work Item 2)
• Confirm why the safe-output-health audit did not run for 11 days and restore its cadence
• File a tooling ticket to include raw stderr/console text in agenticworkflows logs artifact exports (Work Item 3)
• Re-run this audit within the next 24-48h to get a second data point on the new ledger subsystem before concluding its failure rate

Generated by 🔒 Safe Output Health Monitor · claude · agent · 402.2 AIC · ⌖ 7.37 AIC · ⊞ 6.6K · ◷

• expires on Oct 4, 2026, 10:32 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-10-05T06:32:18.858Z.

Closed by Workflow