[safe-output-health] 🏥 Safe Output Health Report - 2026-10-06

[github-actions[bot]]

Executive Summary

• Period: Last 24 hours (logs pre-downloaded to /tmp/gh-aw/aw-mcp/logs, 309 run directories, 156 with a safe_outputs job)
• Monitoring cadence: ✅ Healthy, but drifting — 46.6 hours since the previous successful audit (run 37181164399, 2026-10-04T05:53:19Z), against a 48-hour alert threshold. Not yet stale, but worth watching if the interval keeps creeping up.
• Runs Analyzed: 309 workflow runs; 156 of them produced a safe_outputs job
• Workflows Active: Dozens (see per-run detail below); 2 distinct workflows had a genuine safe-output failure
• Safe Output Jobs Executed: 156 (safe_outputs is a single consolidated job per run in this repo's compiled workflows — there are no separate per-type jobs like create_discussion/add_comment)
• Safe Output Jobs Failed: 2 (genuine) + 3 cancelled as a downstream consequence of upstream agent/detection failures (out of scope, noted for completeness) + 4 benign skips (no items to process)
• Error Clusters Identified: 2 (1 brand-new, 1 that confirms and generalizes a previously-unresolved design flaw)

Safe Output Job Statistics

This repository compiles all safe-output handling into a single safe_outputs job per run (it internally dispatches to whichever tool types — create_issue, add_comment, ledger_append, set_issue_type, etc. — the agent requested), rather than one job per output type. The table below reflects that architecture.

Job	Total Executions	Success	Failure	Skipped (benign)	Cancelled (upstream failure)	Success Rate (excl. skips/cancels)
safe_outputs	156	147	2	4	3	98.66%

Error Clusters

Cluster 1: ledger_append schema validation failure (NEW)

• Count: 1 occurrence
• Affected Job: safe_outputs → "Process Safe Outputs" step (also cascades into push_ledger_changes → "Reconcile and push ledger changes")
• Affected Workflow: Daily Caveman Optimizer (scheduled run, main branch)
• Run: 37372003041
• Sample Error:

  ERR_VALIDATION: Handler manager failed: Invalid built-in transaction fields
  

  Rejected payload:

  {"key":"37372003041","ledger":"caveman-run-history","operation":"upsert",
   "value":{"date":"2026-10-05","files_optimized":0,"files_processed":5,
            "planned_outcome":"noop","record_type":"caveman_run","run_id":"37372003041"},
   "type":"ledger_append"}

• Root Cause: The workflow's ledger_append item carries fields (date, files_optimized, files_processed, planned_outcome, record_type, run_id) that don't match the safe-outputs ledger handler's built-in transaction field schema, so the handler manager rejects the whole transaction.
• Impact: Medium. The workflow silently loses its own run-history ledger entry on every occurrence, and the failure cascades into the separate push_ledger_changes job (nothing valid to reconcile). Distinct from the 2026-10-04 "ledger-reconciliation-empty-transaction-bug" cluster, which was about empty transactions — this one is about invalid fields in a non-empty transaction.

Cluster 2: target: "triggering" hard-fails outside its expected context (ROOT CAUSE CONFIRMED, GENERALIZES A KNOWN UNRESOLVED ISSUE)

• Count: 1 new occurrence today; 3 total confirmed/explained across 2 workflows since 2026-08-31
• Affected Job: safe_outputs → "Process Safe Outputs" step
• Affected Workflow: Smoke Copilot (triggered by a PR comment, PR #65937)
• Run: 37375351430
• Sample Error:

  E099: 1 safe output(s) failed:
    - set_issue_type: Target is "triggering" but not running in issue context, skipping set_issue_type
  

• Root Cause: Any safe-output type that resolves its target as "triggering" hard-fails the entire safe_outputs job when the triggering event lacks the context that type needs — set_issue_type needs an issue, but this run was triggered from a PR comment (no issue context). This is the identical error shape already confirmed on 2026-08-31 for submit_pull_request_review (which needs PR context and hard-fails on non-PR triggers). The semantically identical no-context condition for add_comment is instead correctly treated as a soft skip. This confirms the bug lives in the shared "triggering" target-resolution layer, not in any one tool's handler.
• Impact: Medium. Not a correctness problem — the agent's intent genuinely didn't apply — but it has polluted safe-output success-rate metrics for at least 2 tool types across 2 workflows for over 5 weeks with no fix shipped. It also retroactively explains one of the 5 previously-unexplained failures in the 2026-09-23 "Smoke Copilot partial-tool-failure cluster" (set_issue_type was one of the 5 failed items there; add_comment and haiku_printer in that same cluster remain unexplained).

Root Cause Analysis

Validation Issues

ledger_append's built-in schema mismatch (Cluster 1) is a straightforward data-shape bug: the workflow emits fields the handler doesn't recognize as valid.

Logic / Design Issues

The "triggering" target-resolution hard-fail (Cluster 2) is a design inconsistency: the processor has two different failure modes for "the agent asked for something that doesn't apply in this run's context" — a soft skip (used by add_comment) and a hard job-failing error (used by submit_pull_request_review and now confirmed for set_issue_type). The inconsistency, not the no-context condition itself, is the bug.

Other Issues

3 safe_outputs jobs were marked cancelled this period (Daily Ambient Context Optimizer, Copilot PR Prompt Pattern Analysis, Daily Project Performance Summary Generator) — all 3 belong to overall-failed runs and were almost certainly cancelled as a downstream consequence of an upstream agent/detection job failure, not a safe-outputs-specific bug. Out of scope for this monitor (agent/detection failures are tracked by other workflows) but noted for completeness. 4 jobs were skipped, all in overall-succeeded runs with nothing to process — expected, benign behavior.

Recommendations

Bug Fixes Required

1. Soft-skip "triggering" targets outside their expected context, at the shared resolution layer

   • File/Location: Safe-outputs processor's target-resolution code for target: "triggering" (shared across tool handlers — exact file not retrievable from this audit's log bundle, but the fix point is the one place that resolves "triggering" to an issue/PR number before dispatch)
   • Problem: When the triggering event lacks the context a tool needs (issue for set_issue_type, PR for submit_pull_request_review), the processor raises a job-failing error (E099/##[error]) instead of a soft skip.
   • Fix: Make the no-context case for "triggering" targets a soft skip/no-op for that single item, matching add_comment's existing correct behavior, across all tool types that support target: "triggering".
   • Affected: set_issue_type, submit_pull_request_review confirmed; add_comment/create_pull_request_review_comment/haiku_printer failures in the 2026-09-23 Smoke Copilot cluster remain unexplained and should be re-checked against this same pattern.

2. Fix or allow-list Daily Caveman Optimizer's ledger_append fields

   • File/Location: Daily Caveman Optimizer workflow definition (its ledger_append prompt/config) and/or the safe-outputs ledger handler's built-in transaction field schema
   • Problem: The workflow emits {date, files_optimized, files_processed, planned_outcome, record_type, run_id}, none of which the handler's built-in schema accepts.
   • Fix: Either trim the workflow's emitted fields to the handler's accepted schema, or extend the schema's allow-list if these custom fields are intentional and useful.
   • Affected: Daily Caveman Optimizer's safe_outputs and push_ledger_changes jobs.

Process Improvements

1. Raw error retrieval worked this cycle — keep using it
   • Current State: ~5 of the last 8 prior audits reported an "evidence gap" where raw stderr/error text wasn't retrievable, leaving only grader-inferred failure_kind fields.
   • Proposed: Today, agenticworkflows logs with {"workflow_name": "<slug>", "start_date": ..., "end_date": ..., "artifacts": ["all"]} successfully retrieved safe-output-errors.json (with exact error text and the rejected payload) for both failures. Continue using this exact invocation shape (slug-form workflow_name, narrow date range, artifacts: ["all"]) as the default first move for any future failure investigation, before falling back to grader-inferred fields.
   • Benefits: Root-caused both of today's failures with concrete, actionable error text instead of speculation.

Work Item Plans

Work Item 1: Soft-skip no-context "triggering" targets in the safe-outputs processor

• Type: Bug Fix
• Priority: Medium-High
• Description: Generalize the existing correct add_comment no-context handling to all tool types that accept target: "triggering", so a context mismatch (no issue / no PR) produces a soft skip for that one item instead of failing the whole safe_outputs job.
• Acceptance Criteria:
  • set_issue_type with target: "triggering" and no issue context produces a skipped/no-op result, not a job failure
  • submit_pull_request_review with target: "triggering" and no PR context produces a skipped/no-op result, not a job failure
  • A regression test covers both cases
• Technical Approach: Locate the shared target-resolution code path for "triggering" and change its error path to a soft-skip path, reusing whatever mechanism already makes add_comment behave correctly.
• Estimated Effort: Small–Medium
• Dependencies: None known

Work Item 2: Fix Daily Caveman Optimizer's ledger_append schema mismatch

• Type: Bug Fix
• Priority: Medium
• Description: Align the workflow's emitted ledger_append fields with the ledger handler's built-in transaction schema (or extend the schema).
• Acceptance Criteria:
  • Daily Caveman Optimizer's ledger_append item passes handler validation
  • push_ledger_changes's "Reconcile and push ledger changes" step succeeds in the same run
• Technical Approach: Compare the workflow's ledger_append prompt/config against the handler's accepted built-in field list; adjust whichever side is wrong.
• Estimated Effort: Small
• Dependencies: None known

Historical Context

Trends

• Error rate trend: Stable-to-improving. 97.96% (2026-10-04) → 98.66% (today), though the two periods aren't a true apples-to-apples comparison (different denominators/methodology across the 11-day gap noted on 2026-10-04).
• Most common recurring issue class: "Benign/intended decline surfaced as a hard job failure" — this is now the 4th distinct safe-output type confirmed to hit this shape (approve_workflow_run protected-files decline, push_to_pull_request_branch allowed-files decline, submit_pull_request_review no-PR-context, and now set_issue_type no-issue-context), reinforcing that this is a systemic classification problem in the safe-outputs processor, not isolated per-tool bugs.
• Improvement since last audit: None of the historically dominant signatures (PR Sous Chef approve_workflow_run, Design Decision Gate allowed-files decline, Smoke Issues jira/linear credentials, Smoke Project update_project bad credentials, Auto-Triage Issues add_labels, same-PR concurrency race) recurred in this window — all previously-flagged workflows ran clean today.

Metrics and KPIs

• Overall Safe Output Success Rate: 98.66% (147/149 genuine executions, excluding 4 benign skips and 3 upstream-cancelled jobs)
• Most Reliable Job Type: N/A — single consolidated safe_outputs job per run in this repo's architecture
• Most Problematic Safe-Output Type (this period): ledger_append (Daily Caveman Optimizer) and set_issue_type (Smoke Copilot), one failure each
• Average Time to Failure: Both failures occurred quickly within the "Process Safe Outputs" step (~1–5 seconds), consistent with validation-time rejection rather than a long-running process timeout

Next Steps

• File/track a fix for the shared "triggering" target no-context hard-fail (Work Item 1)
• File/track a fix for Daily Caveman Optimizer's ledger_append schema mismatch (Work Item 2)
• Re-check add_comment and haiku_printer's still-unexplained failures from the 2026-09-23 Smoke Copilot cluster against the newly-confirmed "triggering" pattern, in case they share the same root cause
• Watch the monitoring cadence — it is healthy today but has drifted to 46.6h of a 48h threshold; investigate if the gap keeps growing cycle over cycle

Generated by 🔒 Safe Output Health Monitor · claude · agent · 486.9 AIC · ⌖ 10.1 AIC · ⊞ 6.7K · ◷

• expires on Oct 6, 2026, 8:47 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-10-07T04:47:44.094Z.

Closed by Workflow