[safe-output-health] Safe Output Health Monitor 2026-08-24

[github-actions[bot]]

Executive Summary

Audited all agentic workflow runs in github/gh-aw from the last 24 hours (290 runs analyzed, spanning roughly 2026-08-23 04:00 UTC through 2026-08-24 04:08 UTC), focused exclusively on safe-output job failures (the centralized safe_outputs job that processes create-issue, create-pull-request, create-discussion, add-comment, add-labels, update-issue, push-to-pull-request-branch, missing-tool, etc.). Agent-job failures, detection-job failures, and activation failures are out of scope for this monitor and are covered elsewhere.

2 safe-output job failures were found in the window, both new failure signatures not seen in the prior two daily audits (2026-08-22, 2026-08-23):

1. Daily Harness Experiment Proposer (run 32670714843) — failed at the "Generate GitHub App token" step, before any safe-output was ever processed.
2. Designer Drift Audit (run 32688268842) — failed at the "Process Safe Outputs" step itself, while handling a create-issue output.

The positive news: the previously recurring "PR Sous Chef / safe_outputs / Process Safe Outputs" failure (3 occurrences across the 2026-08-22 and 2026-08-23 audits) did not recur across PR Sous Chef's 39 runs in this window — a break in what had been a flat, unresolved trend.

A tooling note: this audit's persistent cache-memory store at /tmp/gh-aw/cache-memory/safe-output-health/ was read-only in this sandbox run (writes failed with EROFS), so today's findings could not be appended to the historical JSON files there. This report captures the full findings; a future run with write access should backfill 2026-08-24.json using the data below.

Job Statistics

Metric	Value
Runs analyzed	290
Distinct workflows with a safe_outputs job	290 (job present in every run; not all execute safe-output processing)
safe_outputs jobs failed	2
safe_outputs jobs succeeded (best-effort count)	~202
safe_outputs jobs skipped or other (best-effort)	~86
Approx. success rate among matched/executed jobs	~99.0%
missing_tool / missing_data / noop-driven failures	0 confirmed

Caveat: run_summary.json serializes job fields (name/status/conclusion) in inconsistent key order across runs, which breaks exact regex-based aggregation in this sandbox (no PCRE2 lookaround available, and a full 290-file "skipped" scan timed out at 20s). The succeeded/skipped split is a best-effort count, not an authoritative total. The failure count of 2 is confirmed via targeted per-run inspection, independent of the aggregate regex counts.

Error Clusters

Cluster 1: GitHub App token generation failure (NEW)

• Workflow: Daily Harness Experiment Proposer
• Run: 32670714843 (workflow_dispatch, 2026-08-23T22:29:42Z, engine: Claude Code 2.1.237)
• Failed step: Generate GitHub App token (in the safe_outputs job)
• Impact: All downstream steps (Checkout repository, Configure Git credentials, Process Safe Outputs) were skipped. Zero safe-output items were applied, even though the agent turn completed successfully and called the noop safe-output tool.
• Why a GitHub App token at all: this workflow's safe-outputs.github-app config (client-id: ${{ vars.APP_ID }}, private-key: ${{ secrets.APP_PRIVATE_KEY }}) is required because its create-pull-request output writes to protected paths (.github/workflows/*.md, *.lock.yml) that exceed default GITHUB_TOKEN permissions.
• Occurrences this window: 1 (first occurrence of this signature).

Cluster 2: Process Safe Outputs failure in create-issue handling (NEW)

• Workflow: Designer Drift Audit
• Run: 32688268842 (schedule, 2026-08-24T03:58:09Z, engine: copilot 1.0.80)
• Failed step: Process Safe Outputs (in the safe_outputs job) — all preconditions (checkout, git credentials, GH_HOST config) succeeded first.
• Impact: The workflow's single create-issue output (with title-prefix, labels, close-older-issues: true, expires: 7d) was not applied.
• Occurrences this window: 1 (first occurrence of this signature; structurally similar in shape to the previously-tracked PR Sous Chef "process-safe-outputs-batch-abort" pattern, but a different workflow and a single-output-type run rather than a large mixed batch, so tracked as a distinct pattern).

Cluster 0 (for context, not new): PR Sous Chef recurring pattern — broken streak

• Previously: 3 occurrences across 2026-08-22/23 of a safe_outputs / Process Safe Outputs failure, tracked as confirmed_recurring / flat_unresolved.
• This window: 0 failures across 39 PR Sous Chef runs.

Root Cause Analysis

Raw step-level stderr/console output for the safe_outputs job is not retrievable via the agenticworkflows MCP CLI or the pre-downloaded log artifacts in this sandbox — only structured per-step pass/fail conclusions are available (confirmed again this run via a targeted re-fetch with artifacts: ["all"], which still returned only episodes: [] and aggregate counts). This is a standing tooling limitation, not something retried indefinitely.

Given that constraint, root causes are hypothesized from structural evidence:

• Cluster 1 (App token): the failure occurs entirely within the "Generate GitHub App token" step, before checkout or any safe-output logic runs. This isolates the fault to GitHub App authentication/credential configuration — most likely vars.APP_ID or secrets.APP_PRIVATE_KEY being invalid, expired, missing, or the App installation lacking access to github/gh-aw. It is a workflow_dispatch run (manually triggered by pelikhan), so this is not (yet) a scheduled-recurrence risk, but it will recur on every future dispatch/schedule of this workflow until the App credentials are fixed.
• Cluster 2 (Process Safe Outputs): the failure occurs squarely inside the processing/validation logic for a single, simple create-issue output — no unusual batch size or multi-type complexity like PR Sous Chef. This suggests either a transient issue (API rate limit, network blip inside the sandboxed job) or a validation edge case specific to this run's issue content (e.g. title/label conflict with close-older-issues: true, or an oversized/malformed body). Cannot be confirmed without raw logs.

Recommendations

Immediate Actions

• Verify vars.APP_ID and secrets.APP_PRIVATE_KEY are correctly set for the repo/org and that the GitHub App has an active installation with write access to github/gh-aw, specifically for the Daily Harness Experiment Proposer workflow's needs (writing .github/workflows/*).
• Re-run Designer Drift Audit's next scheduled execution and watch whether the Process Safe Outputs failure recurs; a single occurrence is not yet enough to confirm a systemic bug vs. a transient blip.

Bug Fixes

• If the App token failure turns out to be a credential expiry/rotation issue rather than a one-off, add a pre-flight validation/clearer error surface for GitHub App token generation failures in the safe_outputs job, so future audits don't have to infer root cause from step isolation alone.
• Continue tracking whether "Process Safe Outputs" failures with small, single-output payloads (this cluster) share a root cause with the historically larger PR Sous Chef batch-abort pattern, or are genuinely distinct.

Configuration Changes

• No safe-outputs configuration changes are recommended yet for either new cluster — both need either a confirmed credential issue (Cluster 1) or a second occurrence (Cluster 2) before a config-level fix can be scoped.

Process Improvements

• Add raw step-level log capture/upload as a build artifact for the safe_outputs job specifically (even just stderr for the "Generate GitHub App token" and "Process Safe Outputs" steps), since this has now been the single biggest blocker to root-causing safe-output failures across all three daily audits (2026-08-22, 2026-08-23, 2026-08-24).
• This audit's cache-memory store (/tmp/gh-aw/cache-memory/safe-output-health/) was read-only in this run — confirm write access is restored so daily audit history and recurring-pattern tracking stay continuous.

Work Item Plans

Cluster 1 — GitHub App token generation failure (Daily Harness Experiment Proposer)

1. Confirm APP_ID/APP_PRIVATE_KEY validity and App installation scope for github/gh-aw.
2. If invalid/expired, rotate/reconfigure the App credentials.
3. Re-trigger the workflow via workflow_dispatch to confirm the fix.
4. If credentials are valid, escalate to raw-log capture (Process Improvements above) to get the actual failure reason.

Cluster 2 — Process Safe Outputs failure (Designer Drift Audit)

1. Watch the next 1-2 scheduled runs of Designer Drift Audit for recurrence.
2. If it recurs, request raw step logs/artifact capture to identify whether it's a create-issue validation error, rate limit, or transient infra issue.
3. If it does not recur, close this as a transient one-off but keep it in error-patterns.json at occurrence-count 1 for future correlation.

Historical Context and Trends

Audit Date	Runs Analyzed	Safe-Output Failures	Success Rate	Primary Finding
2026-08-22	841	2	99.45%	PR Sous Chef Process Safe Outputs failed in 2/24 sampled runs
2026-08-23	224	2	98.94%	1 recurrence of the PR Sous Chef pattern (3rd occurrence) + 1 new infra-flake at "Checkout actions folder"
2026-08-24 (this audit)	290	2	~99.0%	2 brand-new failure signatures (GitHub App token gen; Process Safe Outputs in a single create-issue run) — PR Sous Chef's recurring pattern did not reproduce

Trend read: the failure count has stayed flat at 2/day for three consecutive audits, but the composition changed entirely today — no repeat of the previously dominant PR Sous Chef pattern, replaced by two first-time signatures in two different workflows. This points away from a single systemic bug and toward either (a) normal background noise at a low, roughly constant rate across many workflows, or (b) coincidence that should be watched for another 1-2 days before drawing conclusions either way.

Metrics and KPIs

• Safe-output job failure rate (3-day rolling): 2/841 (0.24%), 2/224 (0.89%), 2/290 (0.69%)
• Distinct failure signatures seen to date: 4 (process-safe-outputs-batch-abort [PR Sous Chef], checkout-actions-folder-infra-flake [PR Sous Chef], github-app-token-generation-failure [Daily Harness Experiment Proposer, new], process-safe-outputs-create-issue [Designer Drift Audit, new])
• Recurring (2+ occurrence) patterns currently open: 1 (PR Sous Chef batch-abort, now on a broken streak — 0 occurrences in the most recent window)
• Root-cause confirmation rate: 0/4 patterns have a confirmed root cause from raw logs (100% inferred from structural/step-isolation evidence) — this remains the top process gap across all three audits.

Next Steps

• Validate GitHub App credentials for Daily Harness Experiment Proposer and re-trigger to confirm the fix.
• Monitor Designer Drift Audit's next 1-2 scheduled runs for recurrence of the Process Safe Outputs failure.
• Monitor PR Sous Chef for one more audit cycle to confirm its recurring pattern is genuinely resolved rather than a temporary lull.
• Restore write access to /tmp/gh-aw/cache-memory/safe-output-health/ so this audit's findings can be persisted into the historical JSON tracking files.
• Prioritize adding raw step-level log capture for the safe_outputs job so future audits can move from structural inference to confirmed root causes.

Generated by 🔒 Safe Output Health Monitor · claude · agent · 250.9 AIC · ⌖ 11 AIC · ⊞ 10.8K · ◷

• expires on Aug 24, 2026, 9:03 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Safe Output Health Monitor.

A newer discussion is available at Discussion #55646.