Add Agent Job Health Monitor to own fleet-wide agent-job failure rate

[Copilot]

The Safe Output Health Monitor's 2026-08-11 audit found 103/210 runs (49%) had an agent-job failure, but flagged it as out of scope and recommended routing to "whichever workflow owns agent-job health monitoring" — no such owner exists. Safe Output Health covers only safe-output jobs, Detection Analysis Report only detection jobs, and Agent Performance Analyzer reports per-workflow quality rather than the aggregate fleet rate, so the 49% figure has no recurring home and no way to be confirmed or refuted over time.

Adds .github/workflows/agent-job-health.md (plus compiled .lock.yml), a daily workflow that takes that ownership.

Measurement

• Reports the failure rate two ways: run-weighted (raw agent_failures / total_runs) and workflow-weighted (median/mean per-workflow rate, each workflow counted once). A large gap is the direct signal that chronic offenders like PR Sous Chef or Issue Monster dominate the headline number rather than a genuine fleet-wide regression — the skew hypothesis in the issue.
• Appends a daily record to cache memory; day-over-day moves >10pp are flagged as regression signals.

Triage

• Every failing run is classified tracked (open issue already matches the workflow name or normalized error signature) vs. novel; tracked clusters reference the existing issue instead of spawning duplicates.
• Novel failures are clustered by normalized error signature and failing step (Execute Claude Code CLI, Execute GitHub Copilot CLI, Ingest agent output).

Outputs

• Discussion every run; at most one issue, and only when a novel cluster spans ≥2 workflows or ≥10% of runs.
• noop when no runs exist; explicit instruction to state when the sample is <20 runs.
• Read-only permissions; all writes via safe outputs.

Reuses the existing shared/daily-audit-base.md + shared/aw-logs-24h-fetch.md stack, matching Safe Output Health.

imports:
  - uses: shared/daily-audit-base.md
    with:
      title-prefix: "[agent-job-health] "
      expires: 3d
  - shared/aw-logs-24h-fetch.md

Notes for reviewers

• This establishes the owner and recurring measurement; it does not produce the retrospective answer for the 2026-08-10→11 window. That requires a live run, which the first scheduled execution will provide.
• docs/src/content/docs/agent-factory-status.mdx is generated by make generate-agent-factory, but the committed file appears stale — regenerating produces ~20 unrelated row changes (engines committed as claude regenerate as crush/goose/opencode, and several workflows are absent entirely). I reverted that drift and inserted only the new row by hand, matching the generator's output format. A separate regeneration commit is probably warranted.

• Fixes [deep-report] Investigate 49% agent-job failure rate across 210-run fleet sample (2026-08-10→11) #52093

[Copilot]

Pull request overview

Adds daily fleet-wide monitoring for agent-job failures, addressing the ownership gap identified in #52093.

Changes:

• Measures run- and workflow-weighted failure rates.
• Classifies tracked versus novel failure clusters and records trends.
• Publishes reports through safe outputs and updates workflow documentation.

Show a summary per file

File	Description
.github/workflows/agent-job-health.md	Defines the new health-monitor workflow.
.github/workflows/agent-job-health.lock.yml	Compiled GitHub Actions workflow.
docs/src/content/docs/agent-factory-status.mdx	Adds the monitor to the workflow catalog.

Review details

Tip

Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Suppressed comments (4)

.github/workflows/agent-job-health.md:48

• An empty noop config defaults report-as-issue to true, so a no-runs result will create/comment on the repository’s no-op-runs issue. That contradicts the later rule that issues are emitted only for qualifying novel clusters. Disable issue reporting for this expected no-op path.

  noop:

.github/workflows/agent-job-health.md:77

• aw_info.json does not contain either status or total_tokens; it is generated before the agent job runs. The downloaded run_summary.json is the file that contains run metadata and job_details with job/step conclusions. Pointing the agent at nonexistent fields can make it miss failures or issue an unnecessary MCP query for every run.

Logs for the last 24 hours are pre-downloaded to `/tmp/gh-aw/aw-mcp/logs/`. Each `run-<id>/aw_info.json` holds run metadata (`workflow_name`, `engine_id`, `status`, `total_tokens`).

.github/workflows/agent-job-health.md:108

• A workflow-name match alone does not establish that an issue tracks this failure mode. Any unrelated open issue mentioning a workflow would cause all of its new failures to be classified as tracked, suppressing qualifying novel-cluster issues. Require confirmation that the candidate issue covers the same signature, failing step, or root cause.

- **Tracked** — the workflow or the error signature already has an open issue. Search open issues with the GitHub tools (for example by workflow name, by the error signature, and for known chronic-failure workflows such as PR Sous Chef, Issue Monster, and Contribution Check, and for the Copilot CLI segfault). Record the issue number next to each cluster.

.github/workflows/agent-job-health.md:124

• Appending unconditionally creates duplicate records whenever this dispatchable workflow is rerun on the same UTC date. The next run can then compare against another same-day sample rather than the prior day, so the advertised day-over-day signal is incorrect. Upsert by date and compare against the latest earlier calendar date.

Persist a daily record so regressions are detectable over time. Append one JSON line to `/tmp/gh-aw/cache-memory/agent-job-health/history.jsonl` (create the directory first):

• Files reviewed: 2/3 changed files
• Comments generated: 1
• Review effort level: Balanced