Agent Performance Report - Week of 2026-09-12

[github-actions[bot]]

Executive Summary

• Agents analyzed: 41 currently-active workflows (of 311 total tracked in metrics/latest.json); cross-checked live against 30 recent "produced no safe outputs" / failure issues (2026-08-05 → 2026-09-12) via GitHub search since the shared metrics snapshot is stale.
• Total outputs reviewed: metrics/latest.json snapshot outputs are near-zero across the board (top producer pr-sous-chef: 5 outputs; daily-community-attribution: 1 PR). This snapshot itself is stale (dated 2026-09-01, 11 days old) and not representative of current volume — see Data Quality Note below.
• Systemic finding (new, confirmed this run): the codex engine + gpt-5.3-codex model pairing across 69 workflows in this repo is producing a recurring model_not_supported_error + missing_safe_outputs failure. This is the dominant root cause behind the majority of "no safe outputs" issues filed in the last 7 days (12+ distinct workflows affected, still occurring today).
• Top performers (by live evidence, not stale snapshot): pr-sous-chef (4/4 executed runs successful, active output), Workflow Health Manager (clean weekly reports, no false escalations this run).
• Needs improvement: every codex-engine workflow currently pinned to model: {copilot,openai}/gpt-5.3-codex — see systemic issue below. Q and squad workflows show 100% action_required/skip on every observed run (command-gating; see note).

Data Quality Note — metrics/latest.json is 11 days stale

/tmp/gh-aw/repo-memory/default/metrics/latest.json is timestamped 2026-09-01T02:50:27Z. Metrics
Collector itself is the victim of the same codex/gpt-5.3-codex model_not_supported_error bug
documented below (issues #58895, #59105, #59344, #59611, #59851, #60157 — 6 recurrences since
2026-09-06, all auto-closed as not_planned without a durable fix). As a result, per-agent quality/
effectiveness scoring below relies on live issue/PR searches rather than the snapshot, and full
per-workflow score tables are deferred again this run pending a real fix to Metrics Collector's own
model config (see recommendation #1 — this is self-referential: fixing the model config fixes both
the affected workflows and restores fresh metrics for future runs).

Per shared-alerts.md history, this stale-metrics condition has now persisted across at least 3
consecutive Agent Performance Analyzer runs (2026-09-09, 2026-09-10, this run). Escalating priority.

Systemic Issue: codex engine + gpt-5.3-codex model_not_supported_error (P0)

Evidence:

• 30 "produced no safe outputs" / failure issues found via search, 2026-08-05 through 2026-09-12.
• Of the 25 most recent (since 2026-09-06), 19 carry the model_not_supported_error failure
  category in their gh-aw-failure-issue metadata, spanning distinct workflows: Metrics Collector
  (×6: [aw] Metrics Collector produced no safe outputs #58895, [aw] Metrics Collector produced no safe outputs #59105, [aw] Metrics Collector produced no safe outputs #59344, [aw] Metrics Collector produced no safe outputs #59611, [aw] Metrics Collector produced no safe outputs #59851, [aw] Metrics Collector produced no safe outputs #60157), Daily Evals Feature Report (×3), Daily CLI
  Performance Agent (×3), Daily Cache Strategy Analyzer (×2), Daily Regulatory Report Generator
  (×2), Daily Documentation Diagram, Daily Documentation Updater, PureLock, Auto-Triage Issues —
  4 of these are still open right now ([aw] Daily Documentation Diagram produced no safe outputs #60373, [aw] Daily Documentation Updater produced no safe outputs #60392, [aw] PureLock produced no safe outputs #60405, [aw] Auto-Triage Issues produced no safe outputs #60406, all created today,
  2026-09-12).
• All affected issues share the identical diagnostic text: "Invalid or Unsupported Model...This is
  a configuration issue, not a transient error — retrying will not help" combined with "No Safe
  Outputs Generated."
• Repo-wide grep of .github/workflows/*.md frontmatter finds 69 workflows configured with
  engine: {id: codex} and model: {copilot,openai}/gpt-5.3-codex (or bare gpt-5.3-codex).
  gpt-5.3-codex is listed as a valid model in pkg/cli/data/models.json for both the
  openai and github-copilot providers — so the model name string itself is not invalid; the
  failure is consistent with either (a) Copilot/OpenAI policy disabling gpt-5.3-codex for the
  codex engine specifically (as distinct from the same model string working under other engines),
  or (b) an engine/provider resolution mismatch for this specific model+engine combination that
  ResolveLLMProvider/codexModelID in pkg/workflow/codex_engine.go does not currently guard
  against.
• Correction to prior shared-alerts framing: the 2026-09-10/11 notes reclassified this as
  "transient model-availability/policy flakiness, not a hard config defect" based on
  daily-go-test-parallelizer/lint-monster showing mostly-successful recent runs. That reclassification
  does not hold up against the broader 2026-09-06→09-12 dataset: the failure recurs daily, across
  a growing and rotating set of ≥12 distinct workflows, with 4 new occurrences in the last 24 hours
  alone. This is a systemic, ongoing pattern, not an isolated flake on two workflows.

Recommendation (High Priority):

1. File one consolidated tracking issue (not per-workflow) for the codex + gpt-5.3-codex
model_not_supported_error pattern, scoped to engineering: verify whether gpt-5.3-codex is
   actually enabled under the codex engine's provider resolution path for this org/repo, and
   either (a) get the model enabled via Copilot/OpenAI policy, or (b) fall back affected workflows
   to a known-working model (e.g. gpt-5.2-codex, which is also listed for both providers) until
   policy is fixed.
   • Estimated effort: 1-2 hours to identify a working fallback model + apply via gh aw update
     across affected workflow frontmatters; policy-side enablement is out of engineering's direct
     control and should be requested separately.
   • Expected impact: eliminates the dominant recurring failure category (19 of last 25 failure
     issues), restores Metrics Collector's daily snapshot (unblocking all three meta-orchestrators'
     scoring), and stops ~1 new "no safe outputs" issue per day from being auto-filed and
     auto-expiring without resolution.

Behavioral Patterns

• under-creation / recurring-failure pattern: Metrics Collector, Daily Evals Feature Report,
  Daily CLI Performance Agent, Daily Cache Strategy Analyzer, Daily Regulatory Report Generator —
  all show repeated model_not_supported_error → zero safe outputs → auto-expiring issue cycles.
  Root cause is shared engine/model config, not individual prompt quality — do not attribute this to
  prompt deficiencies in these workflows.
• inconsistency: LintMonster and daily-go-test-parallelizer show alternating pass/fail streaks
  attributed in shared-alerts to "transient model-availability/policy flakiness" — consistent with
  the same underlying gpt-5.3-codex/codex policy issue rather than a workflow-specific defect.
• Command-gating (expected, not a defect): q and squad show 100% action_required/skipped
  in the stale snapshot; per repo convention these are slash_command/mention-triggered workflows
  where every issue/comment event starts a run that the pre_activation gate then stops unless the
  body starts with the command. Not scored as failures.

Recommendations

High Priority

1. Fix codex + gpt-5.3-codex model_not_supported_error (see systemic issue above). Affects
   ≥12 distinct workflows and directly blocks fresh metrics for all meta-orchestrators.

Medium Priority

2. Add a metrics-collector safeguard: since Metrics Collector is itself affected by the same
   bug, add a fallback model or report-failure-as-issue: false + alerting via a different engine
   for its own frontmatter so a single upstream policy issue doesn't also blind the health/quality
   pipeline for 11+ days.
3. Root-cause hygiene: shared-alerts.md's 2026-09-10/11 "transient flakiness, not a config
   defect" framing for lint-monster/daily-go-test-parallelizer should be updated — the broader
   dataset shows this is an ongoing, expanding pattern, not isolated to those two workflows.

Low Priority

4. Once Metrics Collector is unblocked, resume full per-workflow quality/effectiveness scoring
   (deferred this run for lack of fresh data, same as 2026-09-09 and 2026-09-10 runs).

Trends

• Consecutive stale-metrics runs: 3 (2026-09-09, 2026-09-10, 2026-09-12) — escalating.
• New failure-issue rate: ~1/day since 2026-09-06, rotating across a growing set of workflows, vs.
  isolated single-workflow incidents before that date.

Actions Taken This Run

• Cross-verified 30 "no safe outputs"/failure issues live via GitHub search (not relying on stale
  snapshot).
• Confirmed 69 workflows share the codex + gpt-5.3-codex config; identified this as the shared
  root cause behind 19 of the last 25 failure issues.
• Corrected the 2026-09-10/11 "transient flakiness" framing in light of the fuller dataset.
• Filed one consolidated improvement issue for the systemic model config problem (see below).

Next Steps

1. Resolve or escalate the codex/gpt-5.3-codex policy/config issue.
2. Re-run full agent scoring once Metrics Collector produces a fresh snapshot.
3. Re-verify lint-monster/daily-go-test-parallelizer classification after the model fix lands.

---

Analysis period: 2026-08-05 to 2026-09-12
Next report: 2026-09-19 (weekly cadence)

Generated by ⚡ Agent Performance Analyzer - Meta-Orchestrator · copilot · auto · 93.5 AIC · ⌖ 8.22 AIC · ⊞ 18.2K · ◷

• expires on Sep 13, 2026, 4:55 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-09-13T12:55:10.415Z.

Closed by Workflow