[prompt-clustering] Prompt Clustering Analysis - 2026-10-17

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-09-17 to 2026-10-17)
Total Tasks Analyzed: 559
Clusters Identified: 10
Overall Success Rate: 84.4% (472 merged / 82 closed / 5 open)

The merge rate has climbed from a low of ~75% in early October back up to 84%, after dipping to 81-82% mid-month.

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 4 — Workflow & Agent Run Accounting / Failure Handling (186 tasks, 33% of total)
• Highest Success Rate: Cluster 1 — Firewall & Network Blocklist Handling (94.8% merged)
• Most Complex Tasks: Cluster 6 — Engine Version Bumps & CLI Compatibility (85.2 files changed on average) and Cluster 1 (76.7 avg files changed, 11.4 avg comments)
• Outliers: 0 clusters flagged this period (no cluster both ≥15 PRs and ≥10 points below the overall merge rate)
• Workflow metrics: gh-aw logs / aw_info.json turn and cost data were not available in this run's sandbox (no gh-aw CLI or workflow-logs directory present); this analysis relies on PR-level signals only (comments, reviews, files changed)

Cluster Analysis

Cluster 1: Workflow & Agent Run Accounting / Failure Handling

• Size: 186 tasks (33.3% of total)
• Success Rate: 79.6%
• Avg Comments / Reviews / Files Changed: 5.7 / 3.4 / 55.7
• Top Keywords: workflow, agent, usage, failure, run, accounting, update, issue
• Characteristics: The largest and most heterogeneous cluster — usage/cost accounting, run discovery caching, failure handling across the agentic-workflow runtime.
• Example PRs: Add frontmatter formatting command #59971, Cache workflow run discovery payloads in logs JSONL #59988, Disable forcePublicRepos for enclave-only GitHub MCP backends #60016, Cache audit results in logs JSONL #60183, Prevent Jira smoke workflow failures when credentials are unavailable #60199

---

Cluster 2: Repository Checkout & Regression Test Coverage

• Size: 104 tasks (18.6% of total)
• Success Rate: 84.6%
• Avg Comments / Reviews / Files Changed: 5.7 / 3.2 / 38.5
• Top Keywords: coverage, repository, checkout, regression, setup, github
• Characteristics: Test/regression-coverage additions and checkout-credential correctness fixes — close to the overall baseline on every metric.
• Example PRs: Fix local cross-repository workflow resolution in gh aw logs #59859, Reject NUL bytes in Git paths and consolidate validation tests #60043, Fix Dependabot configuration syntax #60122, Require immutable gh-aw action-mode references #60227, Remove cloud-hypervisor security review warning #60304

---

Cluster 3: PR Sous Chef & AIC Run Automation

• Size: 67 tasks (12.0% of total)
• Success Rate: 88.1%
• Avg Comments / Reviews / Files Changed: 15.5 / 7.3 / 48.4
• Top Keywords: em, sub, aic, run, pr, sous chef
• Characteristics: Highest review/comment intensity of any cluster — iterative bot-assist fixes (PR Sous Chef, AIC daily accounting) that draw heavy back-and-forth before merge.
• Example PRs: Prevent Copilot CLI staging for Gemini ARC/DinD workflows #62415, Preserve step-installed skills during base-branch restore; allow Claude's Skill tool #62419, Fix Codex engine.config heredoc indentation #62420, Preserve POSIX separators in token-usage artifact paths #62484, Fix pull request creation from same-organization forks #62720

---

Cluster 4: Firewall & Network Blocklist Handling

• Size: 58 tasks (10.4% of total)
• Success Rate: 94.8% (highest of all clusters)
• Avg Comments / Reviews / Files Changed: 11.4 / 5.3 / 76.7
• Top Keywords: blocked, network, firewall, details, summary, domain
• Characteristics: Egress-firewall and domain-allowlist fixes — large diffs and heavy review, but converts to merges reliably once reviewed.
• Example PRs: Fix static GitHub agent enclave startup #59878, Preserve frontmatter formatting during workflow upgrades #59969, Load cached logs JSONL once per command #60040, Upload activation run info as a GHE-compatible artifact #60057, Restore Copilot usage accounting without the agent sandbox #60134

---

Cluster 5: Model Routing & Engine Provider Selection

• Size: 42 tasks (7.5% of total)
• Success Rate: 78.6% (lowest of all clusters, but below the ≥15-PR-and-10pt outlier bar is not met — gap is only 5.9 points)
• Avg Comments / Reviews / Files Changed: 7.0 / 4.0 / 72.6
• Top Keywords: model, routing, copilot, codex, engine, endpoint, provider, selection
• Characteristics: Engine/model-selection logic (Grok, Codex, Pi) — frequent churn as providers change APIs; watch this cluster, it is the closest thing to an outlier this period.
• Example PRs: Route Grok 4.5 and 4.6 through the Responses API #60139, Fix PR Sous Chef threat-detection model #60321, Add reasoning effort support to assign-to-agent #60346, Fix Issue Monster model compatibility with Pi #60561, Fix Issue Monster Pi model selection #60661

---

Cluster 6: Engine Version Bumps & CLI Compatibility

• Size: 29 tasks (5.2% of total)
• Success Rate: 86.2%
• Avg Comments / Reviews / Files Changed: 4.6 / 2.4 / 85.2 (largest average diffs of any cluster)
• Top Keywords: copilot, codex, web, version, cli, workflow, claude
• Characteristics: Dependency/version bumps and CLI-identifier fixes — large mechanical diffs, low review friction.
• Example PRs: Bump gh-aw-firewall to v0.28.15 #60020, fix(codex): strip openai/ provider prefix from Codex model identifiers #60423, Fix Issue Monster’s unsupported Pi model #60461, Improve Copilot organization billing failure guidance #60877, Use default sandbox runtime for Daily Spec Coverage Review #61200

---

Cluster 7: Git-Backed Work Queue & Branch Policy

• Size: 23 tasks (4.1% of total)
• Success Rate: 82.6%
• Avg Comments / Reviews / Files Changed: 8.4 / 4.8 / 31.4
• Top Keywords: queue, work, git, policy, pr, issue, branch, changes
• Characteristics: New "work queue" / "WorkQ" subsystem — actively under construction, several renames/pivots mid-cluster (e.g. "work coordinator" → "work-queue").
• Example PRs: Cancel multi-target log downloads at the shared count limit #60343, Dispatch Work Coordinator — implementation pending #64853, Add Git-backed work queue operator commands #65062, Rename work coordinator tool and CLI command to work-queue #65097, Include WorkQ claims in the admission prompt #65192

---

Cluster 8: Repo-Memory Ledger & Safe-Output Records

• Size: 19 tasks (3.4% of total)
• Success Rate: 89.5%
• Avg Comments / Reviews / Files Changed: 10.1 / 5.1 / 48.7
• Top Keywords: ledger, memory, repo, records, push, output, safe, job
• Characteristics: Repo-memory backend and safe-output temporary-ID plumbing — newer subsystem, high engagement but converts well.
• Example PRs: Apply safe-output temporary IDs to repo memory #60789, Add repo-memory backend for daily AIC guardrail #62958, [WIP] Add distributed ledger capability to repo-memory #64199, Move daily workflow review history to repo-memory ledgers #64337, Preserve default ledgers when importing shared workflows #64464

---

Cluster 9: Multi-Target Log Download Limits

• Size: 18 tasks (3.2% of total)
• Success Rate: 94.4% (second highest)
• Avg Comments / Reviews / Files Changed: 7.1 / 3.5 / 10.4 (smallest average diffs)
• Top Keywords: target, targets, context, count, shared, issue, limit, verify
• Characteristics: Tight, well-scoped fixes to gh aw logs concurrency/count/timeout handling — small diffs, fast consensus.
• Example PRs: Preserve partial logs before command timeouts #59964, Apply logs timeout and count globally across concurrent targets #60058, Stop starting log targets after reaching the shared count limit #60266, Stop queued log targets after timeout #60273, Fix --count/--timeout wiring for multi-target logs downloads #60308

---

Cluster 10: CI Job Failure Root-Cause Fixes

• Size: 13 tasks (2.3% of total)
• Success Rate: 84.6%
• Avg Comments / Reviews / Files Changed: 1.9 / 1.2 / 1.9 (lowest engagement and smallest diffs of any cluster)
• Top Keywords: fix, actions, job, cause, failing, root, work, logs
• Characteristics: Auto-dispatched "[WIP] Fix failing GitHub Actions job" tasks — single-purpose, low-touch, minimal review needed. 18 of the 559 PRs analyzed carry a [WIP] title prefix, and most of them land in this cluster.
• Example PRs: [WIP] Fix failing GitHub Actions job 'build' #60289, [WIP] Fix failing GitHub Actions job lint-go-golangci #60316, [WIP] Fix failing GitHub Actions job #60645, [WIP] Fix failing GitHub Actions job Integration: CMD Tests #60683, [WIP] Fix failing GitHub Actions job for CLI update command #60849

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Files Changed	Top Keywords
1	Workflow & Agent Run Accounting	186	79.6%	55.7	workflow, agent, usage, failure
2	Checkout & Regression Coverage	104	84.6%	38.5	coverage, repository, checkout
3	PR Sous Chef & AIC Automation	67	88.1%	48.4	sous chef, aic, run
4	Firewall & Network Blocklist	58	94.8%	76.7	blocked, firewall, domain
5	Model Routing & Engine Selection	42	78.6%	72.6	model, routing, codex, provider
6	Engine Version Bumps & CLI	29	86.2%	85.2	copilot, codex, version, cli
7	Git-Backed Work Queue & Policy	23	82.6%	31.4	queue, work, git, policy
8	Repo-Memory Ledger & Safe-Output	19	89.5%	48.7	ledger, memory, repo, records
9	Multi-Target Log Download Limits	18	94.4%	10.4	target, count, shared, limit
10	CI Job Failure Root-Cause Fixes	13	84.6%	1.9	fix, actions, job, failing

Representative PR Examples by Cluster (5 per cluster)

PR #	Title	Cluster	Outcome
#59971	Add frontmatter formatting command	1	Merged
#59988	Cache workflow run discovery payloads in logs JSONL	1	Merged
#60016	Disable forcePublicRepos for enclave-only GitHub MCP backends	1	Merged
#60183	Cache audit results in logs JSONL	1	Merged
#60199	Prevent Jira smoke workflow failures when credentials are unavailable	1	Closed
#59859	Fix local cross-repository workflow resolution in gh aw logs	2	Merged
#60043	Reject NUL bytes in Git paths and consolidate validation tests	2	Merged
#60122	Fix Dependabot configuration syntax	2	Merged
#60227	Require immutable gh-aw action-mode references	2	Closed
#60304	Remove cloud-hypervisor security review warning	2	Merged
#62415	Prevent Copilot CLI staging for Gemini ARC/DinD workflows	3	Merged
#62419	Preserve step-installed skills during base-branch restore	3	Merged
#62420	Fix Codex engine.config heredoc indentation	3	Merged
#62484	Preserve POSIX separators in token-usage artifact paths	3	Closed
#62720	Fix pull request creation from same-organization forks	3	Merged
#59878	Fix static GitHub agent enclave startup	4	Merged
#59969	Preserve frontmatter formatting during workflow upgrades	4	Merged
#60040	Load cached logs JSONL once per command	4	Merged
#60057	Upload activation run info as a GHE-compatible artifact	4	Merged
#60134	Restore Copilot usage accounting without the agent sandbox	4	Merged
#60139	Route Grok 4.5 and 4.6 through the Responses API	5	Merged
#60321	Fix PR Sous Chef threat-detection model	5	Closed
#60346	Add reasoning effort support to assign-to-agent	5	Merged
#60561	Fix Issue Monster model compatibility with Pi	5	Merged
#60661	Fix Issue Monster Pi model selection	5	Merged
#60020	Bump gh-aw-firewall to v0.28.15	6	Merged
#60423	fix(codex): strip openai/ provider prefix from Codex model identifiers	6	Merged
#60461	Fix Issue Monster's unsupported Pi model	6	Merged
#60877	Improve Copilot organization billing failure guidance	6	Merged
#61200	Use default sandbox runtime for Daily Spec Coverage Review	6	Closed
#60343	Cancel multi-target log downloads at the shared count limit	7	Merged
#64853	Dispatch Work Coordinator — implementation pending	7	Closed
#65062	Add Git-backed work queue operator commands	7	Merged
#65097	Rename work coordinator tool and CLI command to work-queue	7	Merged
#65192	Include WorkQ claims in the admission prompt	7	Merged
#60789	Apply safe-output temporary IDs to repo memory	8	Merged
#62958	Add repo-memory backend for daily AIC guardrail	8	Merged
#64199	[WIP] Add distributed ledger capability to repo-memory	8	Merged
#64337	Move daily workflow review history to repo-memory ledgers	8	Merged
#64464	Preserve default ledgers when importing shared workflows	8	Merged
#59964	Preserve partial logs before command timeouts	9	Merged
#60058	Apply logs timeout and count globally across concurrent targets	9	Merged
#60266	Stop starting log targets after reaching the shared count limit	9	Merged
#60273	Stop queued log targets after timeout	9	Merged
#60308	Fix --count/--timeout wiring for multi-target logs downloads	9	Merged
#60289	[WIP] Fix failing GitHub Actions job 'build'	10	Merged
#60316	[WIP] Fix failing GitHub Actions job lint-go-golangci	10	Merged
#60645	[WIP] Fix failing GitHub Actions job	10	Closed
#60683	[WIP] Fix failing GitHub Actions job Integration: CMD Tests	10	Merged
#60849	[WIP] Fix failing GitHub Actions job for CLI update command	10	Closed

Full dataset: 559 PRs across 10 clusters; the table above shows 5 representative examples per cluster rather than the complete set, for readability.

Key Findings

1. No outlier cluster this period — every cluster's merge rate is within 6 points of the 84.4% overall baseline, except two clusters that exceed it by 10+ points (Firewall/Network at 94.8%, Multi-Target Log Limits at 94.4%). The codebase's automated task pipeline is currently well-calibrated across task types.
2. Scope drives friction, not failure — the two clusters with the largest average diffs (Engine Version Bumps at 85.2 files, Firewall Handling at 76.7 files) still merge at 86%+ and 95% respectively. Diff size alone doesn't predict rejection here; review engagement (comments/reviews) tracks with diff size but converts to merges.
3. Model-routing churn is the closest thing to a soft spot — Cluster 5 (Model Routing & Engine Provider Selection, 42 PRs, 78.6% merged) sits furthest below baseline of any cluster with 15+ PRs, driven by fast-moving provider API changes (Grok, Codex, Pi) that produce short-lived or superseded PRs.
4. [WIP] Fix failing GitHub Actions job is a distinct, low-touch task template — 18 of 559 PRs (3.2%) carry this auto-generated title pattern, clustering almost entirely into Cluster 10 with the lowest average comments/reviews/files-changed of any cluster — these read as single-purpose CI-triage dispatches rather than open-ended feature work.
5. Merge rate recovered mid-month — the 30-day rolling trend dropped to ~75% in the first days of October before recovering to 84% by today, a 9-point swing (see trend chart above).

Lowest-Merge Cluster Root Cause

• Cluster: 5 — Model Routing & Engine Provider Selection (not a qualifying outlier)
• Merge Rate Gap: 78.6% vs 84.4% overall (gap: 5.9 points)
• Why no formal root-cause investigation: the outlier threshold requires both ≥15 PRs and a ≥10-point gap below the overall merge rate. No cluster met both conditions this period (largest gap was 5.9 points, cluster 5), so Phase 4.5 blocker classification was not triggered this run.
• Watch item: Cluster 5's closed PRs (e.g. Fix PR Sous Chef threat-detection model #60321 "Fix PR Sous Chef threat-detection model") cluster around provider/model identifier churn — a pattern worth re-checking if its gap widens past 10 points in a future run.

Recommendations

1. Keep firewall/network and log-limit task framing as-is — Clusters 4 and 9 (94.8% and 94.4% merge rates) show that narrowly-scoped, single-concern prompts for egress/log-concurrency work convert reliably; use their prompt structure as a template for other infrastructure fixes.
2. Tighten model-routing task scoping — Cluster 5's below-baseline rate suggests prompts referencing fast-moving provider/model identifiers (Grok, Codex, Pi) should pin the specific API version or endpoint shape in the task description to reduce rework from upstream provider changes.
3. Keep the [WIP] Fix failing GitHub Actions job dispatch pattern — Cluster 10's low engagement and solid 84.6% merge rate indicate this auto-triage template is working; no changes needed.
4. Re-enable workflow-log enrichment — this run had no aw_info.json / turn-count data available, so recommendations above are based on PR interaction signals only, not agent iteration counts. Future runs should verify gh-aw logs output is present before Phase 2 to add turn/cost correlation.

---

Generated by Prompt Clustering Analysis (Run: 39022222178)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 99.6 AIC · ⌖ 0.928 AIC · ⊞ 6.9K · ◷

• expires on Oct 18, 2026, 2:02 AM UTC-08:00