[prompt-clustering] Prompt Clustering Analysis - 2026-10-15

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-09-15 to 2026-10-15)
Total Tasks Analyzed: 594
Clusters Identified: 7
Overall Success Rate: 81.7% (485 merged / 594 total)

Full Analysis Report

General Insights

• Most Common Task Type: Cluster "Workflow fixes, coverage & general validation" (221 tasks, 37.2% of all PRs), merge rate 81.5% — a catch-all bucket of general CI/validation/regression-coverage fixes that tracks the overall rate almost exactly.
• Second Most Common: Cluster "Agent job failures & AIC accounting" (100 tasks, 16.8%), merge rate 76.0% — the lowest merge rate among clusters large enough to be reliable, covering AWF startup-failure surfacing, not_started execution evidence, and AIC accounting edge cases.
• Highest Success Rate: Cluster "Firewall/network domain handling & logs stats" (93.3% merged, 45 tasks). Narrowly-scoped, testable firewall/domain and logs-stats changes continue to converge fastest.
• Second Highest: Cluster "PR Sous Chef & agent run / engine staging fixes" (87.5% merged, 72 tasks) — also the heaviest-engagement cluster (14.9 avg comments, 7.0 avg reviews) yet still merges well above average.
• Most Complex Tasks: Cluster "gh-aw CLI version pins & dependency updates" at 113.0 avg files changed per PR — pin bumps regenerate many lockfiles/manifests at once, by far the largest footprint this run.
• Lowest Success Rate: Cluster "gh-aw CLI version pins & dependency updates" (75.5% merged, 49 tasks), closely followed by "Agent job failures & AIC accounting" (76.0%, 100 tasks).
• Outliers: None this run. No cluster met the outlier threshold (≥15 PRs and ≥10-point merge-rate gap below the 81.7% overall rate). The nearest miss is "gh-aw CLI version pins & dependency updates" at a 6.14-point gap — see the note below.
• Data coverage: Full PR detail (comments, reviews, file counts, review decision) was available for all 594 PRs in the 30-day window (100% coverage) via the cached pr-full-data store, which holds 685 PRs total (a superset spanning beyond the current window). Workflow run-log metrics (turns/duration/cost) were not joined this run — no gh-aw/agenticworkflows log-download tool was reachable in this sandbox for the GitHub-native Copilot coding agent's PRs, so findings rely on PR metadata, comments, and reviews only (same limitation as prior runs).

Cluster Analysis

Workflow fixes, coverage & general validation

• Size: 221 tasks (37.2% of total)
• Success Rate: 81.5%
• Avg Files Changed: 32.1 (comments: 5.2, reviews: 3.2)
• Top Keywords: workflow, coverage, fixes, repository, add, validation, regression, regression coverage
• Characteristics: The largest and most heterogeneous cluster this run — general workflow correctness, coverage/regression fixes, and repository-level validation work. Its size and breadth make it behave like a baseline bucket; merge rate sits almost exactly on the overall average, so it isn't dragging the headline number down or up.
• Example PRs: Use default sandbox runtime for Daily Spec Coverage Review #61200, Allow local JavaScript references in packaged HTML #61205, Extend forecast with workflow-run API costs and JSONL history #61230, Stabilize Daily Documentation Healer runtime #61235, Include native Copilot tool calls in the automatic grader trace payload #61426

---

Agent job failures & AIC accounting

• Size: 100 tasks (16.8% of total)
• Success Rate: 76.0%
• Avg Files Changed: 59.8 (comments: 5.3, reviews: 4.1)
• Top Keywords: job, failure, agent, step, usage, issue, run, actions
• Characteristics: Agent-job failure surfacing, startup-error reporting, and AIC (agentic interaction cost) accounting fixes. Second-lowest merge rate this run with above-average review depth relative to its size — this failure-handling surface area appears to need more iteration than most.
• Example PRs: Surface AWF fatal startup errors in agent failure reports #61199, Record not_started execution evidence when AWF fails before the engine harness #61202, Handle agent failures before runner startup in AIC accounting #61206, Handle unassigned jobs in daily AIC accounting #61222, Treat AWF API proxy guardrail 403s as terminal instead of authentication failures #62487

---

Model routing (Copilot/Claude/Codex/Gemini)

• Size: 75 tasks (12.6% of total)
• Success Rate: 80.0%
• Avg Files Changed: 53.3 (comments: 6.5, reviews: 3.7)
• Top Keywords: model, routing, copilot, models, claude, codex, routed, endpoint
• Characteristics: Model-identifier/endpoint routing, pricing/alias fixes, and engine-config preservation across Copilot/Claude/Codex/Gemini. Tracks close to the overall rate.
• Example PRs: Add gemini-3.8-flash alias, fix gpt-6-astra/gpt-5.6-sol pricing, add claude-fable-5.1 #61234, Preserve imported engine config (including auth) when a workflow sets top-level model #61424, Migrate Daily Reliability Review off the hanging opencode engine #62490, Fix Daily Go Test Parallelizer terminal outputs #62494, Add Copilot pricing metadata for grok-4.7 #62512

---

PR Sous Chef & agent run / engine staging fixes

• Size: 72 tasks (12.1% of total)
• Success Rate: 87.5%
• Avg Files Changed: 46.5 (comments: 14.9, reviews: 7.0)
• Top Keywords: em, sub, aic, run, pr sous, chef, sous, sous chef
• Characteristics: Highest comment/review activity of any cluster by a wide margin (engine staging fixes, Codex/Copilot CLI heredoc and config fixes, skill-preservation during base-branch restore), yet still converges well above the overall rate — the back-and-forth resolves rather than stalling.
• Example PRs: Prevent Copilot CLI staging for Gemini ARC/DinD workflows #62415, Preserve step-installed skills during base-branch restore; allow Claude's Skill tool #62419, Fix Codex engine.config heredoc indentation #62420, Enforce configured targets in safe-output handlers #62447, Preserve POSIX separators in token-usage artifact paths #62484

---

gh-aw CLI version pins & dependency updates

• Size: 49 tasks (8.2% of total)
• Success Rate: 75.5%
• Avg Files Changed: 113.0 (comments: 6.1, reviews: 2.7)
• Top Keywords: version, aw, gh aw, gh, v0, pin, cli, update
• Characteristics: Lowest merge rate this run despite the lowest review engagement and by far the largest average diff footprint (pin bumps regenerate many lockfiles/manifests at once). Diff size is not the driver of risk here — see the root-cause note below.
• Example PRs: [ci-coach] Add go ecosystem to network allowlist #61201, Use ${{ runner.temp }} for PLAYWRIGHT_BROWSERS_PATH so install and launch agree on the browser path #61423, Embed reviewed threat-detect digests in compiled workflows #61857, Pin Copilot CLI default back to 1.0.83 to fix instant HTTP 400 regression in 1.0.85 #62421, Pin Copilot CLI to 1.0.80 to avoid MCP regressions #62482

---

Firewall/network domain handling & logs stats

• Size: 45 tasks (7.6% of total)
• Success Rate: 93.3%
• Avg Files Changed: 69.4 (comments: 10.6, reviews: 5.2)
• Top Keywords: blocked, network, domain, firewall, details, summary
• Characteristics: Highest merge rate this run — firewall/blocked-domain handling, log download-duration stats, and multi-target log-query distribution. Narrowly-scoped, verifiable changes continue to converge reliably despite active review discussion.
• Example PRs: logs: track per-run download duration/size and render end-of-run stats summary #60951, Remove legacy Effective Tokens metric #60997, Distribute multi-target logs queries fairly #61027, Validate cached run uniqueness across JSONL shards #61220, Handle legacy pre-harness failures in daily AIC accounting #61232

---

Work-queue protocol & issue dispatch

• Size: 32 tasks (5.4% of total)
• Success Rate: 84.4%
• Avg Files Changed: 30.8 (comments: 8.1, reviews: 5.0)
• Top Keywords: queue, work, claim, issue, git, policy, pr, branch
• Characteristics: Work-pool ledger, deterministic reducers, and work-queue dispatch/claim-policy changes. Above-average merge rate with above-average review depth — newer surface area that still converges well.
• Example PRs: Add deterministic work-pool ledger reducer #64787, Remove work-pool ledger support #64850, Dispatch Work Coordinator — implementation pending #64853, Add Git-backed work queue operator commands #65062, Rename work coordinator tool and CLI command to work-queue #65097

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Files Changed	Top Keywords
6	Workflow fixes, coverage & general validation	221	81.5%	32.1	workflow, coverage, fixes, validation
4	Agent job failures & AIC accounting	100	76.0%	59.8	job, failure, agent, step, usage
2	Model routing (Copilot/Claude/Codex/Gemini)	75	80.0%	53.3	model, routing, copilot, endpoint
1	PR Sous Chef & agent run/engine staging fixes	72	87.5%	46.5	sous chef, chef, pr sous, run
0	gh-aw CLI version pins & dependency updates	49	75.5%	113.0	version, gh aw, pin, cli, update
5	Firewall/network domain handling & logs stats	45	93.3%	69.4	blocked, firewall, network, domain
3	Work-queue protocol & issue dispatch	32	84.4%	30.8	queue, work, claim, dispatch, policy

Nearest-Miss Cluster Note (No Outlier This Run)

• Cluster: "gh-aw CLI version pins & dependency updates" (49 tasks)
• Merge Rate Gap: 75.5% vs 81.7% overall (gap: 6.14 points) — below the 10-point outlier threshold, so no formal root-cause investigation was triggered, but a light-touch check was run anyway given it's the lowest-merging cluster.
• Light-touch review: Of the 12 non-merged PRs in this cluster, 7 (58%) closed or sat open with zero comments and zero reviews (e.g. "Pin Copilot CLI to 1.0.80 to avoid MCP regressions", "Harden installer downloads and security ownership paths", "Update GitHub MCP, Playwright, Syft, and Grype default pins") — consistent with superseded/abandoned pin-bump attempts rather than review friction. Of the remainder with real engagement, Refresh available container image versions and pins #66122 ("Refresh available container image versions and pins") was closed after an automated review flagged "AWF routing integration is incomplete, Grafana telemetry remains enabled, and the required clean rescan is pending" — genuine scope/completeness friction — while Emit AWF agentTimeout for literal timeouts and show delivered steering notices in audit/logs #67597 and Export sparse model-routing attributes to OpenTelemetry #67240 (both still open) show heavy, unresolved review back-and-forth on AWF timeout/telemetry correctness.
• Takeaway: The gap is driven by a mix of superseded/duplicate pin-bump PRs (likely replaced by a newer pin bump before review) and a smaller tail of genuinely incomplete scope — not a single systemic blocker, and not large diff size (this cluster has the largest average footprint of any cluster yet still converges most of the time).

Key Findings

1. No outlier cluster this run — cluster merge rates range from 75.5% to 93.3%, a band broadly consistent with recent prior runs; task routing/sizing across themes remains reasonably stable.
2. Narrowly-scoped, testable work keeps winning — firewall/network domain handling (93.3%) and PR Sous Chef/engine-staging fixes (87.5%) are the top two clusters, continuing the pattern from prior reports that tightly-bounded, verifiable changes converge fastest even with heavy review engagement.
3. Diff size does not predict failure — the largest-footprint cluster (CLI version pins, 113.0 avg files) has the lowest merge rate, while the firewall/network cluster (69.4 avg files, second-largest footprint) has the highest — size alone isn't the driver; what differs is whether PRs get superseded/abandoned mid-review versus iterated to a close.
4. Agent job-failure handling is an iteration-heavy surface — the second-largest cluster (100 tasks, agent job failures & AIC accounting) sits 5.7 points below the overall rate with above-average review depth relative to its size, suggesting this failure-surfacing code path is harder to land cleanly on the first attempt than most.

Recommendations

1. Reduce duplicate/superseded pin-bump PRs — nearly 60% of the lowest-merging cluster's non-merged PRs had zero review engagement, consistent with being replaced by a newer pin bump before review starts; consider de-duplicating or batching these bump tasks so only one is in flight per dependency at a time.
2. Tighten the "agent job failures & AIC accounting" scope — this cluster's below-average merge rate with above-average review depth suggests tasks here may benefit from narrower, single-concern framing (e.g. one failure-surfacing change per task) similar to the firewall/network cluster's pattern.
3. Keep leaning into narrow, testable scopes — the two highest-merging clusters (firewall/network, PR Sous Chef) both reflect well-tested, verifiable diffs; prompt templates for the lower-merging clusters could borrow explicit acceptance-test framing.
4. Re-attempt workflow-log enrichment — turn counts, duration, and cost (via gh-aw/agenticworkflows logs) were unavailable again this run; if a reachable log-download path exists for future runs, joining it in would let future analyses correlate agent iteration counts with merge outcomes, not just PR-level engagement.

---

Generated by Prompt Clustering Analysis (Run: 38715587753)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 177.6 AIC · ⊞ 6.9K · ◷

• expires on Oct 16, 2026, 2:07 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #69310.