[prompt-clustering] Prompt Clustering Analysis - 2026-10-16

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-09-16 to 2026-10-16)
Total Tasks Analyzed: 592
Clusters Identified: 10
Overall Success Rate: 81.4% (482 merged / 592 total)

Full Analysis Report

General Insights

• Most Common Task Type: "Workflow fixes, coverage & general validation" (123 tasks, 20.8% of all PRs), merge rate 83.7% — a broad bucket of CI/regression-coverage/checkout fixes tracking close to the overall rate.
• Second Most Common: "Agent job failures, safe-outputs & artifacts" (116 tasks, 19.6%), merge rate 75.9% — one of the two lowest-merging large clusters, covering safe-output handling, job-failure surfacing, and artifact plumbing.
• Highest Success Rate: "Firewall/network domain handling & AIC accounting" (92.7% merged, 41 tasks) — narrowly-scoped, testable domain/firewall and AIC-accounting changes converge fastest, same pattern as prior runs.
• Most Complex Tasks: "gh-aw CLI version pins & dependency updates" at 170.5 avg files changed per PR — pin bumps regenerate many lockfiles/manifests at once, by far the largest footprint this run, yet still merges at an above-baseline 83.3%.
• Outlier Found: "Threat-detection engine internals (cross-engine)" — 18 tasks, 50.0% merge rate, a 31.4-point gap below the 81.4% overall rate. This is the first outlier cluster flagged in recent runs (prior runs topped out at a 6-point gap) and triggered the root-cause investigation below.
• Data coverage: Full PR detail (comments, reviews, file counts, review decision) was available for all 592 PRs in the 30-day window (100% coverage). Workflow run-log metrics (turns/duration/cost) were not joined this run — no reachable gh-aw/agenticworkflows log-download path exists for the GitHub-native Copilot coding agent's PRs in this sandbox, so findings rely on PR metadata, comments, and reviews only (same limitation as prior runs).

Cluster Analysis

Workflow fixes, coverage & general validation

• Size: 123 tasks (20.8% of total)
• Success Rate: 83.7%
• Avg Files Changed: 38.8 (comments: 5.1, reviews: 3.1)
• Top Keywords: repository, coverage, workflow, checkout, regression, regression coverage, file, add
• Characteristics: The largest, most heterogeneous cluster — general workflow correctness, coverage/regression fixes, and repo-level validation. Behaves as a baseline bucket; merge rate sits close to the overall average.
• Example PRs: Stabilize Daily Documentation Healer runtime #61235, Preserve imported engine auth with root engine preferences #62413, Add Dependency Analysis gallery page #62418, Preserve imported report-failed-jobs safe-output settings #62485, Remove failing shared agentic logs cache restore #62492

---

Agent job failures, safe-outputs & artifacts

• Size: 116 tasks (19.6% of total)
• Success Rate: 75.9%
• Avg Files Changed: 34.2 (comments: 5.5, reviews: 3.5)
• Top Keywords: output, issue, artifact, failure, job, workflow, run, safe
• Characteristics: Safe-output plumbing, job-failure surfacing, and artifact-handling fixes. Second-largest cluster with the second-lowest merge rate — this failure/output surface area needs more iteration than most, consistent with prior runs' "agent job failures" finding.
• Example PRs: Extend forecast with workflow-run API costs and JSONL history #61230, Preserve imported engine config (including auth) when a workflow sets top-level model #61424, Migrate agentic workflows to Docker runtime #62406, create_pull_request: retry initial branch push on transient workflows-scope timeout #62486, Document Issue Monster AIC artifact upload incident #62493

---

PR Sous Chef & AIC / engine staging fixes

• Size: 77 tasks (13.0% of total)
• Success Rate: 88.3%
• Avg Files Changed: 48.0 (comments: 14.6, reviews: 6.8)
• Top Keywords: em, sub, aic, run, pr sous, chef, sous, sous chef
• Characteristics: Highest comment/review activity of any cluster by a wide margin (AIC accounting, Sous Chef bot interplay, engine-staging fixes), yet still converges well above the overall rate — heavy back-and-forth resolves rather than stalling.
• Example PRs: Prevent Copilot CLI staging for Gemini ARC/DinD workflows #62415, Add github-token-for-extra-empty-commit: none to keep GH_AW_CI_TRIGGER_TOKEN out of compiled workflows #62416, Preserve step-installed skills during base-branch restore; allow Claude's Skill tool #62419, Fix Codex engine.config heredoc indentation #62420, Enforce configured targets in safe-output handlers #62447

---

Model routing & pricing (Copilot/Claude/Codex/Gemini)

• Size: 60 tasks (10.1% of total)
• Success Rate: 78.3%
• Avg Files Changed: 56.4 (comments: 6.5, reviews: 4.0)
• Top Keywords: model, routing, copilot, endpoint, awf, provider, models, codex
• Characteristics: Model-identifier/endpoint routing, pricing/alias additions, and engine-config preservation across providers. Slightly below the overall rate with an above-average diff footprint.
• Example PRs: Add gemini-3.8-flash alias, fix gpt-6-astra/gpt-5.6-sol pricing, add claude-fable-5.1 #61234, Add Copilot pricing metadata for grok-4.7 #62512, Fix Daily Code Metrics Copilot model configuration #62764, Set Pi gateway context window for Copilot Claude Sonnet 5 #63494, Remove implicit model defaults from fallback chains #64172

---

Session/job agent fixes & Actions logs

• Size: 49 tasks (8.3% of total)
• Success Rate: 77.6%
• Avg Files Changed: 43.2 (comments: 5.8, reviews: 3.7)
• Top Keywords: session, job, agent, fix, actions, logs, id, request
• Characteristics: Session/job-id plumbing and Actions log fixes. Tracks slightly below the overall average with moderate engagement.
• Example PRs: Use ${{ runner.temp }} for PLAYWRIGHT_BROWSERS_PATH so install and launch agree on the browser path #61423, Document Gemini Vertex WIF adapter dependency #62488, [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #62715, [WIP] Fix failing GitHub Actions job Integration: CLI Logs Live Cache #62716, Report gateway steering events in audit output #62943

---

MCP tools, CLI & native prompt handling

• Size: 45 tasks (7.6% of total)
• Success Rate: 86.7%
• Avg Files Changed: 39.3 (comments: 4.8, reviews: 3.9)
• Top Keywords: mcp, tool, tools, shell, cli, workflow, native, prompt
• Characteristics: Native tool-call grading, safeoutputs CLI transport hardening, and MCP/CLI prompt handling. Above-average merge rate with below-average comment volume — converges cleanly.
• Example PRs: Include native Copilot tool calls in the automatic grader trace payload #61426, Make safeoutputs CLI transport fail loudly instead of fail open #61427, Fix Code Scanning Fixer timeout and tool denials #61605, Add an opt-in shell-free go-repository tool profile for Copilot SDK workflows #62412, Fix shell permissions for Daily Testify workflow #62423

---

Firewall/network domain handling & AIC accounting

• Size: 41 tasks (6.9% of total)
• Success Rate: 92.7%
• Avg Files Changed: 64.2 (comments: 10.4, reviews: 5.1)
• Top Keywords: blocked, firewall, em, domain, details, summary, sub, aic
• Characteristics: Highest merge rate this run — blocked-domain/firewall handling and daily AIC-accounting edge cases. Narrowly-scoped, verifiable changes continue to converge reliably despite active review discussion.
• Example PRs: Handle legacy pre-harness failures in daily AIC accounting #61232, Prevent AI credits rate-limit false positives from MCP echoes #61425, Allow opting out of "[aw] Detection Runs" tracking issue independently of threat detection #61428, Treat empty add_labels list as a no-op instead of a job failure #61429, Fix Copilot SDK multiword shell prefix matching and denial-guard hang #61430

---

gh-aw CLI version pins & dependency updates

• Size: 36 tasks (6.1% of total)
• Success Rate: 83.3%
• Avg Files Changed: 170.5 (comments: 6.8, reviews: 2.9)
• Top Keywords: version, image, pins, update, versions, pin, cli, release
• Characteristics: By far the largest average diff footprint (pin bumps regenerate many lockfiles/manifests at once), yet still merges above the overall rate — diff size alone doesn't predict failure here.
• Example PRs: Bump go modules, npm packages, and GitHub Action pins per Dependabot update plan #61574, Embed reviewed threat-detect digests in compiled workflows #61857, Pin Copilot CLI default back to 1.0.83 to fix instant HTTP 400 regression in 1.0.85 #62421, Pin Copilot CLI to 1.0.80 to avoid MCP regressions #62482, Update default CLI and service versions #62497

---

Work-queue protocol & issue dispatch

• Size: 27 tasks (4.6% of total)
• Success Rate: 81.5%
• Avg Files Changed: 33.1 (comments: 7.9, reviews: 5.0)
• Top Keywords: queue, work, claim, issue, git, policy, branch, pr
• Characteristics: Work-pool ledger, git-backed work-queue, and claim-policy changes. Sits almost exactly on the overall rate.
• Example PRs: Remove work-pool ledger support #64850, Dispatch Work Coordinator — implementation pending #64853, Add Git-backed work queue operator commands #65062, Include WorkQ claims in the admission prompt #65192, Document work queue orchestration and inspection #65384

---

Threat-detection engine internals (cross-engine) — OUTLIER

• Size: 18 tasks (3.0% of total)
• Success Rate: 50.0%
• Avg Files Changed: 32.9 (comments: 6.0, reviews: 4.4)
• Top Keywords: detection, engine, job, agent, jobs, tests, default, step
• Characteristics: Cross-engine threat-detection plumbing (Codex/Copilot/Goose detector install, AIC/accounting leaks across engine switches, guard-state handling). Lowest merge rate by a wide margin — see root-cause analysis below.
• Example PRs: Treat AWF API proxy guardrail 403s as terminal instead of authentication failures #62487, Propagate max-turn-cache-misses into the threat-detection job #62489, Prevent unused shared logs-cache restores #62491, Scope ARC/DinD detection codegen to the detection job's own runner #62513, Prevent cross-engine version reuse in threat detection #62797

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Files Changed	Top Keywords
0	Firewall/network domain handling & AIC accounting	41	92.7%	64.2	blocked, firewall, domain, aic
3	PR Sous Chef & AIC/engine staging fixes	77	88.3%	48.0	aic, sous chef, engine staging
5	MCP tools, CLI & native prompt handling	45	86.7%	39.3	mcp, tool, cli, native, prompt
7	Workflow fixes, coverage & general validation	123	83.7%	38.8	coverage, workflow, checkout, regression
6	gh-aw CLI version pins & dependency updates	36	83.3%	170.5	version, pins, image, release
2	Work-queue protocol & issue dispatch	27	81.5%	33.1	queue, work, claim, dispatch
4	Model routing & pricing (Copilot/Claude/Codex/Gemini)	60	78.3%	56.4	model, routing, endpoint, provider
1	Session/job agent fixes & Actions logs	49	77.6%	43.2	session, job, agent, logs
8	Agent job failures, safe-outputs & artifacts	116	75.9%	34.2	output, artifact, failure, job
9	Threat-detection engine internals (cross-engine)	18	50.0%	32.9	detection, engine, agent, tests

Key Findings

1. First outlier cluster in recent runs — "Threat-detection engine internals" sits 31.4 points below the overall rate (prior runs' worst gap was ~6 points), making it the clearest systemic signal this analysis period has surfaced.
2. Narrowly-scoped, testable work keeps winning — firewall/network domain handling (92.7%) and PR Sous Chef/AIC fixes (88.3%) remain the top clusters, continuing the pattern that tightly-bounded, verifiable changes converge fastest even with heavy review engagement.
3. Diff size does not predict failure — the largest-footprint cluster (CLI version pins, 170.5 avg files) merges at 83.3%, while the threat-detection outlier (32.9 avg files, one of the smallest footprints) merges at just 50% — what differs is cross-engine correctness complexity, not diff size.
4. Agent job-failure/safe-output handling remains iteration-heavy — the second-largest cluster (116 tasks) sits 5.5 points below the overall rate, echoing the same finding from prior runs for this surface area.

Lowest-Merge Cluster Root Cause

• Cluster: "Threat-detection engine internals (cross-engine)" (18 PRs)
• Merge Rate Gap: 50.0% vs 81.4% overall (gap: 31.4 points)
• Primary Blocker: Review friction driven by cross-engine correctness complexity, compounded by merge-conflict staleness
• Confidence: Medium
• Evidence PRs: Prevent unused shared logs-cache restores #62491, Scope ARC/DinD detection codegen to the detection job's own runner #62513, Prevent cross-engine version reuse in threat detection #62797, Preserve detection AIC in unified session accounting #65323, Add engine max-context-rebuild support and normalize Codex guard hits #68504, Keep OTLP configuration out of AWF agent sandboxes #67242, Report skipped threat detection when the detector was not installed #67810, smoke-copilot-routed: run the default threat-detection engine and check that detection stays unrouted (D1–D3) #67837, Detect agentic engine execution outside the agent job #68579

All 9 non-merged PRs in this cluster (5 closed, 4 still open) were individually reviewed using comments, review bodies, and commit history:

Blocker Category	Count	Share	Notes
Review friction (correctness concerns)	5	55.6%	#62491, #62513, #68504, #67242, #68579 — automated review bots ("Changes recommended" / explicit "Request changes") flagged cross-engine leaks: e.g. #68504's review calls out that "the new max-context-rebuild plumbing still has cross-engine threat-detection leaks... one path keeps the limit when the primary engine was selected through legacy ai:"; #68579 flags "shell parsing gaps permit strict-mode bypasses"
CI/merge-conflict staleness	2	22.2%	#65323 and #67810 both received repeated @copilot resolve the merge conflicts on this branch requests as the detection/engine-normalization files kept moving under them; #65323 was closed unresolved
Scope/complexity mismatch	1	11.1%	#62797 — single commit ("Initial plan"), zero reviews, zero comments; abandoned with no engagement, consistent with a narrow engine-version-pin edge case that was superseded or deprioritized
Other/unclear	1	11.1%	#67837 — review signal is mixed/positive ("Lean already. Ship.") yet the PR remains open; likely just pending merge rather than blocked

Comparing against the 9 merged PRs in the same cluster: review counts there are comparable or even higher (up to 12 reviews on #66284 and #66040), so review volume alone does not separate merged from non-merged — the difference is whether the flagged correctness concern (a genuine cross-engine state leak) gets resolved in place or the PR goes stale waiting on a fix, as the detection/engine-normalization code is a hot, frequently-touched area that invites merge-conflict churn.

Recommendations

1. Target the threat-detection outlier cluster directly — give these tasks explicit acceptance criteria naming the specific engines in scope (e.g. "verify behavior under Codex, Copilot, and Goose, not just the default engine") up front, since nearly all flagged review concerns were cross-engine state leaks missed by single-engine testing.
2. Reduce merge-conflict staleness in this hot area — Preserve detection AIC in unified session accounting #65323 and Report skipped threat detection when the detector was not installed #67810 both stalled on repeated conflict-resolution requests; consider serializing or batching threat-detection/engine-normalization PRs so fewer land concurrently against the same files.
3. Keep leaning into narrow, testable scopes — the two highest-merging clusters (firewall/network, PR Sous Chef/AIC) both reflect well-tested, verifiable diffs; prompt templates for "Agent job failures, safe-outputs & artifacts" (75.9%) could borrow the same explicit acceptance-test framing.
4. Re-attempt workflow-log enrichment — turn counts, duration, and cost were unavailable again this run; if a reachable log-download path exists for future runs, joining it in would let future analyses correlate agent iteration counts with merge outcomes, not just PR-level engagement.

---

Generated by Prompt Clustering Analysis (Run: 38882832935)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 122.7 AIC · ⌖ 1.01 AIC · ⊞ 6.9K · ◷

• expires on Oct 17, 2026, 2:08 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #69671.