[prompt-clustering] Prompt Clustering Analysis - 2026-10-12

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-09-12 to 2026-10-12)
Total Tasks Analyzed: 552
Clusters Identified: 10
Overall Success Rate: 84.2% (465 merged / 552 total)

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 4 — safe-output/security hardening (94 tasks, 17.0% of all PRs), merge rate 90.4%. These are scanner/grader/memory-validation hardening fixes, a sign the security-review surface is actively maturing.
• Second Most Common: Cluster 8 — version bumps & release automation (90 tasks, 16.3%), merge rate 84.4%. Routine firewall/MCP/tool version pins; despite being "routine," this cluster has the highest average footprint (82 files changed per PR) of any cluster, since pin bumps often touch many lockfiles/vendor manifests at once.
• Highest Success Rate: Cluster 3 — firewall/network domain handling (92.6% merged, 54 tasks). Same top performer as the past several runs — tightly-scoped, well-understood changes (blocked-domain fixes, firewall version bumps) that reviewers approve quickly.
• Most Complex Tasks: Cluster 8 (version bumps) at 82.4 avg files changed, narrowly ahead of Cluster 2 (usage accounting, 80.3 avg files changed).
• Lowest Success Rate: Cluster 2 — usage accounting & eval detection (76.6% merged, 47 tasks). Below the overall rate by 7.6 points, but short of the outlier threshold (needs ≥10-point gap at ≥15 PRs) — see note below.
• Outliers: None this run. No cluster met the outlier bar (≥15 PRs and ≥10-point merge-rate gap below the 84.2% overall rate). This is the first run in recent weeks without a flagged outlier cluster — the closest is Cluster 2 at a 7.6-point gap.
• Data coverage: Full PR detail (comments, reviews, file counts) was available for all 552 PRs (100% coverage) — the best data completeness of any run to date. Workflow run logs (agenticworkflows logs) were probed for turn/cost/duration metrics; a single count-limited 5-run request took ~17s and returned only a 1-hour slice of the 30-day window, so pulling full-period turn/cost data remains infeasible in this run's time budget. Findings below rely on PR metadata, comments, and reviews only.

Cluster Analysis

Cluster 4: Safe-output/security hardening

• Size: 94 tasks (17.0% of total)
• Success Rate: 90.4%
• Avg Files Changed: 31.0 (comments: 7.0, reviews: 4.1)
• Top Keywords: ledger, validation, memory, add, coverage, workflow, runtime, safe
• Characteristics: Grader sandbox escapes, YAML/scanner hardening, memory baseline repair, PR push file-policy scoping. High merge rate reflects well-defined, test-covered security fixes.
• Example PRs: Clarify safe-output permissions and workflow trigger guidance #67737, Close the custom grader sandbox escape #67475, Harden workflow YAML and scanner file handling #67473, Add tracker-ID payload regressions for PR comments and issues #67441, Add declarative JSON schema validation for memory #67439

---

Cluster 8: Version bumps & release automation

• Size: 90 tasks (16.3% of total)
• Success Rate: 84.4%
• Avg Files Changed: 82.4 (comments: 6.2, reviews: 3.2)
• Top Keywords: version, generated, workflow, aw, release, workflows, setup, gh aw
• Characteristics: Firewall/MCP/tool pin bumps and builtin grader-default loading. Largest footprint of any cluster — single-purpose PRs that nonetheless touch dozens of generated lockfiles/manifests.
• Example PRs: Load builtin grader defaults from setup-action JSON #67777, Support Cloud Hypervisor primaries with static script enclaves #67772, Bump gh-aw-firewall to v0.28.51 #67758, Emit AWF agentTimeout for literal timeouts and show delivered steering notices in audit/logs #67597, Investigate deterministic installs in compiled workflows #67469

---

Cluster 5: Test & CI reliability fixes

• Size: 85 tasks (15.4% of total)
• Success Rate: 80.0%
• Avg Files Changed: 32.5 (comments: 6.1, reviews: 3.9)
• Top Keywords: test, failure, run, issue, retry, error, coverage, existing
• Characteristics: Race-detector coverage, flaky-test replay tests, linter blind-spot fixes. Second-lowest merge rate — these PRs often depend on reproducing intermittent CI failures, which adds review friction.
• Example PRs: Add a replay test for the Copilot startup model/endpoint mismatch fail-fast #67797, Add race-detector coverage for concurrent Go packages #67623, Close recurring Go linter analyzer blind spots #67544, Require evidence before reporting red-team security findings #67542, Restore safe-output server startup after memory schema change #67506

---

Cluster 0: MCP/tool & engine allowlists

• Size: 55 tasks (10.0% of total)
• Success Rate: 85.5%
• Avg Files Changed: 49.3 (comments: 5.4, reviews: 3.1)
• Top Keywords: copilot, mcp, tool, engine, workflow, web, codex, tools
• Characteristics: GitHub MCP/Playwright/Syft/Grype pin updates, tool-allowlist alignment across engines.
• Example PRs: Update GitHub MCP, Playwright, Syft, and Grype default pins #67886, Add governance and Copilot Automation tools to GitHub MCP mapping #67738, Align workflow tool allowlists with required operations #67472, Expose Codex standalone web search for OpenAI Responses Lite models #67246, Allow Copilot coding agents through the bot allowlist without collaborator lookup #67238

---

Cluster 3: Firewall/network domain handling

• Size: 54 tasks (9.8% of total)
• Success Rate: 92.6%
• Avg Files Changed: 65.8 (comments: 14.4, reviews: 6.8)
• Top Keywords: network, blocked, domain, firewall, workflow, domains, com, frontmatter
• Characteristics: Best-performing cluster again this run. High comment/review counts but still the highest merge rate — reviewers engage heavily but these PRs are well-scoped and converge.
• Example PRs: Deduplicate memory allowed-extension parsing #67543, Bump gh-aw-firewall to v0.28.50 #67035, Normalize unified agent session metadata and AIC accounting #65344, Claude: default to strict MCP configuration #64970, Fix Copilot session state collection #64461

---

Cluster 7: Model routing & engine smokes

• Size: 51 tasks (9.2% of total)
• Success Rate: 78.4%
• Avg Files Changed: 61.5 (comments: 7.1, reviews: 4.2)
• Top Keywords: model, routing, copilot, endpoint, pi, models, gpt, codex
• Characteristics: Smoke tests across engine/model combinations (Claude, Copilot, Codex routing), docs on routing modes. Below-average merge rate, consistent with multi-engine smoke work needing more iteration to stabilize.
• Example PRs: smoke-copilot-routed: run the default threat-detection engine and check that detection stays unrouted (D1–D3) #67837, Add Claude-engine model-routing smoke (smoke-claude-routed) #67835, docs(model-routing): explain Copilot CLI vs SDK mode, define allowed-models, add compiling examples #67755, Fix routing smoke assertions: attribute agent traffic by model instead of nonexistent purpose values #67595, Add model-routing smoke workflows with deterministic runner/proxy evidence checks #67523

---

Cluster 2: Usage accounting & eval detection

• Size: 47 tasks (8.5% of total)
• Success Rate: 76.6% (lowest this run)
• Avg Files Changed: 80.3 (comments: 8.0, reviews: 4.1)
• Top Keywords: usage, accounting, artifact, detection, evals, job, ai, failed
• Characteristics: AI-credit/AIC guardrail accounting, threat-detection reporting, eval-run cost attribution. 11 of 47 PRs (23%) did not merge; spot-checked blockers were mixed — stale branches needing manual conflict resolution, security-scan gate failures, and scope creep on multi-job accounting changes — no single dominant blocker, which is why this falls short of the outlier threshold.
• Example PRs: Add an emoji-density experiment to daily-choice-test #67889 (open), Report skipped threat detection when the detector was not installed #67810 (open), Extending work queue protocol to support checkpoints and maintenance jobs #67105, Restore classifier cost attribution in routed-run audits #67092, Prevent unresolved AI credit estimates from blocking workflows #66559 (closed)

---

Cluster 1: Checkout/authorization fallbacks

• Size: 41 tasks (7.4% of total)
• Success Rate: 85.4%
• Avg Files Changed: 28.5 (comments: 8.9, reviews: 4.3)
• Top Keywords: target, repository, checkout, pr, workflow, regression, coverage
• Characteristics: GitHub App checkout allowlisting, activation-output fallbacks, repository-authorization conformance.
• Example PRs: Allow allow-listed GitHub Apps to check out same-repository PRs after comment triggers #67247, Add fallbacks for downstream activation repository outputs #67244, Fix SEC-005 repository authorization conformance for Claim adapters #67132, Avoid GHES masking of cross-job workflow outputs #66678, Add explicit strict-mode policy for pull_request_target checkouts #66649

---

Cluster 9: Work-queue & git policy handling

• Size: 24 tasks (4.3% of total)
• Success Rate: 79.2%
• Avg Files Changed: 31.5 (comments: 9.0, reviews: 5.3)
• Top Keywords: work, queue, git, pr, branch, policy, state, changes
• Characteristics: Work-queue initialization hardening, git-policy scoping to agent commits, frontmatter-SHA identity work.
• Example PRs: Use frontmatter SHA as workflow evaluation identity #67300, Harden work-queue initialization for empty repositories #67297, [WIP] Restore classifier cost attribution in routed-run audits #67230, Dispatch engine conformance through a priority-aware work queue #67222, Use repository labels for work-queue Issue status #67223

---

Cluster 6: CI job failure fixes (WIP)

• Size: 11 tasks (2.0% of total)
• Success Rate: 81.8%
• Avg Files Changed: 1.6 (comments: 1.6, reviews: 1.0)
• Top Keywords: fix, actions, job, cause, make, root, failing, work
• Characteristics: Smallest cluster — auto-filed [WIP] Fix failing GitHub Actions job PRs with minimal diffs and little review activity, consistent with bot-triggered CI-failure auto-repair tasks.
• Example PRs: [WIP] Fix failing GitHub Actions job for integration add dependencies #66673, [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #64076, [WIP] Fix failing GitHub Actions job 'build' #64053, [WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2 #63235, [WIP] Fix failing GitHub Actions job Integration: CLI Logs Live Cache #62716

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Files Changed	Top Keywords
4	Safe-output/security hardening	94	90.4%	31.0	ledger, validation, memory
8	Version bumps & release automation	90	84.4%	82.4	version, release, workflows
5	Test & CI reliability fixes	85	80.0%	32.5	test, failure, retry
0	MCP/tool & engine allowlists	55	85.5%	49.3	copilot, mcp, tool, engine
3	Firewall/network domain handling	54	92.6%	65.8	network, blocked, domain
7	Model routing & engine smokes	51	78.4%	61.5	model, routing, endpoint
2	Usage accounting & eval detection	47	76.6%	80.3	usage, accounting, evals
1	Checkout/authorization fallbacks	41	85.4%	28.5	checkout, repository, pr
9	Work-queue & git policy handling	24	79.2%	31.5	work, queue, git, policy
6	CI job failure fixes (WIP)	11	81.8%	1.6	fix, actions, job

Full Data Table

Representative PRs per cluster (5 each)

PR #	Title	Cluster	Outcome
67737	Clarify safe-output permissions and workflow trigger guidance	4	Merged
67475	Close the custom grader sandbox escape	4	Merged
67777	Load builtin grader defaults from setup-action JSON	8	Merged
67772	Support Cloud Hypervisor primaries with static script enclaves	8	Merged
67797	Add a replay test for the Copilot startup model/endpoint mismatch fail-fast	5	Merged
67623	Add race-detector coverage for concurrent Go packages	5	Merged
67886	Update GitHub MCP, Playwright, Syft, and Grype default pins	0	Merged
67738	Add governance and Copilot Automation tools to GitHub MCP mapping	0	Merged
67543	Deduplicate memory allowed-extension parsing	3	Merged
67035	Bump gh-aw-firewall to v0.28.50	3	Merged
67837	smoke-copilot-routed: run the default threat-detection engine	7	Merged
67835	Add Claude-engine model-routing smoke (smoke-claude-routed)	7	Merged
67889	Add an emoji-density experiment to daily-choice-test	2	Open
67105	Extending work queue protocol to support checkpoints and maintenance jobs	2	Merged
67247	Allow allow-listed GitHub Apps to check out same-repository PRs	1	Merged
66678	Add fallbacks for downstream activation repository outputs	1	Merged
67300	Use frontmatter SHA as workflow evaluation identity	9	Merged
67230	Harden work-queue initialization for empty repositories	9	Merged
66673	[WIP] Fix failing GitHub Actions job for integration add dependencies	6	Merged
64076	[WIP] Fix failing GitHub Actions job Integration: Workflow Misc Part 2	6	Merged

Key Findings

1. Clean run, no outliers: For the first time in recent weeks, no cluster crossed the outlier threshold (≥15 PRs, ≥10-point merge-rate gap). The lowest-performing cluster (usage accounting, 76.6%) is only 7.6 points below the 84.2% overall rate, and its non-merges trace to varied causes (stale branches, security-gate failures, scope creep) rather than one systemic blocker.
2. Firewall/network work keeps winning: Cluster 3 (firewall/network domain handling) has had the highest or near-highest merge rate in every recent analysis — these PRs are narrowly scoped against a concrete, testable target (a blocked domain or firewall pin), making them easy for reviewers to verify.
3. Overall merge rate is improving: 84.2% this run vs. 83.8% (2026-10-11), 75.3% (2026-10-04), and 76.3% (2026-10-03) — a steady upward trend over the past two weeks, coinciding with the disappearance of the previously-recurring "operational-value grader" (0% merge) and "CLI/MCP version pinning" outlier clusters seen in September and early October.
4. Version-bump PRs are the heaviest, not the lightest: Cluster 8 (version bumps & release automation) has the largest average diff footprint (82 files) of any cluster, because dependency/firewall pin updates regenerate multiple lockfiles and manifests at once — worth knowing when estimating review cost for "routine" bump PRs.
5. Model-routing smokes lag slightly: Cluster 7 (model routing & engine smokes) sits 5.8 points under the overall merge rate, suggesting routing/engine-combination smoke tests take more iterations to land than single-engine changes — likely inherent to the combinatorial nature of the work rather than a quality issue.

Recommendations

1. No corrective action needed for outliers this run — the absence of a flagged cluster suggests recent fixes to the operational-value grader and dependency-pinning processes (addressed in prior runs) have taken hold. Continue monitoring for regression.
2. Usage-accounting PRs (Cluster 2, lowest merge rate): triage the 9 closed PRs for patterns before the next cycle — several appear to be stale (merge-conflict) closures rather than design rejections, so a lightweight "rebase or close" sweep could lift this cluster's rate without further scoping work.
3. Model-routing smoke PRs (Cluster 7): consider splitting multi-engine smoke additions into per-engine PRs where feasible, since the combinatorial surface area likely drives the extra review iterations seen here.
4. Workflow-log metrics remain unavailable at 30-day scale through the current agenticworkflows logs tool within this run's time budget; if turn-count/cost trends are wanted going forward, consider a narrower fixed window (e.g., last 3 days) sampled on a rolling basis and cached, rather than attempting a full 30-day pull each run.

---

Generated by Prompt Clustering Analysis (Run: 38233494831)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 186.6 AIC · ⊞ 6.9K · ◷

• expires on Oct 13, 2026, 2:18 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-10-13T10:18:05.814Z.

Closed by Workflow