[prompt-clustering] Prompt Clustering Analysis - 2026-09-08

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-08-17 to 2026-09-08)
Total Tasks Analyzed: 1000 copilot-agent PRs (all with usable prompt text)
Clusters Identified: 4
Overall Merge Rate: 80.7% (807 merged / 188 closed / 5 open)

Full Analysis Report

General Insights

• Most common task type: Cluster 2 — day-to-day feature/bugfix work on gh-aw core (pkg, coverage, schema, tests) — 468 PRs, 46.8% of all tasks, and the highest merge rate of the "normal" clusters (88.5%).
• Highest success rate (normal work): Cluster 2 at 88.5%, followed by Cluster 3 (workflow/CI/Actions failure-response tasks) at 85.0%.
• Standout outlier: Cluster 0 — automated "operational-value grader/study" tasks — merges at only 11.1%, 69.6 points below the overall rate.
• Workflow-log enrichment unavailable this run: no aw_info.json turn/duration data was present locally, so this report is based on PR metadata, bodies, comments, and reviews only (see Edge Cases below).

Cluster Analysis

Cluster 0: Automated operational-value grader/study records

• Size: 63 tasks (6.3% of total)
• Merge Rate: 11.1% (7 merged / 56 closed)
• Top Keywords: evidence, operational, operational value, intent, grader, workflow
• Characteristics: Machine-generated PRs from the operational-value-designer skill, each evaluating whether a different daily workflow (e.g. daily-choice-test, daily-caveman-optimizer, daily-byok-ollama-test) has a defensible, deterministic "operational value" metric. Titles are templated ("Add operational-value grader for X" / "Record blocked operational-value study for X"). 51 of 63 (81%) received zero comments and zero reviews before being closed — nobody engaged with them at all.
• Example PRs: Record blocked operational-value study for daily-choice-test #58559, Add operational-value grader for daily-caveman-optimizer #58557, Add operational-value grader for cache strategy remediation #58555, Record blocked operational-value study for daily-byok-ollama-test #58554, Record blocked operational-value study for AWF spec surfacing #58553

---

Cluster 1: Agent/engine/workflow-config plumbing

• Size: 322 tasks (32.2% of total)
• Merge Rate: 81.1% (roughly at the overall baseline)
• Top Keywords: workflow, copilot, workflows, agent, model, codex, mcp, engine
• Characteristics: Cross-cutting changes to engine/model support, aw.yml configuration (includes, wildcards), and safe-output plumbing. Larger diffs on average (39.8 files, 832 additions) with heavier review load (13 comments, 5.1 reviews avg) than other clusters.
• Example PRs: Restrict agent artifact packaging to known files #59334, Support wildcard mapping destination folders in aw.yml #59317, Provision Node.js for external Copilot threat detection #59314, Stream large Copilot prompts through stdin #59312, Fix Archie’s nested Mermaid fences #59311

---

Cluster 2: Core feature/bugfix work with test coverage

• Size: 468 tasks (46.8% of total)
• Merge Rate: 88.5% (highest of any sizeable cluster)
• Top Keywords: pkg, coverage, added, workflow, existing, test, error, schema
• Characteristics: The bread-and-butter cluster — package-level features, bug fixes, schema changes, each backed by added test coverage. Smaller diffs (19.4 files avg) than Cluster 1, moderate review load.
• Example PRs: Prune logs cache during concurrent downloads #59338, Improve logs cache size diagnostics #59336, Guard OTLP endpoints against scheme-only authorization headers #59315, Retry transient slash-command provenance checks #59313, Cache complete workflow run metadata in logs #59257

---

Cluster 3: CI/Actions failure response

• Size: 147 tasks (14.7% of total)
• Merge Rate: 85.0%
• Top Keywords: pr, job, actions, run, fix, safe, workflow, failure
• Characteristics: Reactive fixes to failing CI jobs, flaky tests, and safe-output pipeline issues.
• Example PRs: Fail workflows when agents report incomplete #59316, Cache GitHub Actions job metadata in logs output #59039, Stage Linear and Jira smoke issue outputs #58786, Validate top-level env expression contexts #58781, Fix flaky SIGPIPE/pipefail false positives in safe-outputs conformance checker #58768

Success Rate by Cluster

Cluster	Tasks	Merge Rate	Avg Comments	Avg Reviews	Avg Files Changed	Top Keywords
0 — operational-value studies	63	11.1%	0.5	0.2	2.5	evidence, operational, grader
1 — engine/config plumbing	322	81.1%	13.0	5.1	39.8	workflow, copilot, agent, mcp
2 — core feature/bugfix + tests	468	88.5%	6.9	3.4	19.4	pkg, coverage, test, schema
3 — CI/Actions failure response	147	85.0%	5.4	2.7	28.3	job, actions, fix, failure

Full Data Table (sample, 5 per cluster — 1000 total PRs analyzed)

PR #	Title	Cluster	Outcome
#58559	Record blocked operational-value study for daily-choice-test	0	Closed
#58557	Add operational-value grader for daily-caveman-optimizer	0	Closed
#58555	Add operational-value grader for cache strategy remediation	0	Closed
#58554	Record blocked operational-value study for daily-byok-ollama-test	0	Closed
#58553	Record blocked operational-value study for AWF spec surfacing	0	Closed
#59334	Restrict agent artifact packaging to known files	1	Merged
#59317	Support wildcard mapping destination folders in aw.yml	1	Merged
#59314	Provision Node.js for external Copilot threat detection	1	Open
#59312	Stream large Copilot prompts through stdin	1	Closed
#59311	Fix Archie's nested Mermaid fences	1	Merged
#59338	Prune logs cache during concurrent downloads	2	Merged
#59336	Improve logs cache size diagnostics	2	Merged
#59315	Guard OTLP endpoints against scheme-only authorization headers	2	Merged
#59313	Retry transient slash-command provenance checks	2	Merged
#59257	Cache complete workflow run metadata in logs	2	Merged
#59316	Fail workflows when agents report incomplete	3	Merged
#59039	Cache GitHub Actions job metadata in logs output	3	Merged
#58786	Stage Linear and Jira smoke issue outputs	3	Merged
#58781	Validate top-level env expression contexts	3	Merged
#58768	Fix flaky SIGPIPE/pipefail false positives in safe-outputs conformance checker	3	Merged

Key Findings

1. The operational-value cluster is not a review-quality problem — it's a batch of automated negative-result records. 81% of Cluster 0 PRs got zero comments and zero reviews before closing. These are self-generated design docs where the agent concludes a target daily workflow has no defensible deterministic "operational value" metric, records that finding as JSON, and sets designOutcome: blocked — not code intended to ship.
2. This is a persistent, recurring pattern, not a one-off. Yesterday's run (2026-09-07) found the same cluster at 65 PRs / 9.2% merge rate with identical top keywords (operational value, blocked, grader). The pattern has now shown up at least two days running with near-identical size and merge rate.
3. Overall merge rate improved day-over-day: 77.1% (2026-09-07, n=1226) → 80.7% (today, n=1000). Excluding Cluster 0 entirely, the "real" merge rate across the other three clusters is 85.4%.
4. The bot-footer boilerplate ("PR Sous Chef" run links, firewall warnings) dominates raw TF-IDF terms if not stripped — an early pass of this analysis produced a nonsense cluster keyed on fragments like "em", "sub", "aic" before the footer-stripping regex was corrected. Future runs of this analysis should keep the footer/marker-stripping step (this repo's <!-- START COPILOT CODING AGENT SUFFIX --> marker demarcates trailing boilerplate that appears after the real task text, not before it — the opposite of the naive assumption).

Lowest-Merge Cluster Root Cause

• Cluster: 0 — "operational-value grader/study" records
• Merge Rate Gap: 11.1% vs 80.7% overall (gap: 69.6 points)
• Primary Blocker: Scope/complexity mismatch (by design) — the task itself is a research/evaluation probe, not a shippable change, and most instances conclude the probe is infeasible
• Confidence: High
• Evidence PRs: Record blocked operational-value study for daily-choice-test #58559, Document blocked Craft operational-value grader design #58542, operational-value-study: block copilot-opt (no defensible direct metric) #58535, Block operational-value grader design for copilot-centralization-optimizer #58532, Record operational value study for archivx-agentic-workflows-analyzer #58501 (all zero comments/reviews, explicit "blocked" designOutcome in the PR body)

Blocker Category	Count	Share	Notes
Scope/complexity mismatch (by design "blocked" record)	~51 of 56 closed	~91%	Zero comments, zero reviews; PR body itself declares the workflow has no defensible direct metric and sets designOutcome: blocked
Review friction	0 of 56 closed	0%	No PRs in this cluster show reviewer pushback — they were closed unreviewed
CI/test failure	0 observed	0%	Not a factor for this cluster (docs/JSON-only changes, no lock file or evaluator changes)
Other/unknown	~5 of 56 closed	~9%	A handful of closed PRs lack the "blocked" keyword in the title but follow the same zero-engagement pattern

The 7 merged PRs in this cluster are qualitatively different: they are the tooling/documentation layer for the operational-value system itself (e.g. "Document the graders CLI command", "Rename aw-value skill to operational-value-designer"), and each received 1–4 actual reviews — i.e., work a human was expected to act on, versus the per-workflow probes which were not.

Recommendations

1. Separate "probe" PRs from "shippable" PRs in reporting. Since Cluster 0 PRs are intentionally-negative research records rather than failed contributions, folding them into the same merge-rate metric as feature work understates the real project health (85.4% excluding them vs. 80.7% blended).
2. Consider auto-closing or skipping PR creation entirely for designOutcome: blocked results. If the operational-value-designer workflow already knows at generation time that the outcome is "blocked," opening a PR that will receive zero review before closing may be unnecessary process overhead — a study-log commit or issue comment could carry the same information without occupying PR review queues.
3. Keep leveraging Cluster 2's pattern (feature/bugfix + test coverage) as the reference for high-merge-rate work — it's the largest cluster and has the best success rate among substantive tasks; nothing here suggests a need for change.
4. Outlier Cluster Recommendation: For Cluster 0, track a distinct "probe success rate" metric (e.g., % of daily workflows that got an actionable grader, not blocked) rather than treating "closed without merging" as project failure — the current framing makes an intentional research process look like a quality problem.

---

Generated by Prompt Clustering Analysis (Run: 34211692380)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

• api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 229.2 AIC · ⌖ 12.2 AIC · ⊞ 16.2K · ◷

• expires on Sep 9, 2026, 2:14 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #59688.