[prompt-analysis] Copilot PR Prompt Analysis - 2026-08-09

[github-actions[bot]]

Summary

Analysis Period: Last 30 days
Total PRs: 1000 | Merged: 749 (74.9%) | Closed: 250 (25.0%) | Open: 1 (0.1%)

Prompt Categories and Success Rates

Category	Total	Merged	Success Rate
Refactoring	3	3	100.0%
Testing	10	9	90.0%
Feature Addition	112	98	87.5%
Other	6	5	83.3%
Bug Fix	866	632	73.1%
Documentation	3	2	66.7%

Note: category detection uses keyword matching on the extracted issue/task text; "Bug Fix" dominates because most Copilot PRs in this repo originate from auto-filed audit/vulnerability issues that use fix-oriented language.

🔑 Headline Finding: [WIP] Title Prefix Predicts Failure

The single strongest signal in the dataset isn't a keyword — it's whether the PR ever carries a [WIP] title prefix (GitHub Copilot coding agent auto-adds this while it's still working):

Group	Total	Merged	Closed	Stall/Close Rate
[WIP]-prefixed PRs	74	1	73	98.6%
Non-[WIP] PRs	886	709	177	20.0% closed

PRs that stayed in (or got closed while still in) the [WIP] state almost never merged. This strongly suggests these are cases where the agent stalled, hit an error loop, or the task was abandoned/superseded before the agent finished — not necessarily a "bad prompt" per se, but a bad outcome correlated with incomplete task framing (ambiguous scope, missing repro info, or conflicting/duplicate work).

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 170.6 words (more concise than closed PRs)
• Most common content words: workflow, actions, run, copilot, engine, agent, sub
• Titles tend to name a concrete action + target: fix/add/refactor + specific file, workflow, or subsystem

Example successful prompts:

1. PR Normalize report formatting guidance in 5 daily audit workflows #51612: "Normalize report formatting guidance in 5 daily audit workflows" → Merged
2. PR Accept ADR-43244 and add predicate traceability #51611: "Accept ADR-43244 and add predicate traceability" → Merged
3. PR Grant agentic engines read/write access to /tmp/gh-aw in AWF sandbox #51608: "Grant agentic engines read/write access to /tmp/gh-aw in AWF sandbox" → Merged

These share a pattern: a specific target (a named workflow set, an ADR ID, a specific sandbox path) rather than a vague ask.

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 218.1 words (~28% longer than merged prompts)
• Frequent terms: cve, vulnerability, debian, gpl, securitytracker — many closed PRs originate from automated container-vulnerability-scan issues bundling long CVE dumps
• 73 of 250 closed PRs (29%) never left [WIP] state

Example unsuccessful prompts:

1. PR [WIP] Create /llms.txt for the docs site #51616: "[WIP] Create /llms.txt for the docs site" → Closed (stalled in WIP)
2. PR [WIP] Fix Copilot billing 403 masking Daily Max Ai Credits Test #51607: "[WIP] Fix Copilot billing 403 masking Daily Max Ai Credits Test" → Closed (stalled in WIP)
3. PR Resolve logger/colorwriter import cycle in CI #51572: "Resolve logger/colorwriter import cycle in CI" → Closed (completed but not merged — likely superseded or duplicate fix)

Key Insights

• [WIP] stalling is the dominant failure mode (98.6% closed among 74 such PRs) — far more predictive of outcome than any single keyword. This likely reflects agent execution issues (timeouts, tool failures, ambiguous instructions) rather than prompt wording alone.
• Longer, CVE-dump-style prompts correlate with lower merge rates. Prompts padded with raw vulnerability-scanner output (debian/gpl/securitytracker boilerplate) are ~28% longer on average and cluster in the closed bucket — likely because bulk/automated CVE issues often get triaged as "no action needed" or superseded by a scanner re-run.
• Concise, specifically-scoped prompts merge more often. Merged PRs reference a named workflow, file, ADR, or subsystem rather than a broad category; specificity appears to correlate with actionability.

Recommendations

1. DO: Scope prompts to a single named file, workflow, or ADR/issue ID rather than a broad category — this matches the pattern in high-success merged PRs.
2. DO: Keep prompts concise; avoid pasting large raw scanner/CVE output directly into the task body — summarize the actionable finding instead and link to the source report.
3. AVOID: Filing tasks where the agent is likely to stall mid-work (e.g., very open-ended or exploratory asks) — monitor for PRs stuck in [WIP] beyond a day and close/re-issue them promptly rather than letting them linger, since the data shows they almost never self-resolve to a merge.

Historical Trends

No prior history was found in repo-memory (memory/prompt-analysis branch) — this is the first recorded analysis. Historical daily_analysis data has been initialized with today's snapshot (2026-08-09) for future trend comparison.

---

Generated by Copilot PR Prompt Analysis (Run: 31335227689)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · auto · 24.6 AIC · ⌖ 5.23 AIC · ⊞ 10.7K · ◷

• expires on Aug 10, 2026, 12:54 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-10T20:54:43.749Z.

Closed by Workflow