[prompt-analysis] Copilot PR Prompt Analysis - 2026-08-18

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-08-18

Summary

Analysis Period: Last 30 days (2026-08-02 → 2026-08-18)
Total PRs: 1000 | Merged: 763 (76.3%) | Closed: 218 (21.8%) | Open: 19 (1.9%)

Note: The dataset is capped at 1000 PRs (the most recent 1000 in the 30-day window), so percentages reflect this sample rather than a full count of Copilot activity.

Prompt Categories and Success Rates

Category assignment uses keyword matching against PR title + body (first matching category wins; most PRs are bug-fix-flavored since the pool includes many automated vulnerability/CI-fix tasks).

Category	Total	Merged	Closed	Open	Success Rate*
Bug Fix	922	696	207	19	77.1%
Feature Addition	67	57	10	0	85.1%
Testing	5	4	1	0	80.0%
Refactoring	3	3	0	0	100.0%
Documentation	2	2	0	0	100.0%
Other	1	1	0	0	100.0%

*Success rate = merged / (merged + closed), excluding still-open PRs.

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: 189 words (notably shorter than closed PRs)
• Most common substantive keywords: workflow, actions, runs, copilot, agent, engine, repo, model, comment, call
• Frequently structured with a clear Summary section followed by a Changes bullet list referencing specific files/functions (e.g. pkg/cli/..., BoundedQueriesConfig)

Example successful prompts:

1. PR [test-parallel] test(cli): parallelize safe Go tests in pkg/cli (batch) #53721: "Analyzed 25 Go test files in pkg/cli... using per-file parallel-safety-checker sub-agents, and added t.Parallel() only where demonstrably safe..." → Merged
2. PR Fix Timeout type drift between BoundedQueriesConfig and AWFBoundedQueriesConfig #53694: "BoundedQueriesConfig... and AWFBoundedQueriesConfig... are parallel structs... but their Timeout fields had drifted: *int in one, int in the other. This risks silently mismatching nil-vs-zero timeouts..." → Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: 255 words (~35% longer than merged PRs)
• Dominant keywords: redacted, debian, securitytracker, ubuntu, lgpl, high/medium (severity labels) — heavily skewed toward automated container/OS-package CVE advisory PRs
• Vulnerability/CVE-cluster PRs are a distinct, much weaker sub-population: of 67 PRs referencing securitytracker/debian/cve/vulnerab*, only 33 merged vs. 34 closed (49.3% success) — dramatically below the 78.4% rate for all other PRs (730 merged / 914 completed = 79.9%)
• Longer prompts often pack in redacted security-advisory boilerplate (package names, severity, LGPL/license text) rather than an actionable, scoped change description

Example unsuccessful prompts:

1. PR Add task-preflight skill to stop zero-diff retry PRs on container-vuln and refactor tasks #53674: "Container-vulnerability-fix and large-refactor tasks repeatedly burned full agent sessions: 5 of 6 closed PRs across three topics had 0 additions/0 deletions and no comments, so each retry restarted from zero..." → Closed
2. PR Suppress RGS-016 false positive in agentic_commands.yml generator #53609: "The daily static analysis report flagged a Critical-severity runner-guard finding (RGS-016...) in the compiler-generated .github/workflows/agentic_commands.yml. Root cause: RGS-016 fires on VARIATION SELECTOR-16..." → Closed
3. PR Support temporary IDs for dispatched workflow runs #53572: "Agents can now assign a temporary ID when dispatching a workflow and reuse it to approve the returned workflow run..." → Closed (feature-shaped prompt, still closed — outcome isn't purely category-driven)

Key Insights

• Vulnerability/CVE-advisory prompts are the single weakest cluster: ~50% success rate vs. ~80% for everything else, despite being individually well-specified. These tasks likely fail for reasons outside the prompt itself (upstream-only fixes, no actionable in-repo change, duplicate/stale advisories) rather than poor prompt quality — this matches the repo's own prior finding that "Container/Image Security Pinning" PRs merge well below the fleet average.
• Shorter, structurally consistent prompts merge more often: merged PRs average 189 words vs. 255 for closed, and tend to follow a Summary → Root cause/Changes (bulleted, file-scoped) → optional example structure rather than pasting large blocks of advisory/license text.
• Feature-addition prompts modestly outperform bug-fix prompts (85.1% vs 77.1%), suggesting well-scoped "add X" tasks with clear boundaries convert at least as well as fixes, contrary to a common assumption that fixes are "safer" asks.

Recommendations

1. DO: Keep prompts concise (~150–200 words) with a short rationale followed by a bulleted, file/function-scoped change list — this pattern dominates the merged set.
2. DO: For feature requests, state the specific API/tool surface being extended (e.g., "adds blocked_by to the create_issue tool schema") rather than a vague goal.
3. AVOID: Auto-generating CVE/vulnerability-fix tasks from raw advisory dumps (package name + severity + license boilerplate) without first confirming the fix is actionable in this repo — this cluster converts at roughly half the overall rate and should be pre-filtered (as already proposed in PR Pre-filter upstream-blocked container/image CVE findings before Copilot assignment #53709) before Copilot assignment.

Historical Trends

Date	PRs	Success Rate	Top Category
2026-07-03	1000	81.8%	bug_fix
2026-07-04	1000	81.1%	bug_fix
2026-07-05	1000	80.8%	bug_fix
2026-07-06	1000	80.4%	test
2026-08-18	1000	77.8%	bug_fix

Trend: Success rate has drifted down roughly 3 points since early July (81.8% → 77.8%), consistent with a growing share of the low-converting CVE/vulnerability-advisory cluster in the PR mix. No historical data exists for the intervening ~6 weeks (data-collection gap), so this comparison is directional only.

---

Generated by Copilot PR Prompt Analysis (Run: 32183878122)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · auto · 23.3 AIC · ⌖ 7.36 AIC · ⊞ 11.1K · ◷

• expires on Aug 19, 2026, 12:53 PM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot PR Prompt Pattern Analysis.

A newer discussion is available at Discussion #54079.