[prompt-analysis] Copilot PR Prompt Analysis - 2026-08-19

[github-actions[bot]]

🤖 Copilot PR Prompt Pattern Analysis - 2026-08-19

Summary

Analysis Period: Last 30 days
Total PRs: 1000 | Merged: 770 (78.3%) | Closed: 214 (21.7%) | Open: 16

Note: the dataset was capped at 1000 PRs (the maximum fetched), so it may not cover the full 30-day window if activity exceeded this limit.

Prompt Categories and Success Rates

(Categories are non-exclusive; a prompt can match multiple keyword patterns. Rates are computed over completed PRs — merged + closed.)

Category	Total	Merged	Success Rate
Bug Fix	612	437	71.4%
Feature Addition	498	365	73.3%
Testing	346	257	74.3%
Documentation	190	142	74.7%
Refactoring	57	30	52.6%
Other (no keyword match)	281	254	90.4%

Prompt Analysis

✅ Successful Prompt Patterns

Common characteristics in merged PRs:

• Average prompt length: ~105 words (notably shorter than closed prompts)
• Most common keywords: github, workflow, sous/chef/souschef (referencing the sous-chef automation agent), agent, comment, actions, copilot, runs
• Frequent pattern: prompts that reference a specific failing job/run, e.g. "Fix the failing GitHub Actions job... Analyze the Actions logs, identify the root cause... Check run ID: ... Job URL: ..."
• Frequent pattern: Dependabot-style dependency bump prompts with explicit package name and version range.
• Frequent pattern: automated "Schema Consistency Checker" / bot-generated prompts that describe a narrow, well-scoped gap.

Example successful prompts:

1. PR [WIP] Fix failing GitHub Actions job Integration: Workflow Compiler #54008: "Fix the failing GitHub Actions job "Integration: Workflow Compiler" Analyze the Actions logs, identify the root cause of the failure, and implement a..." → Merged
2. PR Document threat-detection-suppress and max-runs deprecation; fix stale check-for-updates link #53990: "The Schema Consistency Checker found three documentation gaps: threat-detection-suppress was fully implemented but undocumented outside the generate..." → Merged
3. PR Update charmbracelet golden test helper dependency #53970: "Dependabot identified a safe patch-level update for the direct Go dependency github.com/charmbracelet/x/exp/golden. The upstream range does not modi..." → Merged

❌ Unsuccessful Prompt Patterns

Common characteristics in closed PRs:

• Average prompt length: ~176 words — roughly 68% longer than successful prompts
• Most common keywords: github, workflow, issue, agent, daily, engine, test, start — more generic/discussion-oriented vocabulary, less task-specific
• Frequent pattern: prompts embedding a quoted original GitHub issue (<issue_title> blocks) rather than a distilled, actionable task
• Frequent pattern: broad, exploratory asks covering many files/behaviors at once (e.g., "the remaining seven depend on event-specific payloads...") rather than one focused fix
• Frequent pattern: prompts describing subjective UX/doc restructuring ("presented a flat 5-tab... forcing the reader...") without a concrete acceptance criterion

Example unsuccessful prompts:

1. PR [WIP] Fix safe-outputs.runs-on type mismatch #53989: "---- This section details on the original issue you should resolve (issue_title)[deep-report] Fix safe-outputs.runs-on type mismatch: parser only..." → Closed
2. PR docs: default Quick Start auth section to Copilot, name the curl fallback trigger #53959: "The Quick Start guide's Step 2 presented a flat 5-tab authentication block (Copilot/Claude/Codex/Gemini/Pi) before the reader had run anything, forcin..." → Closed
3. PR Verify manual-trigger exclusions for event-context workflows #53897: "The workflow fleet has 279 manual-dispatchable lockfiles. The remaining seven depend on event-specific payloads and cannot safely support a context-fr..." → Closed

Key Insights

• Conciseness correlates with success: merged-PR prompts average ~105 words vs. ~176 words for closed PRs — a ~68% length gap. Long, multi-part prompts appear harder for the agent to fully satisfy.
• Narrow, verifiable scope wins: the highest success rate (90.4%) is in the "Other" bucket — prompts that don't match common action-verb keywords, largely automated/bot-triggered tasks (CI failure fixes, dependency bumps, schema checks) with a single clear, checkable target. Refactoring prompts have the lowest rate (52.6%), consistent with vaguer, broader scope.
• CI-log-driven and dependency-bump prompts perform best: prompts that cite a specific run ID/job URL or a specific dependency+version tend to merge, since the fix is mechanically verifiable.
• Quoting a raw GitHub issue verbatim underperforms: prompts that paste <issue_title>/original issue text rather than restating a focused, actionable task show up disproportionately in the closed set.

Recommendations

1. DO: Keep prompts short and single-purpose (~100 words); state one clear, verifiable goal (e.g., a specific failing check, file, or dependency).
2. DO: Include concrete pointers — run IDs, job URLs, file paths, error messages — so the agent can verify its own fix.
3. AVOID: Pasting entire original issue bodies or bundling multiple unrelated asks into one prompt; distill them into a focused instruction instead.

Historical Trends

Date	PRs	Success Rate	Top Category
2026-08-19	1000	78.3%	bug_fix
2026-07-06	1000	80.4%	test
2026-07-05	1000	80.8%	bug_fix
2026-07-04	1000	81.1%	bug_fix
2026-07-03	1000	81.8%	bug_fix

Trend: Success rate has declined gradually from ~82% (early July) to ~78% today, a ~4 point drop over the observed period (note the gap between the July and August data points). Bug fixes remain the dominant category driving PR volume.

---

Generated by Copilot PR Prompt Analysis (Run: 32300344094)

Generated by 🔍 Copilot PR Prompt Pattern Analysis · auto · 22.3 AIC · ⌖ 4.69 AIC · ⊞ 11.1K · ◷

• expires on Aug 20, 2026, 12:58 PM UTC-08:00