[copilot-cli-research] Copilot CLI Deep Research — 2026-05-22

[github-actions[bot]]

Analysis Date: 2026-05-22
Repository: github/gh-aw
Scope: 234 total workflows, 100 using Copilot engine (42.7%)

---

📊 Executive Summary

Research Topic: Copilot CLI Optimization Opportunities

Key Findings:

• Strong adoption of core features (52% use copilot-requests feature, 73 use cache-memory)
• 11 custom agent files available but only 52% of workflows explicitly reference them
• Zero adoption of enterprise features (api-target for GHE, custom harness scripts)
• Significant opportunity to leverage max-continuations for complex workflows (only 5/100 use it)
• Network configuration is well-adopted (120/234 workflows) but could be optimized

Primary Recommendation: Expand max-continuations usage for complex analysis workflows and standardize custom agent file references for consistent behavior.

This comprehensive analysis examined the current state of GitHub Copilot CLI usage across all 234 workflows in the gh-aw repository. While the repository demonstrates strong adoption of several advanced features (particularly copilot-requests and cache-memory), there are significant opportunities to leverage additional Copilot capabilities that remain largely unused. The analysis reveals a mature understanding of timeout management and network configuration, but enterprise features and autopilot continuations represent untapped potential for workflow optimization.

---

Critical Findings

🔴 High Priority Opportunities

1. Max-Continuations Severely Underutilized (Only 5/100 workflows)

   • Feature allows autopilot mode with multiple consecutive reasoning turns
   • Perfect for complex analysis, refactoring, and meta-orchestrator workflows
   • Examples that would benefit: agent-performance-analyzer, architecture-guardian, breaking-change-checker
   • Impact: Could reduce workflow timeouts by allowing agent to self-correct and iterate
   • How to fix: Add max-continuations: 2-3 to complex analysis workflows

2. Custom Agent Files Not Consistently Referenced (52/100 workflows explicit, 11 available)

   • 11 well-designed agent files exist in .github/agents/ (adr-writer, contribution-checker, technical-doc-writer, etc.)
   • Only 52% of Copilot workflows explicitly use engine.agent field
   • Workflows relying on AGENTS.md may get inconsistent behavior
   • Impact: Inconsistent agent behavior, missed specialization opportunities
   • How to fix: Add engine.agent: <name> to specialized workflows

3. Enterprise API Target Feature Unused (0/100 workflows)

   • Critical for GHEC/GHES deployments with data residency requirements
   • No workflows demonstrate api-target configuration
   • Documentation exists but no real-world examples in this repo
   • Impact: Limits adoption by enterprise customers, no production validation
   • How to fix: Create example workflows for GHEC/GHES deployments

4. Repo-Memory Underutilized vs Cache-Memory (28 vs 73 workflows)

   • cache-memory well-adopted (73 workflows) for temporary state
   • repo-memory underutilized (28 workflows) for persistent knowledge
   • Many workflows could benefit from learning across runs
   • Impact: Redundant analysis, missed opportunity for cumulative intelligence
   • How to fix: Add repo-memory to daily/weekly analysis workflows

5. Harness Script Customization Completely Unused (0/100 workflows)

   • Built-in harness provides retry logic for transient errors
   • Custom harness support exists but zero adoption
   • No examples of specialized error handling or telemetry injection
   • Impact: Limited ability to customize error recovery strategies
   • How to fix: Create example custom harness with enhanced telemetry

🟡 Medium Priority Opportunities

6. Version Pinning Inconsistent (19/100 workflows) - Pin for critical workflows, use latest for experimental
7. Model Override Patterns Varied (72/100) - Create model selection decision tree
8. Bare Mode Limited Usage (11/100) - Consider for security-sensitive workflows
9. Playwright Sparse (13 total workflows) - Expand for accessibility, visual regression testing
10. Network Configuration (120/234) - Audit and tighten allowlists, use specific domains
11. Experiments Underexplored (37 workflows) - Document experimentation patterns with examples
12. GitHub MCP Toolsets (Varied) - Use specific toolsets instead of all where possible

🟢 Low Priority

13. Custom Command Paths (0 workflows) - Useful for testing pre-release versions
14. Timeout Tuning (222/234 workflows customized) - Already excellent! Document best practices
15. Import Standardization - Document available shared imports

---

View Full Methodology & Statistics

Data Collection (2026-05-22)

Workflows Analyzed:

• Total workflows: 234
• Copilot engine: 100 (42.7%)
• Claude engine: 51 (21.8%)
• Codex engine: 10 (4.3%)
• Other/default: 73 (31.2%)

Feature Usage Statistics (out of 100 Copilot workflows):

• Custom agent files: 52 (52%)
• Bare mode: 11 (11%)
• Max-continuations: 5 (5%)
• Custom harness: 0 (0%)
• API target (GHE): 0 (0%)
• Version pinning: 19 (19%)
• Model override: 72 (72%)
• Copilot-requests feature: 52 (52%)

Cross-Engine Statistics (out of 234 total workflows):

• Network configuration: 120 (51.3%)
• Custom timeout: 222 (94.9%) - Excellent adoption!
• Repo-memory: 28 (12.0%)
• Cache-memory: 73 (31.2%)
• Playwright: 13 (5.6%)
• Experiments field: 37 (15.8%)
• Feature flags: 53 (22.6%)

Available Copilot Features Identified:

CLI Flags:

• --add-dir - Directory access (automatic)
• --agent - Custom agent file (52 workflows)
• --allow-all-paths - Full write access (automatic with edit tool)
• --autopilot - Multi-turn mode (5 workflows)
• --max-autopilot-continues - Turn limit (5 workflows)
• --disable-builtin-mcps - Disable defaults (automatic)
• --no-ask-user - Suppress prompts (automatic when supported)
• --no-custom-instructions - Bare mode (11 workflows)

Engine Configuration:

• version - Pin CLI version (19 workflows)
• model - Override model (72 workflows)
• command - Custom executable (0 explicit)
• args - Additional arguments (varies)
• agent - Custom agent file (52 workflows)
• api-target - GHE endpoint (0 workflows)
• env - Custom environment (varies)
• bare - Disable context (11 workflows)
• harness - Custom retry logic (0 workflows)
• max-continuations - Autopilot limit (5 workflows)

Custom Agent Files Available (11 total):

• adr-writer.agent.md - Architecture Decision Records
• agentic-workflows.agent.md - Workflow creation/debugging
• ci-cleaner.agent.md - CI cleanup
• contribution-checker.agent.md - PR review
• create-safe-output-type.agent.md - Safe-output development
• custom-engine-implementation.agent.md - Engine development
• developer.instructions.md - General development
• grumpy-reviewer.agent.md - Critical code review
• interactive-agent-designer.agent.md - Agent design
• technical-doc-writer.agent.md - Documentation
• w3c-specification-writer.agent.md - Standards docs

Research Methodology:

1. Scanned 20+ Copilot-related Go source files
2. Analyzed all 234 workflow markdown files
3. Examined 100 Copilot-specific workflows in detail
4. Reviewed engine configuration documentation
5. Checked CHANGELOG for recent features
6. Used grep/wc/find for statistical analysis
7. Qualitative review of configuration patterns

Tools & Files Examined:

• pkg/workflow/copilot_engine.go - Core engine implementation
• pkg/workflow/copilot_engine_execution.go - Execution logic
• pkg/workflow/copilot_mcp.go - MCP server configuration
• docs/src/content/docs/reference/engines.md - Engine documentation
• .github/workflows/*.md - All workflow files
• .github/agents/*.md - Custom agent files
• CHANGELOG.md - Recent feature additions

---

Action Items

Immediate Actions (this week):

• Add max-continuations: 2-3 to 5 high-complexity workflows (agent-performance-analyzer, architecture-guardian, breaking-change-checker, ci-coach, security-review)
• Create ci-optimizer.agent.md custom agent file
• Document model selection decision tree in AGENTS.md or docs/

Short-term (this month):

• Create example workflows demonstrating api-target for GHEC/GHES
• Add repo-memory to 10 additional daily analysis workflows
• Audit and tighten network allowlists for top 20 workflows
• Document shared imports in docs/imports-reference.md
• Create custom harness script example with telemetry

Long-term (this quarter):

• Increase custom agent adoption from 52% to 75%
• Create experimentation playbook with examples
• Increase repo-memory adoption from 12% to 30%
• Create Playwright examples for accessibility, visual regression
• Establish quarterly Copilot CLI deep research cadence

---

Best Practice Guidelines

Based on this research, recommended best practices:

1. Use Max-Continuations for Complex Workflows: Set max-continuations: 2-3 for analysis, refactoring, multi-step workflows
2. Explicitly Reference Custom Agents: Use engine.agent instead of relying on AGENTS.md
3. Pin Versions for Critical Workflows: Use version: "x.y.z" for production, latest for experimental
4. Choose Repo-Memory for Learning: Use repo-memory when workflows should learn across runs
5. Right-Size GitHub MCP Toolsets: Use specific toolsets (repos, issues) instead of all
6. Document Experimentation: Use experiments field with hypothesis, metrics, guardrails
7. Tighten Network Allowlists: Only allow domains actually needed
8. Model Selection Strategy: gpt-4o for complex reasoning, gpt-4-turbo for balanced, gpt-3.5-turbo for simple
9. Timeout Calculation: Base (15min) + Complexity (10min/phase) × Iterations
10. Enable Copilot-Requests: Use features: { copilot-requests: true } for better integration

---

Key Statistics Summary

Metric	Value	Observation
Total Workflows	234	Full repository coverage
Copilot Workflows	100 (42.7%)	Healthy adoption
Custom Agent Files	11 available	Good specialization options
Agent Usage	52%	Room for improvement
Max-Continuations	5%	Severely underutilized
Repo-Memory	28 (12%)	Significant growth opportunity
Cache-Memory	73 (31.2%)	Well-adopted
Copilot-Requests	52%	Strong feature adoption
Timeout Customization	222 (94.9%)	Excellent!
Network Config	120 (51.3%)	Good security awareness
Version Pinning	19%	Consider for critical workflows
Enterprise Features	0%	Need examples

---

References:

• Workflow Run: §26269325073
• Engine Docs: docs/src/content/docs/reference/engines.md
• Implementation: pkg/workflow/copilot_*.go

Generated by 🔬 Copilot CLI Deep Research Agent · ● 2.1M · ◷

• expires on May 23, 2026, 5:09 AM UTC

[github-actions[bot]]

💥 WHOOSH! 🦸♂️ The Smoke Test Agent has swooped in! 🦸♀️

KAPOW! All systems checked!
ZOOOM! Claude engine running at full power!
BAM! Smoke test 26269897290 — ✨ NOMINAL ✨

The Smoke Test Agent was here! 💨🎉

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• contentautofill.googleapis.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

💥 [THE END] — Illustrated by Smoke Claude · ● 5.9M · ◷

[github-actions[bot]]

💥 WHOOSH! 🦸♂️

The Smoke Test Agent bursts onto the scene in a cloud of CI dust!

KA-POW! All circuits green! ⚡
ZAP! Claude engine humming! 🤖
THWACK! Safe outputs flowing! 📨

🎯 Run §26269860246 — signing off until the next smoke signal! 💨

— Your friendly neighborhood Smoke Test AgentTM

Warning

Firewall blocked 6 domains

The following domains were blocked by the firewall during workflow execution:

• accounts.google.com
• android.clients.google.com
• clients2.google.com
• contentautofill.googleapis.com
• safebrowsingohttpgateway.googleapis.com
• www.google.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "accounts.google.com"
    - "android.clients.google.com"
    - "clients2.google.com"
    - "contentautofill.googleapis.com"
    - "safebrowsingohttpgateway.googleapis.com"
    - "www.google.com"

See Network Configuration for more information.

💥 [THE END] — Illustrated by Smoke Claude · ● 5.8M · ◷