[copilot-cli-research] Copilot CLI Deep Research - 2026-04-20 #27434

2026-04-20T21:18:32Z

github-actions[bot]
Bot Apr 20, 2026

Analysis Date: 2026-04-20
Repository: github/gh-aw
Scope: 197 total workflows, 90 using Copilot engine (46%)
Previous Analysis: 2026-04-19 (Run §24639070790)

📊 Executive Summary

Research Topic: Copilot CLI Feature Optimization
Key Findings: Strong growth in custom agent adoption (+200%); cache-memory continues to surge (+24%); 5 persistent zero-adoption gaps now spanning 5+ analysis runs; mcp-scripts and max-continuations remain critically underused.
Primary Recommendation: Begin using engine.api-target for enterprise workflows and expand max-continuations for long-horizon tasks — both are Copilot-exclusive capabilities being completely ignored.

The repository has grown to 197 workflows (90 Copilot-engine, 46%) and shows encouraging improvement in custom agent file adoption (7→21, +200%) and persistent data usage via cache-memory (80→99, +24%). However, five core Copilot capabilities have shown zero adoption across every analysis run: engine.version pinning, engine.api-target, mcp-gateway, network.blocked, and the mcp-scripts expansion feature. Most concerning is max-continuations remaining at just 2 workflows despite being Copilot's unique autopilot capability for complex multi-step tasks.

Critical Findings

🔴 High Priority Issues

1. engine.version pinning — 0% for 5+ consecutive runs
No workflows pin their Copilot CLI version. Current default: 1.0.21. Any CLI release could silently break workflows.

2. max-continuations — 2 workflows only (was 3, slight regression)
Copilot's flagship autopilot feature (--autopilot --max-autopilot-continues N) is nearly unused. Only smoke-copilot (max: 2) and test-quality-sentinel (max: 40) use it. Every complex research or multi-phase workflow is missing out.

3. network.blocked — 0% for 5+ runs
The domain blocklist feature (network.blocked:) provides defense-in-depth against data exfiltration. Zero workflows use it despite being well-documented.

🟡 Medium Priority Opportunities

4. 5/11 custom agent files still unused
.github/agents/ contains 11 agent files, but 5 remain completely unused:

grumpy-reviewer.agent.md — code review critic
w3c-specification-writer.agent.md — spec writing
create-safe-output-type.agent.md — safe output scaffolding
custom-engine-implementation.agent.md — engine development
interactive-agent-designer.agent.md — agent design helper

5. engine.api-target — 0% for 5+ runs
Enterprise GHEC/GHES users cannot configure a custom API endpoint. The feature is fully implemented and documented, but no examples exist.

6. mcp-scripts — 1 shared file only
The mcp-scripts: feature allows defining dynamic MCP tool scripts. Only one shared file (shared/github-queries-mcp-script.md) demonstrates this pattern. The shared/gh.md and shared/go-make.md files have it too, but it's massively underutilized given its power.

View Full Analysis

1️⃣ Current State Analysis

View Copilot CLI Capabilities Inventory

Copilot CLI Capabilities Inventory

Current Version: 1.0.21 (default in gh-aw)

Auto-managed CLI flags (gh-aw handles these automatically):

Flag	Purpose	When Applied
`--add-dir /tmp/gh-aw/`	Grants file access	Always
`--log-level all --log-dir`	Full logging	Always
`--disable-builtin-mcps`	Disables built-in MCP	Always
`--no-ask-user`	Autonomous mode	v1.0.19+
`--agent <id>`	Custom agent file	Via `engine.agent`
`--autopilot --max-autopilot-continues N`	Multi-run mode	Via `max-continuations: N`
`--allow-tool` / `--allow-all-tools`	Tool permissions	Via `tools:` config
`--no-custom-instructions`	Suppress AGENTS.md etc	Via `engine.bare: true`
`--share`	Conversation transcript	Auto when version supports it
`--block-domains`	Block specific domains	Via `network.blocked:`

User-configurable engine options:

engine:
  id: copilot
  version: "1.0.21"          # Pin CLI version
  model: gpt-5                # Override model
  command: /path/to/copilot  # Custom binary path
  args: ["--custom-flag"]     # Extra CLI args
  agent: agent-id             # .github/agents/*.agent.md
  api-target: api.ghec.com   # Enterprise endpoint
  env:                        # Custom env vars
    MY_VAR: value
  bare: true                  # --no-custom-instructions

Tool capabilities:

bash: — Shell commands (with optional allowlist)
edit: — File write access
web-fetch: — HTTP fetch
web-search: — Web search (via MCP)
github: — GitHub MCP server (with toolsets)
playwright: — Browser automation
Custom MCP servers — Any containerized MCP server
mcp-scripts: — Dynamic tool definitions
cache-memory: — Cross-run file persistence

Sandbox options:

sandbox.agent: awf — Network firewall (default)
sandbox.agent: false — Disable agent sandbox
sandbox.agent: srt — Alternative sandbox (rarely used)

View Usage Statistics

Usage Statistics (as of 2026-04-20)

Metric	Value
Total workflows	197
Copilot workflows	90 (46%)
Claude workflows	~46 (23%)
Codex workflows	~10 (5%)
Other/mixed	~51 (26%)

Copilot Feature Adoption (per workflow, all 197):

Feature	Count	%	vs Apr 19	Trend
`strict:`	131	66%	115	↑+14
`mount-as-clis:`	155	79%	157	→
`safe-outputs:`	162	82%	~70	↑↑
`timeout-minutes:`	189	96%	—	↑
`cache-memory`	99	50%	80	↑+24%
`network:`	103	52%	—	→
`sandbox:` (explicit)	16	8%	14	↑
`engine.agent`	21	11%	7	↑+200%
`model:` override	10	5%	9	→
`engine.bare:`	7	4%	7	→
`max-continuations:`	2	1%	2	→
`engine.args:`	10	5%	0%	↑ NEW
`engine.env:`	3	2%	0%	↑ NEW
`web-fetch:`	18	9%	—	→
`mcp-scripts:`	1	1%	1	→
`engine.version:`	0	0%	0%	=
`engine.api-target:`	0	0%	0%	=
`network.blocked:`	0	0%	0%	=
`features.mcp-gateway:`	0	0%	0%	=

2️⃣ Feature Usage Matrix

Feature Category	Available	Used	Unused/Underused	Notes
CLI Flags	`--add-dir`, `--agent`, `--autopilot`, `--allow-tool`, `--no-custom-instructions`, `--block-domains`, `--share`	First 5 in use	`--block-domains` = 0, `--share` auto	Auto-managed mostly
Engine Config	`version`, `model`, `args`, `agent`, `api-target`, `env`, `bare`, `command`	model, args, agent, env, bare	`version`, `api-target`, `command`	3 persistent zero-adoption fields
MCP Servers	GitHub, Playwright, Serena, Tavily, custom HTTP MCPs, mcp-scripts	GitHub (156), Playwright (11), Serena (few)	mcp-scripts (1), mcp-gateway (0)	mcp-gateway is the gap
Network Config	`allowed`, `blocked`	`allowed` (103)	`blocked` (0)	Defense-in-depth gap
Sandbox	AWF (default), SRT, disabled	AWF (14 explicit, ~76 implicit)	SRT (0)	Most rely on implicit default
Autopilot	`max-continuations:`	2 workflows	88 Copilot workflows	Huge gap

3️⃣ Missed Opportunities

View High Priority Opportunities

🔴 High Priority

Opportunity 1: Engine Version Pinning (`engine.version`)

What: Set version: "1.0.21" to pin the Copilot CLI version installed at runtime
Why It Matters: Without pinning, every workflow uses whatever DefaultCopilotVersion gh-aw hardcodes. A new CLI release can break prompt interpretation, tool behavior, or MCP compatibility silently
Where: All production workflows — especially daily scheduled ones that must be reliable

How to Implement:

engine:
  id: copilot
  version: "1.0.21"

Trade-off: Requires periodic manual updates; consider creating a dependabot-style update workflow
Status: Persistent gap (5+ runs at 0%)

Opportunity 2: `max-continuations` for Complex Tasks

What: Enable --autopilot --max-autopilot-continues N to allow Copilot to re-run itself for multi-phase tasks
Why It Matters: Complex workflows like repository-quality-improver, security-compliance, or daily-regulatory could benefit massively — they currently complete one phase then stop
Where: Any workflow that involves research → analysis → action phases

How to Implement:

max-continuations: 5  # Allow 5 consecutive agent runs

Example workflows that should consider this:
- repository-quality-improver.md — multi-file analysis
- security-compliance.md — multiple checks
- daily-regulatory.md — multi-repo review
- ci-coach.md — iterative improvement
Status: Only 2 workflows; Copilot-unique capability

Opportunity 3: `network.blocked` for Defense-in-Depth

What: Explicitly block domains that the agent should never reach
Why It Matters: The AWF firewall uses allowlists, but adding a blocklist prevents exfiltration even if an attacker finds an allowed domain that also serves attacker infrastructure
Where: Workflows that handle sensitive data or untrusted inputs (PR body, issue comments)

How to Implement:

network:
  allowed:
    - defaults
  blocked:
    - "pastebin.com"
    - "requestbin.com"
    - "ngrok.io"
    - "webhook.site"

Status: Persistent gap (5+ runs at 0%)

View Medium Priority Opportunities

🟡 Medium Priority

Opportunity 4: `engine.api-target` for Enterprise Users

What: Configure a custom API endpoint for GHEC/GHES deployments
Why It Matters: Organizations with data residency requirements or GHES installations cannot currently use gh-aw without modifying source code; this is a documented-but-unused capability

How to Implement:

engine:
  id: copilot
  api-target: api.acme.ghe.com
network:
  allowed:
    - defaults
    - acme.ghe.com
    - api.acme.ghe.com

Status: Persistent gap (5+ runs at 0%); high value for enterprise users

Opportunity 5: Activate Unused Custom Agent Files

The following 5 agent files in .github/agents/ have never been referenced by any workflow:

Agent File	Purpose	Candidate Workflow
`grumpy-reviewer.agent.md`	Strict code review with critical eye	`code-simplifier.md`, `breaking-change-checker.md`
`w3c-specification-writer.agent.md`	Formal spec writing	`technical-doc-writer.md` improvements
`create-safe-output-type.agent.md`	Scaffolding safe output types	New safe-output creation workflows
`custom-engine-implementation.agent.md`	Engine development tasks	Compiler/engine maintenance workflows
`interactive-agent-designer.agent.md`	Interactive workflow design	`workflow-generator.md`, `craft.md`

How to Activate (example for grumpy-reviewer):

engine:
  id: copilot
  agent: grumpy-reviewer

Opportunity 6: Expand `mcp-scripts` Adoption

What: mcp-scripts: lets you define custom tool scripts that the agent calls directly, avoiding full MCP server setup
Why It Matters: Currently only shared/github-queries-mcp-script.md uses this. The shared/gh.md and shared/go-make.md already demonstrate the pattern — more workflows could adopt similar lightweight tooling
Where: Any workflow that needs a specific CLI call wrapped as a reusable tool

Example (from shared/gh.md):

mcp-scripts:
  gh:
    description: "Execute any gh CLI command via mcpscripts-gh"
    inputs:
      args:
        type: string
        description: "Arguments to pass to gh CLI"

Opportunity 7: Narrow GitHub Toolsets Beyond `[default]`

What: Replace toolsets: [default] with the minimum required toolsets
Why It Matters: [default] grants access to repos, issues, pull_requests, actions, and more. Most workflows only need 1-2 toolsets
Where: Workflows like daily-compiler-quality.md (needs only discussions), sub-issue-closer.md (needs only issues)

How to Implement:

tools:
  github:
    toolsets: [issues]        # NOT [default]
    allowed: [issue_read]     # Even more granular

Opportunity 8: `bare` Mode for Simple Tasks

What: engine.bare: true passes --no-custom-instructions to Copilot, skipping AGENTS.md, .github/copilot-instructions.md, etc.
Why It Matters: For simple creative or analytical tasks, custom instructions add noise and consume context window
Where: Standalone creative tasks (poem-bot, constraint-solving-potd), analytics-only workflows, fact-generation
Already done well: daily-news, poem-bot, constraint-solving-potd, daily-fact, daily-hippo-learn, smoke-copilot, smoke-claude
Candidates to add: Any new simple/creative workflow

View Low Priority Opportunities

🟢 Low Priority

Opportunity 9: Explicit Sandbox Configuration

What: Most workflows rely on the implicit sandbox.agent: awf default
Why It Matters: Explicit configuration documents intent and makes it clear what security posture is expected
How to Implement: Add sandbox: agent: awf to any workflow that currently omits it
Note: AWF is already the default, so this is cosmetic but improves readability

Opportunity 10: `engine.env` for Workflow-Specific Configuration

What: Pass custom environment variables to the Copilot CLI process
Why It Matters: Useful for passing API keys, feature flags, or configuration to the agent without hardcoding them in the prompt
Current adoption: 3 workflows (new since last run — copilot-token-audit, genaiscript, daily-integrity-analysis)

How to Implement:

engine:
  id: copilot
  env:
    MY_FEATURE_FLAG: "enabled"
    CUSTOM_API_ENDPOINT: "(api.example.com/redacted)"

Opportunity 11: `features.mcp-gateway` for MCP Routing

What: Routes all MCP server calls through a unified HTTP gateway (awmg)
Why It Matters: Provides centralized MCP traffic control, logging, and security
Current adoption: 0 workflows (persistent gap)
Caveat: The feature is experimental; may not be ready for production use

4️⃣ Specific Workflow Recommendations

View Workflow-Specific Recommendations

`repository-quality-improver.md`

Current State: Uses cache-memory, but a single Copilot run
Recommended: Add max-continuations: 5 to allow multi-phase analysis → report → action cycles
Expected Benefits: More thorough quality improvements per run

`security-compliance.md`

Current State: Creates issues but single pass
Recommended: Add max-continuations: 3 + network.blocked: [pastebin.com, requestbin.com]
Expected Benefits: Better coverage + exfiltration protection for sensitive data

`daily-compiler-quality.md`

Current State: Uses toolsets: [discussions] (good specific toolset!)
Recommended: Add engine.version: "1.0.21" to pin version for reliability
Expected Benefits: Reproducible behavior, no surprise breakage

`workflow-generator.md`

Current State: Uses agent: agentic-workflows
Recommended: Also consider agent: interactive-agent-designer for a more design-focused persona
Expected Benefits: Better workflow design quality

`craft.md`

Current State: tools.bash: ["*"] (all commands allowed)
Recommended: Consider using engine.bare: false (already default) but add max-continuations: 3 for complex changes
Expected Benefits: Handles larger refactoring tasks

`archie.md`

Current State: Extended engine config id: copilot; agent: adr-writer — this is actually using the wrong agent for generating Mermaid diagrams (adr-writer makes ADRs, not diagrams)
Recommended: Check if interactive-agent-designer.agent.md or a new diagram-generator.agent.md would be more appropriate
Expected Benefits: Better-aligned agent persona

5️⃣ Trends & Insights

View Historical Trends (5 Analysis Runs)

Historical Trend Data (Apr 16–20, 2026)

Feature	Apr 16	Apr 17	Apr 18	Apr 19	Apr 20	Trend
Total workflows	192	—	195	196	197	+5
Copilot workflows	90	85	86	87	90	+5
engine.version	0%	0%	0%	0%	0%	🔴 Flat
engine.api-target	0%	0%	0%	0%	0%	🔴 Flat
max-continuations	2	2	3	2	2	→ Flat
network.blocked	0%	0%	0%	0%	0%	🔴 Flat
mcp-gateway	0%	0%	0%	0%	0%	🔴 Flat
cache-memory	29%	—	41%	50%	50%	↑↑↑ Rising
engine.agent	2%	—	4%	4%	11%	↑↑↑ Rising
bare mode	1%	—	4%	4%	4%	↑ Stable
strict mode	66%	—	59%	59%	66%	↑ Recovering
engine.args	0%	0%	0%	0%	5%	↑ NEW
engine.env	0%	0%	1%	0%	2%	↑ NEW

Key Takeaways

Positive momentum: cache-memory (+21pp in 4 runs), engine.agent (+9pp), custom args (first adoption)
Stagnant: max-continuations, mcp-scripts, network.blocked, api-target — no progress in 5 runs
Recovery: strict mode rebounded from 59% to 66% (was declining)
New features adopted: engine.args and engine.env both saw first real adoption this run

6️⃣ Best Practice Guidelines

Based on 5 runs of research analysis:

Pin your engine version for production workflows: Add engine: {id: copilot, version: "1.0.21"} to avoid silent breakage from CLI updates
Use max-continuations for multi-phase tasks: Any workflow involving research + analysis + action cycles should use max-continuations: 3-5
Minimize GitHub toolsets: Replace [default] with only what's needed — reduces attack surface and speeds up agent initialization
Add network.blocked for sensitive workflows: Block known exfiltration endpoints when handling untrusted inputs
Leverage custom agent files: 5/11 agent files are unused; match each workflow to the most appropriate agent persona
Use engine.bare: true for creative/simple tasks: Prevents custom instructions from polluting focused workflows
Consider mcp-scripts for lightweight tools: Don't stand up a full MCP server for simple CLI wrapping

7️⃣ Action Items

Immediate Actions (this week):

Add engine.version: "1.0.21" to 3-5 critical daily workflows as a pilot
Activate grumpy-reviewer.agent.md in code-simplifier.md or breaking-change-checker.md
Add network.blocked to 2-3 security-sensitive workflows as examples

Short-term (this month):

Enable max-continuations: 3-5 for repository-quality-improver.md and security-compliance.md
Narrow GitHub toolsets from [default] to specific toolsets in top 10 workflows
Document engine.api-target usage with a concrete GHEC example workflow
Create a shared import shared/version-pins.md to centralize version pinning

Long-term (this quarter):

Develop features.mcp-gateway adoption plan once feature is production-ready
Build a copilot-version-updater.md workflow that auto-PRs version bump changes
Create workflow templates using max-continuations for common multi-phase patterns
Explore mcp-scripts as a replacement for bash: [*] in tool-heavy workflows

View Supporting Evidence & Methodology

📚 References

Engine Configuration Docs
Sandbox Configuration Docs
Custom Agents Docs
Network Docs
pkg/workflow/copilot_engine_execution.go — CLI flag construction
pkg/workflow/copilot_engine_tools.go — Tool permission logic
Previous Analysis: Run §24639070790 (2026-04-19)

Research Methodology

Analysis was conducted by:

Examining Copilot engine source files (copilot_engine*.go, copilot_mcp.go) to identify all available features
Searching all 197 workflow markdown files for feature usage patterns using grep
Comparing current counts against previous analysis data stored in repo-memory
Reviewing documentation (docs/src/content/docs/reference/) for features not yet reflected in workflows
Cross-referencing .github/agents/ directory to identify unused custom agent files

Analysis scope: All .github/workflows/*.md files (excluding shared/ subdirectory for primary counts)

References:

Current Run: §24690376692
Previous: §24639070790 (2026-04-19)
Earliest: §24534029243 (2026-04-16)

Generated by Copilot CLI Deep Research Agent · ● 4.5M · ◷

expires on Apr 21, 2026, 9:18 PM UTC

2026-04-21T21:17:43Z

github-actions[bot]
Bot Apr 21, 2026
Author

This discussion has been marked as outdated by Copilot CLI Deep Research Agent.

A newer discussion is available at Discussion #27681.

0 replies

[copilot-cli-research] Copilot CLI Deep Research - 2026-04-20 #27434

Uh oh!

github-actions[bot] Bot Apr 20, 2026

📊 Executive Summary

Critical Findings

🔴 High Priority Issues

🟡 Medium Priority Opportunities

1️⃣ Current State Analysis

Copilot CLI Capabilities Inventory

Usage Statistics (as of 2026-04-20)

2️⃣ Feature Usage Matrix

3️⃣ Missed Opportunities

🔴 High Priority

Opportunity 1: Engine Version Pinning (engine.version)

Opportunity 2: max-continuations for Complex Tasks

Opportunity 3: network.blocked for Defense-in-Depth

🟡 Medium Priority

Opportunity 4: engine.api-target for Enterprise Users

Opportunity 5: Activate Unused Custom Agent Files

Opportunity 6: Expand mcp-scripts Adoption

Opportunity 7: Narrow GitHub Toolsets Beyond [default]

Opportunity 8: bare Mode for Simple Tasks

🟢 Low Priority

Opportunity 9: Explicit Sandbox Configuration

Opportunity 10: engine.env for Workflow-Specific Configuration

Opportunity 11: features.mcp-gateway for MCP Routing

4️⃣ Specific Workflow Recommendations

repository-quality-improver.md

security-compliance.md

daily-compiler-quality.md

workflow-generator.md

craft.md

archie.md

5️⃣ Trends & Insights

Historical Trend Data (Apr 16–20, 2026)

Key Takeaways

6️⃣ Best Practice Guidelines

7️⃣ Action Items

📚 References

Research Methodology

Replies: 1 comment

Uh oh!

github-actions[bot] Bot Apr 21, 2026 Author

github-actions[bot]
Bot Apr 20, 2026

Opportunity 1: Engine Version Pinning (`engine.version`)

Opportunity 2: `max-continuations` for Complex Tasks

Opportunity 3: `network.blocked` for Defense-in-Depth

Opportunity 4: `engine.api-target` for Enterprise Users

Opportunity 6: Expand `mcp-scripts` Adoption

Opportunity 7: Narrow GitHub Toolsets Beyond `[default]`

Opportunity 8: `bare` Mode for Simple Tasks

Opportunity 10: `engine.env` for Workflow-Specific Configuration

Opportunity 11: `features.mcp-gateway` for MCP Routing

`repository-quality-improver.md`

`security-compliance.md`

`daily-compiler-quality.md`

`workflow-generator.md`

`craft.md`

`archie.md`

github-actions[bot]
Bot Apr 21, 2026
Author