Exploring Multi-Engine Enhancements for Actions Migrations via Copilot

[dhruvg20]

Hey team! 👋

First off — thank you to everyone who built and contributed to actions-migrations-via-copilot. It's a well-designed solution — the KB-driven agents, IssueOps batch orchestration, 5-phase process, and automation scripts work together really well. We've been using it as the foundation for customer migration programs and it's been serving us well.

With that as our starting point, we've been exploring enhancements that could extend the solution's reach into scenarios where the Copilot Coding Agent alone has limitations. We wanted to share our thinking early — before opening a formal PR — to get the team's perspective and input.

Nothing here replaces the existing solution. We're building on top of it, and the Coding Agent remains the default. We'd love your feedback on the direction.

---

TL;DR

We're exploring adding two complementary engines — the Copilot SDK and Copilot CLI — alongside the existing Copilot Coding Agent. All three would share the same agent prompts, knowledge base, and 5-phase migration process. The idea is to cover deployment scenarios (like GHES) and use cases (like ad-hoc developer migrations) where the Coding Agent can't reach today.

We're actively working on this in a fork and plan to open a PR to the main repo once we've gathered feedback and reached a solid state. This discussion is the first step.

---

What Prompted This?

We exploring this solution path for SAP which is GHES environment with Copilot Enabled on their IDE through a GHEC-EMU setup.

The existing solution works great for GHEC-based enterprise batch migrations. Through customer engagements, though, we've run into a few recurring scenarios where we needed more flexibility:

1. GHES Customers

The Coding Agent runs on github.com. Customers on GitHub Enterprise Server (GHES) — especially those with large on-prem Jenkins estates — can't use it directly. We've been looking at ways to run migrations inside the customer's network while still leveraging Copilot's LLM capabilities.

2. Quality Control

With the Coding Agent, the LLM decides when to fetch KB documents, run validation, and handle errors. In some engagements, we've wanted a way to programmatically enforce quality gates — ensuring actionlint actually ran, capturing its real output, and handling failures explicitly.

3. Model Flexibility

Being able to pin a specific model for reproducibility, or pick a model suited for a particular task (e.g., a reasoning model for complex Groovy expansion), has come up as a need.

4. Complex Pipeline Handling

Platforms like Jenkins — with shared libraries, Groovy DSLs, and multi-type pipelines — can benefit from custom tools and lifecycle hooks for more controlled processing.

5. Quick Developer Migrations

Not every migration needs a full IssueOps workflow. Engineers sometimes want to test a migration from their terminal in a couple of minutes.

---

What We're Exploring

A three-engine architecture that builds on the existing solution's foundation:

┌─────────────────────────────────────────────────────┐
│              Multi-Engine Migration Platform          │
│                                                       │
│  ┌──────────────┐  ┌──────────────┐  ┌────────────┐ │
│  │ Coding Agent │  │  Copilot SDK │  │ Copilot CLI│ │
│  │  (GitHub.com)│  │(Programmatic)│  │(Lightweight)│ │
│  └──────┬───────┘  └──────┬───────┘  └─────┬──────┘ │
│         │                 │                 │         │
│         └─────────────────┴─────────────────┘         │
│                           │                           │
│              ┌────────────┴────────────┐              │
│              │    Shared Foundation     │              │
│              │  • 9 Agent Prompts      │              │
│              │  • Knowledge Base (29+) │              │
│              │  • 5-Phase Process      │              │
│              │  • actionlint Validation│              │
│              └─────────────────────────┘              │
└─────────────────────────────────────────────────────┘

Engine Summary

Engine	Where It Runs	How It's Triggered	Best For
Coding Agent (existing)	GitHub.com cloud	Issue assignment via IssueOps	Enterprise batch migrations (100+ repos)
Copilot SDK (new)	Self-hosted runners, local machines	Python script or workflow_dispatch	GHES environments, complex pipelines, BYOK
Copilot CLI (new)	Any machine with gh CLI	Shell script	Ad-hoc single-repo migrations, prototyping

What's Shared Across All Engines

• Agent prompts — The same 9 markdown agent definitions (one per CI platform + reusable workflow builder)
• Knowledge base — 29+ documents: syntax mappings, migration patterns, security guidelines, report templates
• 5-phase process — Analyze → Convert → Validate → Document → PR
• Supported platforms — Jenkins, CircleCI, GitLab, Azure DevOps, Travis CI, Bamboo, Bitbucket, Drone CI

---

What Would Each Engine Bring?

Copilot Coding Agent (Existing — Keep as Default)

• Autonomous: reads source files, fetches KB, generates workflows, opens PRs
• Batch orchestration via IssueOps
• No local infrastructure needed

Limitation: GHEC-only, no custom tools, no lifecycle hooks, no model selection.

Copilot SDK (New — Programmatic Engine)

• Custom tools: Define typed tools for KB fetching, validation, PR creation
• Lifecycle hooks: Audit tool calls, capture errors, control retry/skip/abort behavior
• Model selection: Choose which model to use per migration
• GHES support: Dual-token pattern — KB and PRs stay on GHES, only LLM API calls go outbound
• BYOK: Use your own Azure OpenAI, OpenAI, or Anthropic keys
• Validation: Programmatically ensure actionlint runs and results are captured

Copilot CLI (New — Lightweight Engine)

• Single command to run a migration
• Works on any machine with gh CLI installed
• Supports local knowledge base for offline/testing use

Limitation: Sequential, no batch support, no hooks, no automated PR creation.

---

How Does Engine Routing Work?

A unified config.yaml routes migrations to the right engine with priority-based resolution:

migration_engine:
  default: "coding-agent"    # Fallback for everything
  
  per_type:                  # Override by CI platform
    "Jenkins": "copilot-sdk"       # Complex shared libs → SDK
    "GitLab": "copilot-cli"        # Simple → CLI
    "CircleCI": "coding-agent"     # Standard → Coding Agent
    
  per_org:                   # Override by organization (highest priority)
    "ghes-org": "copilot-sdk"      # GHES → must use SDK

Resolution priority: Per-Org > Per-Type > Default

This is fully backward-compatible. If you don't add the migration_engine section, everything uses the Coding Agent exactly as before.

---

Challenges We're Thinking Through

Challenge	Mitigation
SDK is in Technical Preview	Pin versions, monitor releases, maintain backward-compatible config
SDK requires infrastructure setup	Pre-built Docker image available; also works on host-based runners
Batch orchestration for SDK/CLI	Extend existing submit-repositories.js with engine-aware routing (dispatch to SDK/CLI workflows)
Feature parity with CCA	Tracked via gap matrix — 12 critical/high gaps identified, working through them
GHES MCP restrictions	SDK custom tools work independently of MCP, so GHES limitations don't apply
Auth complexity across engines	Tiered access strategy: MCP → GitHub App → PAT fallback, documented per environment
Knowledge base fetch reliability	SDK uses a typed tool with fixed paths instead of relying on the LLM to construct them

---

Our plan: Once we've incorporated feedback from this discussion and stabilized the implementation, we'll open a PR against the main repo with the proposed enhancements. We want to make sure the direction makes sense to the broader team first.

---

Questions for the Team

We'd really appreciate the team's perspective on a few things:

1. GHES Deployment Model

For GHES customers, should we recommend:

• (a) Pre-built Docker images (bake everything in, deploy on any runner infrastructure)
• (b) Dynamic copilot-setup-steps.yml (install at runtime on existing runners)
• (c) Both, with Docker as the recommended default?

2. Batch Orchestration Strategy

The Coding Agent uses IssueOps for batch. For SDK, we're considering workflow_dispatch triggers in target repos. For CLI, batch doesn't apply (sequential only). Is this the right paradigm split, or should we unify batch orchestration differently?

3. Knowledge Base Access Pattern

Currently:

• CCA: MCP auto-fetch (GitHub MCP Server)
• SDK: Custom fetch_knowledge_base tool (GitHub API via gh api)
• CLI: MCP via --enable-all-github-mcp-tools

Should we standardize on one pattern, or keep engine-specific access methods?

---

What Happens Next

1. This discussion — gather feedback, questions, and concerns from the team
2. Incorporate input — adjust the approach based on what we hear
3. Open a PR to the main repo with the proposed enhancements
4. Pilot — test with 1-2 customer engagements (GHES + GHEC mix)
5. Measure — compare migration quality across engines using the same source files

---

Thanks again for actions-migrations-via-copilot — it's a solid foundation and we want to make sure any enhancements align with the team's direction.

Please drop your thoughts, questions, or concerns in the comments. Even a quick 👍 or 👎 on the overall direction helps. We'll incorporate feedback before opening the PR.

👀 @antgrutta @robandpdx

[antgrutta]

@dhruvg20, I'm curious who the "we" is in "we've been exploring enhancements." @ssulei7 and I (@antgrutta) have been fighting an uphill battle for quite a while to get this adopted, so I'd love to know who else is investing this much time but hasn't found the actions-migrations-via-copilot Slack channel.

There is a lot happening in the background. Certain technical decisions are being made based on other pending platform features, and we are hoping to have everything lined up for the ES offsite, where we plan to do a mass enablement and host a hackathon focused on the technical changes needed for the full release of the new Actions Migration Offering which centers on using this tooling.

Much of what you mentioned has already been considered. There are plans to move toward agentic workflows, use skills to replace MCP, and add compatibility with GHES via Copilot CLI based workflows. Some of what you described is actually already covered by the system today. There are also customer/developer experience improvemetents that need to be taken into account as we make decisions about implementing techncial changes.

That said, while I appreciate the enthusiasm, there are a number of points in your post that I think need to be addressed. It feels like the AI may have gotten a little ahead of itself here. If you're genuine about contributing, here's what needs to happen next:

• Please have your team join us in the actions-migrations-via-copilot Slack channel.
• Let's break down the specific enhancements you're interested in and see how they align with our current roadmap and technical plans.
• We can set up a meeting to discuss your ideas in more detail and give you a better picture of the work happening in the background that may not be visible yet.

[dhruvg20]

Hey @antgrutta — thanks for the honest reply, really helpful to hear what's going on behind the scenes. A few things from my side:

1. Future Features — is there a board or issue list somewhere that tracks upcoming work for this? Would love to see what's already planned so we're not stepping on anything or redoing stuff that's already in flight.
2. Slack — on it! Will join the channel and read through what's there before saying anything more. Makes sense to catch up first.
3. The "we" — fair question it's me and a Partner engineer — we're supporting SAP on their migration (GHES setup), which is where a lot of this exploration started. Quick one though — any guidance on what's okay to share with customers directly? Just want to make sure we're not getting ahead of ourselves on the external side.
4. KB & guardrails — this one's been on my mind. Are there plans to keep growing the knowledge base? And if a customer ends up running this on their own, is there anything that spells out "you can extend these parts" vs. "don't touch these without thinking twice"? Feels like the migration standards, validation rules, security patterns etc. need that kind of clarity — especially if it's being handed off.

FYI - SAP has also build a Jenkins based solution with internally hosted LLM model, but they seems to be doing it as side project, and they did share it. Seems like they doing a Q/A with LLM, compared to full Agentic flow, so they were really impressed with our solution, and wanted to get try it to get feedback to us. Their main challenge is GHES, and not having access to CCA, at the moment. They have massive set of Jenkins Pipeline ~50,000 pipelines, so migration at scale, but keeping it phased and they want their end teams to do these migrations by themselves, would be a great way to hear back feedback and enhance customer experience. Also I am expecting visibility from them on cost it will take for them, as in if the PRU consumption rate crosses a threshold of Enterprise/Business seats they have, and making it a phased migration to be done by them/their teams.

Happy to hop on a call whenever works, just being in opposite timezones IST here.
Looking forward to the Offsite hackathon being planned.