Copilot Studio Simulator

A local, scriptable version of Microsoft Copilot Studio built on the pi harness SDK. Define an agent declaratively (instructions, topics, tools, MCP servers, code execution, knowledge, embedded agents), chat with it, run eval suites, and auto-tune instructions/descriptions — no clicking through the Copilot Studio UI.

Important

Unofficial project. This is an independent, community-built simulator. It is not affiliated with, endorsed by, sponsored by, or supported by Microsoft Corporation. "Microsoft", "Copilot", and "Copilot Studio" are trademarks of the Microsoft group of companies and are used here only to describe compatibility/interoperability. This tool does not use any Microsoft code or proprietary assets — behavior is approximated from publicly observable behavior and documentation. For the real product, see Microsoft Copilot Studio.

Why

Iterating on a Copilot Studio agent through the web UI is slow: edit → publish → open test pane → eyeball → repeat. This simulator lets you:

• Version-control your agent (instructions, topics, tool descriptions are plain files).
• Test against the real dependencies — the same MCP servers, HTTP APIs, and scripts your production agent uses.
• Run eval suites in CI instead of eyeballing the test pane.
• Auto-tune instructions and descriptions with an optimizer loop until evals pass.
• Inspect everything afterwards — every run is a persisted pi session with full tool-call transcripts and token usage.
• Use subscription auth (GitHub Copilot, ChatGPT Plus/Pro, Claude Pro/Max) instead of juggling API keys.

Feature mapping

Copilot Studio	Simulator
Instructions	instructions.md → system prompt
Microsoft's hidden injection	preamble.md — tunable approximation (see below)
Topics (generative orchestration)	one topic__<name> tool per topic; triggering injects the topic playbook into context
Tools / actions / connectors	declarative tools (static, http, script)
MCP servers	bridged via @modelcontextprotocol/sdk (mcpServers: in agent.yaml)
Code execution	run_python sandbox tool
Knowledge sources	search_knowledge retrieval tool over local files (BM25-ish, cited chunks)
Embedded / child agents	delegate_to_<name> tools spawning linked child pi sessions
Test pane	pnpm chat <agent.yaml>
Evaluations	pnpm eval <agent.yaml> <suite.yaml> (deterministic checks + LLM judge)
— (no equivalent)	pnpm optimize — auto-tunes instructions & descriptions against evals
— (no equivalent)	pnpm inspect <session.jsonl> — full post-hoc transcript/usage dump

Prerequisites

• Node.js 20+
• The pi coding agent CLI (npm i -g @earendil-works/pi-coding-agent) — used for /login subscription auth and for inspecting sessions interactively (optional but recommended).
• python3 on PATH if you enable the code execution tool.

Quick start

npm install

# 1. Auth: subscription (no API keys)
pi  # then /login → GitHub Copilot (or ChatGPT Plus/Pro, Claude Pro/Max, xAI)
#    Tokens land in ~/.pi/agent/auth.json and the simulator picks them up.

# 2. Chat with the example agent (the "test pane")
pnpm chat examples/hr-agent/agent.yaml

# 3. Single prompt
pnpm run examples/hr-agent/agent.yaml "How many vacation days do I get?"

# 4. Run the eval suite (sessions + report saved under runs/<timestamp>/)
pnpm eval examples/hr-agent/agent.yaml examples/hr-agent/evals/baseline.yaml

# 5. Inspect any session afterwards
pnpm inspect runs/<timestamp>/sessions/<file>.jsonl

# 6. Auto-tune instructions/descriptions until evals pass
pnpm optimize examples/hr-agent/agent.yaml examples/hr-agent/evals/baseline.yaml \
  --optimizer github-copilot/gpt-4.1 --iterations 3

Model auth: subscriptions, not API keys

The simulator resolves models through pi's ModelRuntime, so every auth method pi supports works:

• GitHub Copilot subscription — pi → /login → GitHub Copilot, then model: "github-copilot/gpt-4.1" (as in the example).
• ChatGPT Plus/Pro, Claude Pro/Max, xAI — same /login flow.
• Azure OpenAI — AZURE_OPENAI_API_KEY + AZURE_OPENAI_BASE_URL, or Entra ID (keyless) via ~/.pi/agent/models.json with a command-resolved token:

  {
    "providers": {
      "my-foundry": {
        "baseUrl": "https://<resource>.openai.azure.com/openai/v1",
        "api": "openai-completions",
        "apiKey": "!az account get-access-token --query accessToken -o tsv --scope https://cognitiveservices.azure.com/.default",
        "models": [{ "id": "gpt-4o" }]
      }
    }
  }

Agent definition (agent.yaml)

See examples/hr-agent/agent.yaml for a fully commented example exercising topics, tools, code execution, and knowledge. Paths are relative to the config file. src/types.ts is the authoritative schema.

Evals

Suites are YAML (examples/hr-agent/evals/baseline.yaml). Each case supports multi-turn input and expectations:

• topic: — topic that must be triggered
• tools: — tools that must be called
• mustMention: / mustNotMention: — substring checks on the final answer
• rubric: — free-form grading criteria for the LLM judge (judge.model)

Every case runs in its own persisted pi session under runs/<ts>/sessions/, so you can pnpm inspect exactly what the agent did (tool calls, token usage, topic triggers, child-agent session links).

Auto-tuning (optimize)

Hill-climbing loop: run evals → hand the failure digest to an optimizer agent (read/edit tools scoped to the agent dir) → it edits instructions, preamble, topic playbooks, and description fields (it is forbidden from touching the eval suite) → re-run evals → keep on improvement, revert otherwise. History is appended to optimize-history.jsonl.

The preamble: what does Microsoft inject?

Unknown — and explicitly treated as such. preamble.md is our working approximation of Copilot Studio's server-side injection (identity/scope, orchestration rules, citation policy, safety wrapper), inferred from observed behavior. It is a first-class, separately-editable artifact so you can A/B it (toggle the preamble: key) and refine it by trial and error. examples/hr-agent/preamble-notes.md is a logbook for hypotheses and findings.

Known divergences from real Copilot Studio

• Topic execution: real topics run a deterministic node graph (message / question / condition nodes with slot-filling). We simulate topics as playbooks the model follows after the orchestrator triggers them. Good for orchestration/routing iteration; not a node-runtime replica.
• Knowledge retrieval: real sources use embeddings + Bing/etc.; we use local chunking + term scoring. Swap KnowledgeBase.search() for embeddings without changing the tool contract.
• Code execution sandbox: real one is a managed Python container with specific libraries; ours is local python3 in a temp dir.
• Entities/slot filling, adaptive cards, channels, auth passthrough: not simulated.

Architecture

src/
  types.ts      declarative agent schema
  config.ts     agent.yaml loading + system prompt composition (preamble+instructions)
  runtime.ts    pi session construction (topics/tools/MCP/code-exec/knowledge/children)
  mcp.ts        MCP server → pi tool bridge
  tools.ts      declarative tools + run_python sandbox
  knowledge.ts  chunking + retrieval
  eval.ts       suite runner, deterministic checks, report (md+json) + session capture
  judge.ts      LLM judge (own pi session)
  optimize.ts   auto-tuning loop
  inspect.ts    session JSONL transcript dumper
  cli.ts        chat / run / eval / optimize / inspect

Contributing

See CONTRIBUTING.md. Particularly welcome: preamble / orchestration findings, a real topic node-graph runtime, and importers from Copilot Studio exports.

Disclaimer

This project is provided "as is" under the MIT license. It is an unofficial approximation of Microsoft Copilot Studio behavior for local development and evaluation purposes. It does not replicate Microsoft's proprietary orchestration, safety, or grounding systems, and results obtained here may differ from the real product. You are responsible for complying with the terms of service of any model provider or subscription you connect (including GitHub Copilot, OpenAI, Anthropic, and Microsoft Azure).

License

MIT © 2026 butelo