Skip to content

feat(copilot): add a pull-request integrity check for GitHub Copilot - #68

Merged
imran-siddique merged 3 commits into
mainfrom
feat/copilot-integration
Aug 1, 2026
Merged

feat(copilot): add a pull-request integrity check for GitHub Copilot#68
imran-siddique merged 3 commits into
mainfrom
feat/copilot-integration

Conversation

@imran-siddique

Copy link
Copy Markdown
Contributor

The fourth engine, and the first one that is not a session hook.

Why the shape is different

Copilot's composition lives in the repository. Its instructions, skills and MCP configuration are files that arrive by pull request.

The Claude Code and Codex integrations watch a developer's machine and warn at session start, after the fact, one developer at a time. They have to: that composition lives in a home directory. Here it is reviewed code, so drift can be caught on entry:

  • One baseline, shared. Committed at .agentrust/copilot-baseline.json, not one per laptop.
  • Reviewed like code. A change to the agent's instructions appears in a diff, with an author.
  • Enforceable. As a required check, a pull request that changes the agent's behaviour without updating the baseline does not merge.

CODEOWNERS for the agent's brain, roughly.

The surface is wider than I expected

I said I would verify the paths against GitHub's docs before building, and it changed the design:

Category Paths
Instructions .github/copilot-instructions.md, .github/instructions/**/*.instructions.md, AGENTS.md anywhere in the tree, root CLAUDE.md and GEMINI.md
Skills .github/skills/<n>/, .claude/skills/<n>/, .agents/skills/<n>/
MCP copilot/mcp-config.json, .vscode/mcp.json

AGENTS.md is matched anywhere, because Copilot resolves the nearest one. A file three directories down changes how the agent behaves in that subtree without touching anything at the root, and that is precisely the change worth catching. Vendored directories are skipped, so a dependency shipping its own AGENTS.md is not counted as yours. Tested both ways.

Skills are digested across the whole directory via the shared core, so the bypass that was live in two other engines (#63, #65) does not reappear. Copilot skills use the same SKILL.md plus supporting-files shape, so it would have.

Two deliberate omissions, both load-bearing

No baseline sealing, unlike the other three. They seal because a local baseline can be rewritten with nothing to show for it. A committed baseline gets provenance from git: every change appears in a diff, carries an author, passes review. A digest on top would be ceremony, and shipping ceremony as security is the habit this repo argues against.

No integration.yaml. The schema requires integrates_with to be one of cmcp, trace or agent-manifest. This check emits none of them, so claiming one would be an unverifiable claim, and CONTRIBUTING rule 2 is explicit about those. The README says so plainly and names emitting a TRACE record per checked pull request as what would make one true. That is also the natural first consumer for trace-registry.

Worth noting the schema assumes every integration integrates with the AgenTrust stack. A drift checker that uses the shared core but emits no record has no honest value to put there, which may be a gap in the schema rather than in this integration.

Adoption details that matter more than they look

  • A missing baseline reports and exits 0. A repository adopting this should not have its first pull request blocked by the absence of a file nobody told it to create.
  • fail-on-drift can be turned off. Report without blocking is the sensible first move on a busy repo; flip it on once the baseline settles.
  • One comment per pull request, edited in place. A comment per push is noise people mute.
  • No install step. Engine plus vendored core are standard library only.

Test plan

  • 25 tests, standard library only. All three skill roots, nested .github/instructions, AGENTS.md at depth, vendored AGENTS.md excluded, the skill-script bypass, state churn not alarming, both MCP paths, and verify's exit codes as a status check.
  • Dogfooded: snapshot and verify run against this repository in CI. It currently has none of these surfaces, so the counts are zero and verify says there is no baseline. Honest, and it becomes a real gate on ourselves once one is committed.
  • ruff check --target-version py39 clean. Vendored sync check passes across all four engines.

One repo-wide problem I hit and worked around. Four engines each define a module named capture, so sys.path insertion made this suite run against another engine's code when the repository is collected in one pytest command: 25 failures against the wrong module. This suite now imports its engine by path under a unique name, and is named test_copilot_capture.py so it does not join the pre-existing test_capture.py basename collision between the other three. A single root-level pytest still fails on that older collision, which predates this PR and deserves a follow-up.

imran-siddique and others added 3 commits July 31, 2026 17:04
Copilot's composition lives in the repository. Its instructions, skills and MCP
configuration are files that arrive by pull request, which makes a different control
correct here than for the other engines.

The Claude Code and Codex integrations watch a developer's machine and warn at
session start, after the fact, one developer at a time. They have to: that
composition lives in a home directory. Here the composition is reviewed code, so
drift is caught on entry. One baseline committed at .agentrust/copilot-baseline.json,
and a status check that fails a pull request changing what Copilot reads without
updating the baseline in the same change.

Paths were verified against GitHub's documentation, and the surface is wider than
expected. Copilot reads AGENTS.md ANYWHERE in the tree, nearest wins, plus root
CLAUDE.md and GEMINI.md as alternatives, plus .github/copilot-instructions.md and
.github/instructions/**/*.instructions.md. Skills resolve from three in-repo roots:
.github/skills, .claude/skills and .agents/skills. So a file three directories down
changes how the agent behaves in that subtree without touching anything at the root,
which is exactly the change worth catching. Vendored directories are skipped so a
dependency shipping its own AGENTS.md is not counted as ours.

Skills are digested across the whole directory via the shared core, so the bypass
that was live in two other engines does not reappear here. Copilot skills use the
same SKILL.md plus supporting-files shape, so it would have.

This engine deliberately does NOT seal its baseline, unlike the others. They seal
because a local baseline can be rewritten with nothing to show for it. A committed
baseline gets provenance from git: every change appears in a diff, carries an author,
and passes review. A digest on top would be ceremony.

The action needs no install step, since the engine and its vendored core are
standard library only. fail-on-drift defaults true but can be turned off, which is
the sensible first move on a busy repository. The comment is one per pull request,
edited in place, because a comment per push is noise people mute. A missing baseline
reports and exits 0 rather than blocking a repository that has not adopted one.

No integration.yaml. The schema requires integrates_with to be one of cmcp, trace or
agent-manifest, and this check emits none of them, so claiming one would be an
unverifiable claim and CONTRIBUTING is explicit about those. The README says so and
names emitting a TRACE record per checked pull request as what would make one true.

25 tests. The suite imports its engine by path under a unique module name: four
engines here each define a module called `capture`, so sys.path insertion made this
file run against another engine's code when the repository was collected in one
command. Also named test_copilot_capture.py rather than test_capture.py so it does
not join the pre-existing basename collision between the other suites, which still
breaks a single root-level pytest run and is worth a follow-up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Imran Siddique <imran.siddique@opaque.co>
#67 added the mode parameter to core.atomic_write while this branch was open, so
the copilot vendored copy was a version behind. Caught by the vendored-in-sync
job against the merge with main, which is exactly what that check is for.

Signed-off-by: Imran Siddique <imran.siddique@opaque.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant