feat(copilot): add a pull-request integrity check for GitHub Copilot - #68
Merged
Conversation
Copilot's composition lives in the repository. Its instructions, skills and MCP configuration are files that arrive by pull request, which makes a different control correct here than for the other engines. The Claude Code and Codex integrations watch a developer's machine and warn at session start, after the fact, one developer at a time. They have to: that composition lives in a home directory. Here the composition is reviewed code, so drift is caught on entry. One baseline committed at .agentrust/copilot-baseline.json, and a status check that fails a pull request changing what Copilot reads without updating the baseline in the same change. Paths were verified against GitHub's documentation, and the surface is wider than expected. Copilot reads AGENTS.md ANYWHERE in the tree, nearest wins, plus root CLAUDE.md and GEMINI.md as alternatives, plus .github/copilot-instructions.md and .github/instructions/**/*.instructions.md. Skills resolve from three in-repo roots: .github/skills, .claude/skills and .agents/skills. So a file three directories down changes how the agent behaves in that subtree without touching anything at the root, which is exactly the change worth catching. Vendored directories are skipped so a dependency shipping its own AGENTS.md is not counted as ours. Skills are digested across the whole directory via the shared core, so the bypass that was live in two other engines does not reappear here. Copilot skills use the same SKILL.md plus supporting-files shape, so it would have. This engine deliberately does NOT seal its baseline, unlike the others. They seal because a local baseline can be rewritten with nothing to show for it. A committed baseline gets provenance from git: every change appears in a diff, carries an author, and passes review. A digest on top would be ceremony. The action needs no install step, since the engine and its vendored core are standard library only. fail-on-drift defaults true but can be turned off, which is the sensible first move on a busy repository. The comment is one per pull request, edited in place, because a comment per push is noise people mute. A missing baseline reports and exits 0 rather than blocking a repository that has not adopted one. No integration.yaml. The schema requires integrates_with to be one of cmcp, trace or agent-manifest, and this check emits none of them, so claiming one would be an unverifiable claim and CONTRIBUTING is explicit about those. The README says so and names emitting a TRACE record per checked pull request as what would make one true. 25 tests. The suite imports its engine by path under a unique module name: four engines here each define a module called `capture`, so sys.path insertion made this file run against another engine's code when the repository was collected in one command. Also named test_copilot_capture.py rather than test_capture.py so it does not join the pre-existing basename collision between the other suites, which still breaks a single root-level pytest run and is worth a follow-up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Imran Siddique <imran.siddique@opaque.co>
#67 added the mode parameter to core.atomic_write while this branch was open, so the copilot vendored copy was a version behind. Caught by the vendored-in-sync job against the merge with main, which is exactly what that check is for. Signed-off-by: Imran Siddique <imran.siddique@opaque.co>
This was referenced Aug 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The fourth engine, and the first one that is not a session hook.
Why the shape is different
Copilot's composition lives in the repository. Its instructions, skills and MCP configuration are files that arrive by pull request.
The Claude Code and Codex integrations watch a developer's machine and warn at session start, after the fact, one developer at a time. They have to: that composition lives in a home directory. Here it is reviewed code, so drift can be caught on entry:
.agentrust/copilot-baseline.json, not one per laptop.CODEOWNERS for the agent's brain, roughly.
The surface is wider than I expected
I said I would verify the paths against GitHub's docs before building, and it changed the design:
.github/copilot-instructions.md,.github/instructions/**/*.instructions.md,AGENTS.mdanywhere in the tree, rootCLAUDE.mdandGEMINI.md.github/skills/<n>/,.claude/skills/<n>/,.agents/skills/<n>/copilot/mcp-config.json,.vscode/mcp.jsonAGENTS.mdis matched anywhere, because Copilot resolves the nearest one. A file three directories down changes how the agent behaves in that subtree without touching anything at the root, and that is precisely the change worth catching. Vendored directories are skipped, so a dependency shipping its ownAGENTS.mdis not counted as yours. Tested both ways.Skills are digested across the whole directory via the shared core, so the bypass that was live in two other engines (#63, #65) does not reappear. Copilot skills use the same
SKILL.mdplus supporting-files shape, so it would have.Two deliberate omissions, both load-bearing
No baseline sealing, unlike the other three. They seal because a local baseline can be rewritten with nothing to show for it. A committed baseline gets provenance from git: every change appears in a diff, carries an author, passes review. A digest on top would be ceremony, and shipping ceremony as security is the habit this repo argues against.
No
integration.yaml. The schema requiresintegrates_withto be one ofcmcp,traceoragent-manifest. This check emits none of them, so claiming one would be an unverifiable claim, andCONTRIBUTINGrule 2 is explicit about those. The README says so plainly and names emitting a TRACE record per checked pull request as what would make one true. That is also the natural first consumer fortrace-registry.Worth noting the schema assumes every integration integrates with the AgenTrust stack. A drift checker that uses the shared core but emits no record has no honest value to put there, which may be a gap in the schema rather than in this integration.
Adoption details that matter more than they look
fail-on-driftcan be turned off. Report without blocking is the sensible first move on a busy repo; flip it on once the baseline settles.Test plan
.github/instructions,AGENTS.mdat depth, vendoredAGENTS.mdexcluded, the skill-script bypass, state churn not alarming, both MCP paths, and verify's exit codes as a status check.snapshotandverifyrun against this repository in CI. It currently has none of these surfaces, so the counts are zero andverifysays there is no baseline. Honest, and it becomes a real gate on ourselves once one is committed.ruff check --target-version py39clean. Vendored sync check passes across all four engines.One repo-wide problem I hit and worked around. Four engines each define a module named
capture, sosys.pathinsertion made this suite run against another engine's code when the repository is collected in one pytest command: 25 failures against the wrong module. This suite now imports its engine by path under a unique name, and is namedtest_copilot_capture.pyso it does not join the pre-existingtest_capture.pybasename collision between the other three. A single root-levelpyteststill fails on that older collision, which predates this PR and deserves a follow-up.