Normalize skill context replay snapshots#1269
Merged
Merged
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR stabilizes cross-SDK E2E replay snapshots for skills by normalizing <skill-context> user messages so YAML frontmatter from SKILL.md metadata is stripped before snapshot matching/writing. This prevents snapshot cache mismatches during runtime transitions where the metadata block may be present or omitted.
Changes:
- Normalize skill-context user messages in the replay proxy by removing
--- ... ---metadata frontmatter after the “Base directory for this skill:” line. - Update the shared skills replay snapshot to store the normalized (no-frontmatter) form.
- Add a harness unit test to ensure the normalization is applied when recording snapshots.
Show a summary per file
| File | Description |
|---|---|
| test/snapshots/skills/should_load_and_apply_skill_from_skilldirectories.yaml | Updates the stored conversation to the normalized skill-context form (frontmatter removed). |
| test/harness/replayingCapiProxy.ts | Adds skill-context frontmatter stripping as part of user-message normalization. |
| test/harness/replayingCapiProxy.test.ts | Adds coverage asserting frontmatter is removed from recorded skill-context user messages. |
Copilot's findings
- Files reviewed: 3/3 changed files
- Comments generated: 0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The live runtime changed the skill prompt context so
SKILL.mdYAML frontmatter may be omitted before the model call. That made the shared replay snapshot for the skills E2E brittle during the transition, causing cache mismatches even though the behavior under test is unchanged.This updates the replay proxy to canonicalize skill-context user messages by stripping skill metadata frontmatter before matching or writing snapshots. The shared skills snapshot now stores the normalized no-frontmatter form, and a harness unit test covers the normalization so Node, Python, Go, .NET, and Rust E2Es all use the same compatible replay behavior.
Validation:
npm test -- replayingCapiProxy.test.tsfromtest/harnessdotnet test dotnet\test\GitHub.Copilot.SDK.Test.csproj --filter "FullyQualifiedName~SkillsE2ETests.Should_Load_And_Apply_Skill_From_SkillDirectories" --no-restore --no-buildwithGITHUB_ACTIONS=true