feat: per-agent skill scoping, add skills: to .agent.md frontmatter - #328237
Open
Hadrian de Oliveira (hd-o) wants to merge 7 commits into
Open
feat: per-agent skill scoping, add skills: to .agent.md frontmatter#328237Hadrian de Oliveira (hd-o) wants to merge 7 commits into
skills: to .agent.md frontmatter#328237Hadrian de Oliveira (hd-o) wants to merge 7 commits into
Conversation
Allow .agent.md authors to restrict which skills appear in the model-invocable catalog via skills: (omit=all, []=none, ['*']=all), mirroring agents: filtering in automatic instructions. Includes parser/mode plumbing, editor completion and validation, and tests. Co-authored-by: Hadrian de Oliveira <hd-o@users.noreply.github.com>
Fail closed on malformed skills: values, tighten validator lookups, precompute the catalog allow-set, and clean up test call formatting. Co-authored-by: Hadrian de Oliveira <hd-o@users.noreply.github.com>
Hadrian de Oliveira (hd-o)
changed the title
skills: to .agent.md frontmatter
Author
|
@microsoft-github-policy-service agree |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds per-agent skills: allow-lists for custom agents and threads them through parsing, chat modes, instruction collection, editor tooling, and Copilot.
Changes:
- Parses, validates, completes, and documents
skills:frontmatter. - Filters model-invocable skill catalogs in core and Copilot.
- Adds API plumbing, caching, and test coverage.
Reviewed changes
Copilot reviewed 27 out of 28 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
src/vscode-dts/vscode.proposed.chatParticipantAdditions.d.ts |
Exposes allowed skills. |
src/vs/workbench/contrib/chat/test/common/promptSyntax/service/promptsService.test.ts |
Tests service plumbing. |
src/vs/workbench/contrib/chat/test/common/promptSyntax/service/promptFileParser.test.ts |
Tests parsing semantics. |
src/vs/workbench/contrib/chat/test/common/promptSyntax/computeAutomaticInstructions.test.ts |
Tests catalog filtering. |
src/vs/workbench/contrib/chat/test/common/chatModeService.test.ts |
Tests mode refreshes. |
src/vs/workbench/contrib/chat/test/browser/promptSyntax/languageProviders/promptValidator.test.ts |
Tests validation. |
src/vs/workbench/contrib/chat/test/browser/promptSyntax/languageProviders/promptHovers.test.ts |
Tests hover text. |
src/vs/workbench/contrib/chat/common/tools/builtinTools/runSubagentTool.ts |
Updates subagent collection call. |
src/vs/workbench/contrib/chat/common/promptSyntax/service/promptsServiceImpl.ts |
Maps parsed skills to agents. |
src/vs/workbench/contrib/chat/common/promptSyntax/service/promptsService.ts |
Extends custom-agent data. |
src/vs/workbench/contrib/chat/common/promptSyntax/promptFileParser.ts |
Parses skills:. |
src/vs/workbench/contrib/chat/common/promptSyntax/languageProviders/promptValidator.ts |
Validates skill names. |
src/vs/workbench/contrib/chat/common/promptSyntax/languageProviders/promptHeaderAutocompletion.ts |
Suggests available skills. |
src/vs/workbench/contrib/chat/common/promptSyntax/languageProviders/promptFileAttributes.ts |
Registers field metadata. |
src/vs/workbench/contrib/chat/common/promptSyntax/computeAutomaticInstructions.ts |
Filters the core catalog. |
src/vs/workbench/contrib/chat/common/model/chatModel.ts |
Extends mode instructions. |
src/vs/workbench/contrib/chat/common/chatService/chatServiceImpl.ts |
Forwards enabled skills. |
src/vs/workbench/contrib/chat/common/chatService/chatService.ts |
Extends request options. |
src/vs/workbench/contrib/chat/common/chatModes.ts |
Stores and observes skills. |
src/vs/workbench/contrib/chat/browser/widget/input/chatInputPart.ts |
Adds skills to mode instructions. |
src/vs/workbench/contrib/chat/browser/widget/chatWidget.ts |
Supplies skills during collection. |
src/vs/workbench/api/test/common/extHostTypeConverters.test.ts |
Tests API conversion. |
src/vs/workbench/api/common/extHostTypeConverters.ts |
Converts allowed skills. |
extensions/copilot/src/util/vs/workbench/contrib/chat/common/promptSyntax/promptFileParser.ts |
Mirrors parser support. |
extensions/copilot/src/platform/promptFiles/test/node/automaticInstructionsCollector.spec.ts |
Tests extension filtering. |
extensions/copilot/src/platform/promptFiles/node/automaticInstructionsCollector.ts |
Filters the extension catalog. |
extensions/copilot/src/extension/agents/vscode-node/agentTypes.ts |
Serializes generated agents. |
extensions/copilot/assets/prompts/skills/agent-customization/references/agents.md |
Documents the field. |
| // When the extension is responsible for instruction collection, skip the core path entirely. | ||
| if (this.configurationService.getValue<boolean>(ChatConfiguration.CollectInstructionsInExtension) !== true) { | ||
| const computer = this.instantiationService.createInstance(ComputeAutomaticInstructions, ChatModeKind.Agent, modeTools, undefined, getChatSessionType(invocation.context.sessionResource)); | ||
| const computer = this.instantiationService.createInstance(ComputeAutomaticInstructions, ChatModeKind.Agent, modeTools, undefined, undefined, getChatSessionType(invocation.context.sessionResource)); |
There was a problem hiding this comment.
Valid. Will push update soon
There was a problem hiding this comment.
Done: dcf7bd1
| enabledTools: modeKind === ChatModeKind.Agent ? this.input.selectedToolsModel.userSelectedTools.get() : undefined, | ||
| enabledSubAgents: modeKind === ChatModeKind.Agent ? this.input.currentModeObs.get().agents?.get() : undefined | ||
| enabledSubAgents: modeKind === ChatModeKind.Agent ? this.input.currentModeObs.get().agents?.get() : undefined, | ||
| enabledSkills: modeKind === ChatModeKind.Agent ? this.input.currentModeObs.get().customSkills?.get() : undefined |
There was a problem hiding this comment.
Intentional for this PR. Can be a follow-up PR if needed?
| return { id, uri, name, description, model, tools, handOffs, argumentHint, target, visibility, agents, agentInstructions, source, sessionTypes, hooks, enabled }; | ||
| // Claude's `skills:` means "preload into subagent context", not a catalog whitelist. | ||
| // Only VS Code / default-target agents use `skills:` as an allow-list. | ||
| const skills = target === Target.Claude ? undefined : ast.header.skills; |
There was a problem hiding this comment.
Valid. Will push update soon
There was a problem hiding this comment.
Done: 1cb0d06
Comment on lines
+331
to
+335
| for (const item of value.items) { | ||
| if (item.type === 'scalar' && item.value) { | ||
| skills.push(item.value); | ||
| } | ||
| } |
There was a problem hiding this comment.
Copilot shouldn't this have the same behavior as the tools: field?
Comment on lines
+333
to
+337
| for (const item of value.items) { | ||
| if (item.type === 'scalar' && item.value) { | ||
| skills.push(item.value); | ||
| } | ||
| } |
There was a problem hiding this comment.
Awaiting on the discussion above for decision
Pass the selected subagent's skills (or the current mode's allowedSkills) into modeInstructions and ComputeAutomaticInstructions so skills: [] no longer falls back to the full catalog. Co-authored-by: Hadrian de Oliveira <hd-o@users.noreply.github.com>
Only propagate header skills for VS Code and default-target agents. GitHub Copilot does not support skills as an attribute, so ignore it there the same way Claude preload semantics are ignored. Co-authored-by: Hadrian de Oliveira <hd-o@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 29 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (4)
src/vs/workbench/contrib/chat/test/common/tools/builtinTools/runSubagentTool.test.ts:1139
- As above, the captured
createInstancearguments include the constructor at index 0, so index 3 is theenabledSubagentsargument. Read index 4 to assert the allowed-skills value; otherwise this test always fails.
assert.deepStrictEqual(capturedComputeArgs[0][3], ['security-review']);
extensions/copilot/src/platform/promptFiles/node/automaticInstructionsCollector.ts:184
allowedSkillsnow changes the automatic catalog, but the Responses API mode-change check still compares only URI/name/content/builtin/metadata. If the same agent is edited from all/subset skills toskills: [], the request reusesprevious_response_id, whose state still contains the old catalog, so the removed skills remain visible to the model. IncludeallowedSkillsinareModeInstructionsEqual(with array equality) and cover a skills-only mode update.
const indexEntry = await this._buildCustomizationsIndex(instructionFiles, tools, modeInstructions2?.allowedSubagents, modeInstructions2?.allowedSkills, sessionType, telemetry, token);
src/vs/workbench/contrib/chat/common/promptSyntax/languageProviders/promptFileAttributes.ts:149
- The hover says these are the skills “available” to the agent, although this field only filters the automatically advertised catalog and does not remove user-invocable skill commands. Reword the hover to say “advertised in the model-invocable skill catalog” and update the duplicate description/test so the UI matches the documented semantics.
description: localize('promptHeader.agent.skills', 'Skills available to this agent. Omit for all skills; use `[]` for none; use `[\'*\']` for all.'),
src/vs/workbench/contrib/chat/test/common/tools/builtinTools/runSubagentTool.test.ts:1114
capturedComputeArgsrecords the fullcreateInstancecall, includingComputeAutomaticInstructionsat index 0. The enabled-skills constructor argument is therefore at index 4, while index 3 isenabledSubagents; this assertion currently comparesundefinedwith[]and fails.
This issue also appears on line 1139 of the same file.
assert.deepStrictEqual(capturedComputeArgs[0][3], []);
Author
|
Martin Aeschlimann (@aeschli) Paul (@pwang347) [not urgent] copilot is ignoring my comments 🥲 (I increased my budget already). A review from you would be highly appreciated 🙏 no rush, whenever you have time |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #307630
Summary
Adds a
skills:frontmatter field on custom agents (.agent.md) so authors can control which skills appear in that agent’s model-invocable skill catalog. Similar totools:field.This change makes loop engineering easier with vscode-copilot (I'm not sure how copilot cloud agents work), allowing minimal session context on specialized agents, specially for teams using monorepos with lots of shared skills. It also gets vscode-copilot closer to the capabilities offered by other harnesses like Codex and Open Code that already provide this feature.
This is different than Claude Code's
skills:which means "preloaded skills into subagent context", which I recommend to be implemented in vscode/copilot as field namedpreload-skills(#293752).Semantics
skillsskills: []skills: [a, b]a/b(that also pass existing filters)skills: ['*']Existing per-skill filters still apply after the whitelist (
disable-model-invocation, required description, session type, etc.). Unknown skill names do not fail agent load; the editor reports a warning.Implementation
skillsonPromptHeader(sequence or comma-separated scalar)ICustomAgent.skills→CustomChatMode.customSkillsComputeAutomaticInstructionsand CopilotAutomaticInstructionsCollectorallowedSkillsonChatRequestModeInstructionsfor the extension pathfindAgentSkills(), validation, hoversTests
*/ scalar / omit[]/['*']/ subset /disableModelInvocationcustomSkillsallowedSkillsfilter