agentHost: pin tool-search behavior in prompt snapshots - #333689
Draft
Bhavya U (bhavyaus) wants to merge 8 commits into
Draft
agentHost: pin tool-search behavior in prompt snapshots#333689Bhavya U (bhavyaus) wants to merge 8 commits into
Bhavya U (bhavyaus) wants to merge 8 commits into
Conversation
Contributor
There was a problem hiding this comment.
Copilot review overview
Review tier: Balanced
Findings: 2
New issues introduced by this change (3)
| Severity | Finding |
|---|---|
 |
src/vs/platform/agentHost/test/node/e2e/harness/agentHostE2ETestHarness.ts — The E2E architecture requires this suite to remain external to the Agent Host and explicitly… |
|
src/vs/platform/agentHost/test/node/e2e/providers/copilotPromptsE2E.integrationTest.ts — These expectations come directly from the implementation under test, contrary to the E2E suite's… |
 |
src/vs/platform/agentHost/test/node/e2e/providers/copilotPromptsE2E.integrationTest.ts — The current predicate returns true for gpt-5.6-sol, gpt-5.6-luna, and gpt-5.6-terra, and… |
What changed in this PR
Updates Agent Host prompt snapshots to exercise and pin tool-search gating.
Changes:
- Adds reusable E2E helpers for root configuration and active-client tools.
- Asserts tool-search guidance and carrier behavior.
- Regenerates all 19 model prompt snapshots.
| File | Description |
|---|---|
copilotCoverageSuite.ts |
Reuses canonical harness helpers. |
copilotPromptsE2E.integrationTest.ts |
Enables and asserts tool-search behavior. |
agentHostE2ETestHarness.ts |
Adds configuration and client-tool helpers. |
toolInstructions.ts |
Exports the tool-search guidance text. |
…gpt-5.prompt.md |
Adds canonical client tools. |
…gpt-5-mini.prompt.md |
Adds canonical client tools. |
…gpt-5-codex.prompt.md |
Adds canonical client tools. |
…gpt-5_1.prompt.md |
Adds canonical client tools. |
…gpt-5_1-codex.prompt.md |
Adds canonical client tools. |
…gpt-5_1-codex-mini.prompt.md |
Adds canonical client tools. |
…gpt-5_6-sol.prompt.md |
Adds client tools and guidance. |
…gpt-5_6-luna.prompt.md |
Adds client tools and guidance. |
…gpt-5_6-terra.prompt.md |
Adds client tools and guidance. |
…gemini-2_0-flash.prompt.md |
Adds canonical client tools. |
…claude-haiku-4_5.prompt.md |
Adds client tools and guidance. |
…claude-sonnet-4_5.prompt.md |
Adds client tools and guidance. |
…claude-sonnet-4_6.prompt.md |
Adds client tools and guidance. |
…claude-sonnet-5.prompt.md |
Adds client tools and guidance. |
…claude-opus-4_5.prompt.md |
Adds client tools and guidance. |
…claude-opus-4_6.prompt.md |
Adds client tools and guidance. |
…claude-opus-4_7.prompt.md |
Adds client tools and guidance. |
…claude-opus-4_8.prompt.md |
Adds client tools and guidance. |
…claude-opus-5.prompt.md |
Adds client tools and guidance. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Bhavya U (bhavyaus)
force-pushed
the
dev/bhavyau/tool-search-prompt-snapshot
branch
from
August 31, 2026 23:42
15a630a to
ed8f780
Compare
Contributor
|
Base:
|
The prompt snapshot suite never advertised the `toolSearch` client tool, so `toolSearchActive` was always false and the snapshots pinned nothing about tool search. Re-enabling GPT tool search in #333657 produced a zero-line snapshot diff, which is what surfaced the gap. Give sessions the harness's canonical client-tool profile so the host's client-tool-gated launch path is actually exercised, and assert on the host-authored guidance line in both directions using the production `agentHostModelSupportsToolSearch` predicate rather than a hand-copied model list. The CLI applies a second gate before exposing `tool_search_tool` that is unsatisfied under replay, so the carrier itself cannot be asserted here. `toolSearchToolWithoutHostGate` still catches a carrier leaking onto a model the host gate excludes. Extracts `setRootConfigValues`, `canonicalClientTools` and `registerCanonicalActiveClient` into the harness, de-duplicating the inline active-client setup in the tool-search coverage tests. Also hoists the tool-search guidance string in `prompts/toolInstructions.ts` to an exported constant so the test asserts against it rather than duplicating 318 characters of prompt text. The template literal had no interpolation, so this is a pure extraction. All 19 model snapshots regenerate. 12 gain the guidance line (9 Claude plus gpt-5.6-sol/luna/terra, which `agentHostModelSupportsToolSearch` covers via `isGpt56Model`); the other 7 gain only the canonical client tools. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep the prompt harness focused on the VS Code Copilot extension's toolSearch AHP reference rather than asserting on the SDK runtime's native carrier. Remove the duplicated investigation note from the repository. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Describe the root-config seam now used by the Copilot prompt snapshot suite so maintainers know config-gated contributors can be covered end to end. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Match action echoes by channel, sequence, and server ordering; surface rejections; and use structural equality so server-suppressed no-op patches cannot hang tests. Reuse the helper across remaining root-config writers and reject unexpected tool calls in record-only coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bhavya U (bhavyaus)
force-pushed
the
dev/bhavyau/tool-search-prompt-snapshot
branch
from
September 1, 2026 18:27
d480c1a to
1c1726e
Compare
Assert and snapshot the Copilot extension natural-language tool-search override across Claude and GPT Responses transports. Claude keeps deferred tools marked in place; GPT client search withholds them until discovery. Document the temporary dependency on runtime PR #18448. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove imports left unused after centralizing root-config updates so compile and hygiene checks pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The client profile models the Copilot extension boundary: its tool-search implementation is published over AHP by the
toolSearchreference name. The snapshots then verify the downstream Agent Host prompt contribution.Runtime dependency
Blocked on github/copilot-agent-runtime#18448, which restores SDK override behavior lost in Rust-port PR #13629. The currently pinned 1.0.83 runtime advertises and executes its regex search tool instead of the extension's natural-language client override.
The locally verified target snapshots are committed here. After #18448 releases, this PR must bump
@github/copilotto the containing build so normal CI reproduces them:tool_search_toolwith{ query }and 18 totaldefer_loading: truemarkers{ type: "tool_search", execution: "client" }with{ query }; deferred tools omitted initiallyThe shared helper uses the same structural equality as the server's no-op check, so reordered object-valued config cannot dispatch a suppressed no-op and hang. Record-only tool-search coverage also fails unexpected client tools instead of returning a fabricated success.
Validation
npm run transpile-client./scripts/test-integration.sh --run src/vs/platform/agentHost/test/node/e2e/providers/copilotPromptsE2E.integrationTest.tsReview
A full adversarial review traced the rebased PR and the subsequent helper-hardening changes. The confirmed branch-review findings are addressed; no blocking issues remain.