Skip to content

feat(testing): support alternative semantic index endpoints#308394

Merged
rwoll merged 6 commits intomainfrom
move-blackbird-harness-to-vscode
Apr 8, 2026
Merged

feat(testing): support alternative semantic index endpoints#308394
rwoll merged 6 commits intomainfrom
move-blackbird-harness-to-vscode

Conversation

@rwoll
Copy link
Copy Markdown
Member

@rwoll rwoll commented Apr 7, 2026

This follow-up tightens the scenario-automation Blackbird path added in this PR by aligning behavior with production expectations from review feedback. It closes gaps around glob filtering, cancellation propagation, and robustness of response handling, and adds focused unit coverage for this integration point.

  • Service behavior updates

    • ScenarioAutomationWorkspaceChunkSearchService.searchFileChunks now applies WorkspaceChunkSearchOptions.globPatterns (include/exclude) before emitting chunks.
    • Request cancellation is now wired end-to-end via IFetcherService.makeAbortController() and signal, with graceful empty-result returns on abort.
    • JSON parsing is wrapped to treat malformed responses as non-fatal ({ chunks: [] }) with error logging parity to other failure paths.
    • CancellationToken import was converted to type-only in this file.
  • New targeted unit tests

    • Added scenarioAutomationWorkspaceChunkSearchService.spec.ts under platform/workspaceChunkSearch/test/node/.
    • Covers:
      • request payload shape to localhost Blackbird endpoint
      • include and exclude glob filtering behavior
      • non-OK response fallback
      • invalid JSON fallback
      • cancellation/abort wiring and cancellation path handling
const abortController = this._fetcherService.makeAbortController();
const tokenListener = token.onCancellationRequested(() => abortController.abort());

response = await this._fetcherService.fetch(BLACKBIRD_EMBEDDINGS_URL, {
	callSite: 'ScenarioAutomationWorkspaceChunkSearchService.searchFileChunks',
	method: 'POST',
	headers: { 'Content-Type': 'application/json' },
	body: JSON.stringify(requestBody),
	signal: abortController.signal,
});

Updates to support semantic search with blackbird in MSBench.

- ScenarioAutomationWorkspaceChunkSearchService: directly calls Blackbird
  localhost:4443 embeddings endpoint bypassing production strategies
- ScenarioAutomationEndpointProviderImpl: add fallback to copilot-base when
  model family resolution fails in scenario automation
- Move IWorkspaceChunkSearchService registration into isScenarioAutomation
  if/else branch in services.ts
- Move IGithubCodeSearchService and IGithubAvailableEmbeddingTypesService
  registrations to shared (non-branched) section

Moved from microsoft/vscode-copilot-chat#3947

Co-authored-by: gryan11 <415784+gryan11@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds scenario-automation-specific semantic search support by introducing a lightweight workspace chunk search service that queries a local Blackbird embeddings endpoint, and by adjusting service registrations and endpoint resolution behavior to better support MSBench runs.

Changes:

  • Add ScenarioAutomationWorkspaceChunkSearchService that directly calls http://localhost:4443/api/embeddings/code/search for embeddings-based search.
  • Add a fallback in ScenarioAutomationEndpointProviderImpl.getChatEndpoint to use copilot-base when resolving a model family fails in scenario automation.
  • Refactor service registrations so IWorkspaceChunkSearchService is selected per scenario-automation vs production branch, while GitHub code search-related services are registered unconditionally.
Show a summary per file
File Description
extensions/copilot/src/platform/workspaceChunkSearch/node/scenarioAutomationWorkspaceChunkSearchService.ts New scenario-automation chunk search implementation that calls a local embeddings endpoint and parses results into FileChunkAndScore.
extensions/copilot/src/extension/prompt/vscode-node/scenarioAutomationEndpointProviderImpl.ts Adds try/catch fallback to copilot-base when model family resolution fails under scenario automation.
extensions/copilot/src/extension/extension/vscode-node/services.ts Moves IWorkspaceChunkSearchService registration into the scenario-automation branch and hoists GitHub code search service registrations to the shared section.

Copilot's findings

  • Files reviewed: 3/3 changed files
  • Comments generated: 5

@rwoll
Copy link
Copy Markdown
Member Author

rwoll commented Apr 7, 2026

@copilot apply changes based on the comments in this thread

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 7, 2026

@copilot apply changes based on the comments in this thread

Implemented the requested thread feedback in commit 227bd4b: wired cancellation into the fetch call, applied glob include/exclude filtering, switched CancellationToken to a type import, and added unit tests covering request payload, filtering, non-OK responses, invalid JSON, and cancellation behavior.

Copilot AI changed the title Add localhost embeddings support for scenario automation Address review feedback for scenario automation workspace chunk search Apr 7, 2026
Copilot finished work on behalf of rwoll April 7, 2026 23:36
@rwoll rwoll enabled auto-merge April 8, 2026 16:20
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 8, 2026

Screenshot Changes

Base: eb014b61 Current: 9ff5d936

Added (4)

editor/inlineChatAffordance/InlineChatAffordance/Dark

current

editor/inlineChatAffordance/InlineChatOverlay/Dark

current

editor/inlineChatAffordance/InlineChatAffordance/Light

current

editor/inlineChatAffordance/InlineChatOverlay/Light

current

@rwoll rwoll changed the title Address review feedback for scenario automation workspace chunk search feat(testing): support alternative semantic index endpoints Apr 8, 2026
@rwoll rwoll disabled auto-merge April 8, 2026 16:29
@rwoll rwoll enabled auto-merge (squash) April 8, 2026 16:29
The vscode repo's strict TypeScript config requires override modifiers
on methods that override base class members. Add them to all methods
in TestFetcherService.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
mjbvz
mjbvz previously approved these changes Apr 8, 2026
Fix: add missing override modifiers in test mock class
Add 'override' keyword to members that override base class members:
- TestGitService.repositories (overrides IGitService)
- TestLogService.trace (overrides ILogService)
- TestLogService.error (overrides ILogService)

Fixes TS4114 compilation errors in Copilot CI.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rwoll rwoll merged commit 2d7de17 into main Apr 8, 2026
23 checks passed
@rwoll rwoll deleted the move-blackbird-harness-to-vscode branch April 8, 2026 19:00
@vs-code-engineering vs-code-engineering bot added this to the 1.116.0 milestone Apr 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants