chat: add multiple 'initiator' types to `provideLanguageModelResponse` #250018

connor4312 · 2025-05-28T22:06:55Z

MCP servers can make chat requests via sampling. This swaps extensionId
in provideLanguageModelResponse with an initiator so we can
represent these calls properly.

MCP servers can make chat requests via sampling. This swaps `extensionId` in `provideLanguageModelResponse` with an `initiator` so we can represent these calls properly.

jrieken · 2025-06-02T10:27:05Z

This swaps extensionId in provideLanguageModelResponse with an initiator so we can represent these calls properly.

I don't understand this. AFAIK the extensionId in the provideLanguageModelResponse direction isn't used for presentation but for extension specific checks, e.g to enable different models for friend extensions or for extension-specific token limits

connor4312 · 2025-06-02T14:54:43Z

I don't understand this. AFAIK the extensionId in the provideLanguageModelResponse direction isn't used for presentation but for extension specific checks, e.g to enable different models for friend extensions or for extension-specific token limits

In copilot-chat this is indeed the case. However we lack any way to say that a chat didn't come from an extension, and giving an invalid extension ID will cause copilot-chat to throw. This API change allows for us to express the initator correctly.

jrieken · 2025-06-02T16:22:30Z

However we lack any way to say that a chat didn't come from an extension, and giving an invalid extension ID will cause copilot-chat to throw. This API change allows for us to express the initator correctly.

AFAIK that's something we never really used. There is an EXP from last year that serves as a kill switch (in case an extension goes bad) and then there is some CAPI logic with the x-onbehalf-on header. cc @lramos15 to check if CAPI ever used this

@connor4312 are you planning on extending that to MCP servers? IMO we should reconsider this and instead of glorifying the initiator we should just make extensionId optional and eventually phase it out. There will also be cases in which the initiator is neither MCP nor an extension but just VS Code itself

jrieken

Let's first agree on this being needed

jrieken · 2025-06-02T16:28:21Z

So, I suggest to pass undefined or '' as extension id and handle this case gracefully in the chat extension, like not doing any of the extension specific checks we have there and later revisit if we can remove all of this

connor4312 · 2025-06-02T16:30:16Z

Currently in main I just use the copilot extension ID to make it work, which is a hack. If we don't need this I'm good with making the extension ID optional instead.

There will also be cases in which the initiator is neither MCP nor an extension but just VS Code itself

Agree, my thought was, if we keep the initiator then we can add an additional type to the union when this case arises.

are you planning on extending that to MCP servers?

Yes, MCP supports "sampling" which is basically making an LM request. At the moment, until we get the registry fully in, we don't know the 'identity' of MCP servers and only have their user-defined label. But once we get the registry in we'll know (for registry-based servers) their name in a deterministic way.

jrieken · 2025-06-03T12:28:09Z

At the moment, until we get the registry fully in, we don't know the 'identity' of MCP servers and only have their user-defined label.

Tho, stdio servers, that I have locally and that I configure via mcp.json, always have a user-defined label, right?

connor4312 · 2025-06-03T12:41:16Z

Tho, stdio servers, that I have locally and that I configure via mcp.json, always have a user-defined label, right?

Yes

connor4312 · 2025-06-03T12:47:57Z

Also, I think we may need/want this because the extension's onDidReceiveLanguageModelResponse is what's used to drive the usage graph, which we will want for MCP too

jrieken · 2025-06-03T12:50:35Z

Also, I think we may need/want this because the extension's onDidReceiveLanguageModelResponse is what's used to drive the usage graph,

That's only needed because the chat extension itself doesn't use the API to make LM requests. For all other cases we know the initiator of LM requests (based on the scoped API) and this event is something we won't finalize.

jrieken · 2025-06-03T12:56:29Z

CAPI does utilize this value for some switching and usage reporting. Does it make sense to have MCP servers send a distinct header? Is the Copilot Chat extension the initiator in this case?

Then let's plan a future with CAPI where

this value can be an extension id (as today and guaranteed to be correct)
this value can be a MCP id (in case of MCP servers from the registery)
this value can be a random, untrusted string (local MCP server)
this value can be an identifier for core itself (for vscode native LM usage outside of any extension)

@lramos15 can you check with CAPI folks on this please

connor4312 · 2025-06-06T21:58:56Z

I made some API updates to reflect that, lmk what you think (or feel free to just push changes)

jrieken

Given we aim to finalize this API soon, I have done some API nit.

(Tho, I am unsure if the initiator aspect is something we would finalise or not?)

jrieken · 2025-06-09T14:11:19Z

src/vscode-dts/vscode.proposed.chatProvider.d.ts

+		reason: string;
+	}
+
+	export type LanguageModelRequestInitiator = ExtensionLanguageModelRequestInitiator | McpServerLanguageModelRequestInitiator | InternalLanguageModelRequestInitiator;


In the API we don't do kind-discriminated or-types but plain simple classes, like InlineValue. I think we have two options:

have different initiators be represented as classes, or

have one fits-all initiator which has the kind-attribute and something generic like identifier

The one case I was looking at was quickpick items vs separators that have a kind like we have here. The only reason I didn't do classes is that there's no reason for an extension to instantiate these types, but that could also be done with a protected or private constructor.

yikes that quick pick. tho, it is same, same but different. there is no or-type. just a single type that's different depending on the value of kind. That's something we can have too (basically my 2nd suggestion)

jrieken · 2025-06-09T14:12:54Z

src/vscode-dts/vscode.proposed.chatProvider.d.ts


-		provideLanguageModelResponse(messages: Array<LanguageModelChatMessage | LanguageModelChatMessage2>, options: LanguageModelChatRequestOptions, extensionId: string, progress: Progress<ChatResponseFragment2>, token: CancellationToken): Thenable<any>;
+		provideLanguageModelResponse(messages: Array<LanguageModelChatMessage | LanguageModelChatMessage2>, options: LanguageModelChatRequestOptions, initiator: LanguageModelRequestInitiator, progress: Progress<ChatResponseFragment2>, token: CancellationToken): Thenable<any>;


Likely out of scope for this PR but the initiator argument should probably be merged with the option argument and then be a new, fresh type for the provider-side. So that LanguageModelChatRequestOptions is reused (in a cyclic API) and so that things are contained nicely

jrieken · 2025-06-09T14:17:11Z

src/vscode-dts/vscode.proposed.chatProvider.d.ts

 	/**
 	 * Represents a large language model that accepts ChatML messages and produces a streaming response
 	*/
 	export interface LanguageModelChatProvider {

 		// TODO@API remove or keep proposed?
-		onDidReceiveLanguageModelResponse2?: Event<{ readonly extensionId: string; readonly participant?: string; readonly tokenCount?: number }>;
+		onDidReceiveLanguageModelResponse2?: Event<{ readonly initiator: LanguageModelRequestInitiator; readonly participant?: string; readonly tokenCount?: number }>;


I know it follows the beaten path but this needs revisiting. Core should just know when a LM has been used and AFAIK this is only needed because our extension makes requests without going through the API. Maybe @sandy081 can clarify if/how this is still needed

chat: add multiple 'initiator' types to provideLanguageModelResponse

b6ed7f0

MCP servers can make chat requests via sampling. This swaps `extensionId` in `provideLanguageModelResponse` with an `initiator` so we can represent these calls properly.

connor4312 assigned jrieken and roblourens May 28, 2025

connor4312 requested review from roblourens and jrieken May 28, 2025 22:07

connor4312 assigned connor4312 and unassigned roblourens and jrieken May 28, 2025

vs-code-engineering bot added this to the May 2025 milestone May 28, 2025

roblourens previously approved these changes May 31, 2025

View reviewed changes

Merge remote-tracking branch 'origin/main' into connor4312/sampling-2

1fd6a52

connor4312 dismissed roblourens’s stale review via 1fd6a52 May 31, 2025 16:09

jrieken requested changes Jun 2, 2025

View reviewed changes

connor4312 modified the milestones: May 2025, June 2025 Jun 6, 2025

connor4312 added 2 commits June 6, 2025 14:52

Merge remote-tracking branch 'origin/main' into connor4312/sampling-2

44f3f04

update with editor kind

c2df403

connor4312 requested a review from jrieken June 6, 2025 21:59

jrieken requested changes Jun 9, 2025

View reviewed changes


		provideLanguageModelResponse(messages: Array<LanguageModelChatMessage \| LanguageModelChatMessage2>, options: LanguageModelChatRequestOptions, extensionId: string, progress: Progress<ChatResponseFragment2>, token: CancellationToken): Thenable<any>;
		provideLanguageModelResponse(messages: Array<LanguageModelChatMessage \| LanguageModelChatMessage2>, options: LanguageModelChatRequestOptions, initiator: LanguageModelRequestInitiator, progress: Progress<ChatResponseFragment2>, token: CancellationToken): Thenable<any>;

chat: add multiple 'initiator' types to provideLanguageModelResponse #250018

Are you sure you want to change the base?

chat: add multiple 'initiator' types to provideLanguageModelResponse #250018

Conversation

connor4312 commented May 28, 2025

Uh oh!

jrieken commented Jun 2, 2025

Uh oh!

connor4312 commented Jun 2, 2025

Uh oh!

jrieken commented Jun 2, 2025

Uh oh!

jrieken left a comment

Choose a reason for hiding this comment

Uh oh!

jrieken commented Jun 2, 2025

Uh oh!

connor4312 commented Jun 2, 2025

Uh oh!

jrieken commented Jun 3, 2025

Uh oh!

connor4312 commented Jun 3, 2025

Uh oh!

connor4312 commented Jun 3, 2025

Uh oh!

jrieken commented Jun 3, 2025

Uh oh!

jrieken commented Jun 3, 2025

Uh oh!

connor4312 commented Jun 6, 2025

Uh oh!

jrieken left a comment

Choose a reason for hiding this comment

Uh oh!

jrieken Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

connor4312 Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

jrieken Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

jrieken Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

jrieken Jun 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chat: add multiple 'initiator' types to `provideLanguageModelResponse` #250018

chat: add multiple 'initiator' types to `provideLanguageModelResponse` #250018