Add requestKind to chat response telemetry events#312262
Add requestKind to chat response telemetry events#312262kevin-m-kent wants to merge 10 commits intomicrosoft:mainfrom
Conversation
Emit a new 'requestKind' property on response.success, response.cancelled, and response.error. Values: 'background', 'subagent', or 'normal' (derived from the same requestKindOptions that sets the X-Interaction-Type header). This lets us distinguish background and subagent requests from primary user requests in telemetry, complementing the existing initiatorType (user/agent) signal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR extends Copilot chat response telemetry to include a requestKind dimension ('normal' | 'background' | 'subagent') on the CAPI path, derived from existing requestKindOptions that already influence the X-Interaction-Type header. This enables slicing success/cancel/error metrics by request kind without changing runtime behavior.
Changes:
- Adds
requestKindOptionsto the telemetry payload interfaces and emitsrequestKindonresponse.success,response.cancelled, andresponse.error. - Updates the GDPR schemas for the three events to include
requestKind(SystemMetaData / FeatureInsight). - Threads
requestKindOptionsthroughChatMLFetcherImplsuccess/cancel/error telemetry call paths (including retry/error paths).
Show a summary per file
| File | Description |
|---|---|
| extensions/copilot/src/extension/prompt/node/chatMLFetcherTelemetry.ts | Adds requestKind emission and GDPR schema entries for success/cancel/error response telemetry. |
| extensions/copilot/src/extension/prompt/node/chatMLFetcher.ts | Passes requestKindOptions through to telemetry senders and into processSuccessfulResponse. |
Copilot's findings
- Files reviewed: 2/2 changed files
- Comments generated: 0
|
@roblourens curious what you think of this. We are looking to filter out background requests in a robust way for our A/Bs, rather than maintain a list of sources. Copilot seems to think that this header for background interaction type never actually fires, but I'm suspicious of that claim. |
…estId on response.cancelled/error
Inverts requestKind defaulting in chatMLFetcher.fetchMany so that any
request reaching the fetcher without an explicit requestKindOptions is
classified as 'background'. Adds 'mainagent' as the explicit opt-in for
primary user-driven turns. Now:
- defaultIntentRequestHandler passes { kind: 'subagent' } for subagents
and { kind: 'mainagent' } for normal user turns.
- inlineChatIntent (both chat2 and inline edit paths) passes
{ kind: 'mainagent' }.
- All other callers (title gen, summarization, intent detection, commit
message gen, terminal fix gen, PR title/desc gen, rename suggestions,
semantic search, virtualToolGrouper, etc.) automatically get
'background' without per-call edits.
Also adds parentRequestId and conversationId to response.success,
response.cancelled, and response.error events so subagent requests can
be joined back to the main agent request, matching the
panel.request schema.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Conversation history summarization is in the critical path of a user turn (it gates the next agent step), so classifying it as 'background' alongside truly out-of-band utility calls (chat title gen, intent detection, commit message gen, etc.) loses an important signal. Adds a 'summarization' kind and marks both summarization sites (SummarizedConversationHistory full/simple path and AgentIntent inline path) explicitly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- executionSubagentToolCallingLoop: explicitly set kind: 'subagent' (was defaulting to 'background') - ChatSummarizerProvider (vscode.ChatSummarizer): kind: 'summarization' (was defaulting to 'background'). Switched the call from positional makeChatRequest to makeChatRequest2 so requestKindOptions can be passed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collapse the four IBackgroundRequestOptions / ISubagentRequestOptions / ISummarizationRequestOptions / IMainAgentRequestOptions interfaces into a single 'as const' object + derived type, matching the existing ToolName / ContributedToolName pattern in this codebase. Call sites now use RequestKind.MainAgent etc. instead of hand-typing 'mainagent', so a future rename only has to update the RequestKind definition. Compile-time safety is unchanged — IRequestKindOptions.kind is still typed as the union of literals. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# Conflicts: # extensions/copilot/src/extension/inlineChat2/node/inlineChatIntent.ts
xtabProvider and xtabNextCursorPredictor were defaulting to 'background' but they're a distinct workload (speculative inline edit prediction with different cost/latency/cache characteristics than other utility callers). Tag them as 'nes' so telemetry can slice them out. Header behavior unchanged — 'nes' falls through to the location-derived intent on X-Interaction-Type, like 'mainagent' and 'summarization'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add resolveInteractionType helper that maps RequestKind+intent to conversation-* values matching the server's documented vocabulary: Subagent → conversation-subagent, Background → conversation-background, Summarization → conversation-agent, Nes → conversation-other, MainAgent → location-derived (panel/inline/edits/agent/other). - Unify chatMLFetcher and networking.ts header paths via the helper, so X-Interaction-Type is now set unconditionally on every request (panel, inline, edits, etc.) instead of only for subagent/background/agent. - Telemetry property 'requestKind' now emits the resolved interaction type (matches the wire header value exactly) instead of the internal enum string. GDPR comments updated. - Flip newNotebookIntent (3 sites) and setupTestsInvocation (2 sites) — which run on the user's main agent model — from the Background default to MainAgent so they no longer report as background traffic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RequestKind is now { MainAgent, Subagent, Background } — 1:1 with the
server's X-Interaction-Type taxonomy. Summarization and NES sites use
MainAgent and rely on location to resolve to conversation-agent /
conversation-other respectively.
- summarizer.ts: location Panel -> Agent
- summarizedConversationHistory.tsx: location Other -> Agent
- agentIntent inline summarization: kind Summarization -> MainAgent
- xtabProvider / xtabNextCursorPredictor: kind Nes -> MainAgent
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
SummaryGoalClassify every Copilot chat request with a ChangesTelemetry (
Inverted default (
Subagent coverage
Summarization & NES rely on
Net effect
|
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Emit a new
requestKindproperty onresponse.success,response.cancelled, andresponse.errortelemetry events. Values are'background','subagent', or'normal', derived from the samerequestKindOptionsthat already sets the outgoingX-Interaction-Typeheader in CAPI requests.Motivation
Today we can tell from telemetry whether a chat request was initiated by the user vs. an agent tool call (via
initiatorType), but we can't distinguish which agent requests are background (speculative/system) vs. subagent (spawned by a parent agent) vs. normal primary-turn requests. That signal is already on the wire as theX-Interaction-Typeheader, but it isn't logged client-side. This makes it hard to slice perf / success metrics by request kind.Changes
extensions/copilot/src/extension/prompt/node/chatMLFetcherTelemetry.tsrequestKindOptionstoIChatMLFetcherSuccessfulData,IChatMLFetcherCancellationProperties,IChatMLFetcherErrorData.requestKind: requestKindOptions?.kind ?? 'normal'onresponse.success,response.cancelled,response.error.__GDPR__schema (classificationSystemMetaData, purposeFeatureInsight).extensions/copilot/src/extension/prompt/node/chatMLFetcher.tsrequestKindOptions(already destructured at the top offetchMany) into all five call sites: the success, cancel, and error calls infetchManyand_retryAfterError, plus the newprocessSuccessfulResponseparameter.Notes
initiatorTypein theirresponse.successGDPR schema but don't actually emit it today; intentionally leaving BYOK out of this PR to keep it scoped to the CAPI path.