Skip to content

Add requestKind to chat response telemetry events#312262

Open
kevin-m-kent wants to merge 10 commits intomicrosoft:mainfrom
kevin-m-kent:kevin-m-kent/telemetry-request-kind
Open

Add requestKind to chat response telemetry events#312262
kevin-m-kent wants to merge 10 commits intomicrosoft:mainfrom
kevin-m-kent:kevin-m-kent/telemetry-request-kind

Conversation

@kevin-m-kent
Copy link
Copy Markdown
Contributor

Emit a new requestKind property on response.success, response.cancelled, and response.error telemetry events. Values are 'background', 'subagent', or 'normal', derived from the same requestKindOptions that already sets the outgoing X-Interaction-Type header in CAPI requests.

Motivation

Today we can tell from telemetry whether a chat request was initiated by the user vs. an agent tool call (via initiatorType), but we can't distinguish which agent requests are background (speculative/system) vs. subagent (spawned by a parent agent) vs. normal primary-turn requests. That signal is already on the wire as the X-Interaction-Type header, but it isn't logged client-side. This makes it hard to slice perf / success metrics by request kind.

Changes

extensions/copilot/src/extension/prompt/node/chatMLFetcherTelemetry.ts

  • Adds requestKindOptions to IChatMLFetcherSuccessfulData, IChatMLFetcherCancellationProperties, IChatMLFetcherErrorData.
  • Emits requestKind: requestKindOptions?.kind ?? 'normal' on response.success, response.cancelled, response.error.
  • Adds the property to each event's __GDPR__ schema (classification SystemMetaData, purpose FeatureInsight).

extensions/copilot/src/extension/prompt/node/chatMLFetcher.ts

  • Threads requestKindOptions (already destructured at the top of fetchMany) into all five call sites: the success, cancel, and error calls in fetchMany and _retryAfterError, plus the new processSuccessfulResponse parameter.

Notes

  • BYOK (Anthropic / Gemini native) providers declare initiatorType in their response.success GDPR schema but don't actually emit it today; intentionally leaving BYOK out of this PR to keep it scoped to the CAPI path.
  • No behavior change; telemetry-only.

Emit a new 'requestKind' property on response.success, response.cancelled,
and response.error. Values: 'background', 'subagent', or 'normal' (derived
from the same requestKindOptions that sets the X-Interaction-Type header).

This lets us distinguish background and subagent requests from primary user
requests in telemetry, complementing the existing initiatorType (user/agent)
signal.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 23, 2026 23:34
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR extends Copilot chat response telemetry to include a requestKind dimension ('normal' | 'background' | 'subagent') on the CAPI path, derived from existing requestKindOptions that already influence the X-Interaction-Type header. This enables slicing success/cancel/error metrics by request kind without changing runtime behavior.

Changes:

  • Adds requestKindOptions to the telemetry payload interfaces and emits requestKind on response.success, response.cancelled, and response.error.
  • Updates the GDPR schemas for the three events to include requestKind (SystemMetaData / FeatureInsight).
  • Threads requestKindOptions through ChatMLFetcherImpl success/cancel/error telemetry call paths (including retry/error paths).
Show a summary per file
File Description
extensions/copilot/src/extension/prompt/node/chatMLFetcherTelemetry.ts Adds requestKind emission and GDPR schema entries for success/cancel/error response telemetry.
extensions/copilot/src/extension/prompt/node/chatMLFetcher.ts Passes requestKindOptions through to telemetry senders and into processSuccessfulResponse.

Copilot's findings

  • Files reviewed: 2/2 changed files
  • Comments generated: 0

@kevin-m-kent
Copy link
Copy Markdown
Contributor Author

@roblourens curious what you think of this. We are looking to filter out background requests in a robust way for our A/Bs, rather than maintain a list of sources. Copilot seems to think that this header for background interaction type never actually fires, but I'm suspicious of that claim.

kevin-m-kent and others added 8 commits April 29, 2026 10:32
…estId on response.cancelled/error

Inverts requestKind defaulting in chatMLFetcher.fetchMany so that any
request reaching the fetcher without an explicit requestKindOptions is
classified as 'background'. Adds 'mainagent' as the explicit opt-in for
primary user-driven turns. Now:

- defaultIntentRequestHandler passes { kind: 'subagent' } for subagents
  and { kind: 'mainagent' } for normal user turns.
- inlineChatIntent (both chat2 and inline edit paths) passes
  { kind: 'mainagent' }.
- All other callers (title gen, summarization, intent detection, commit
  message gen, terminal fix gen, PR title/desc gen, rename suggestions,
  semantic search, virtualToolGrouper, etc.) automatically get
  'background' without per-call edits.

Also adds parentRequestId and conversationId to response.success,
response.cancelled, and response.error events so subagent requests can
be joined back to the main agent request, matching the
panel.request schema.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Conversation history summarization is in the critical path of a user
turn (it gates the next agent step), so classifying it as 'background'
alongside truly out-of-band utility calls (chat title gen, intent
detection, commit message gen, etc.) loses an important signal.

Adds a 'summarization' kind and marks both summarization sites
(SummarizedConversationHistory full/simple path and AgentIntent
inline path) explicitly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- executionSubagentToolCallingLoop: explicitly set kind: 'subagent'
  (was defaulting to 'background')
- ChatSummarizerProvider (vscode.ChatSummarizer): kind: 'summarization'
  (was defaulting to 'background'). Switched the call from positional
  makeChatRequest to makeChatRequest2 so requestKindOptions can be passed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collapse the four IBackgroundRequestOptions / ISubagentRequestOptions /
ISummarizationRequestOptions / IMainAgentRequestOptions interfaces into
a single 'as const' object + derived type, matching the existing
ToolName / ContributedToolName pattern in this codebase. Call sites now
use RequestKind.MainAgent etc. instead of hand-typing 'mainagent', so a
future rename only has to update the RequestKind definition.

Compile-time safety is unchanged — IRequestKindOptions.kind is still
typed as the union of literals.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# Conflicts:
#	extensions/copilot/src/extension/inlineChat2/node/inlineChatIntent.ts
xtabProvider and xtabNextCursorPredictor were defaulting to 'background'
but they're a distinct workload (speculative inline edit prediction with
different cost/latency/cache characteristics than other utility
callers). Tag them as 'nes' so telemetry can slice them out.

Header behavior unchanged — 'nes' falls through to the location-derived
intent on X-Interaction-Type, like 'mainagent' and 'summarization'.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add resolveInteractionType helper that maps RequestKind+intent to
  conversation-* values matching the server's documented vocabulary:
  Subagent → conversation-subagent, Background → conversation-background,
  Summarization → conversation-agent, Nes → conversation-other,
  MainAgent → location-derived (panel/inline/edits/agent/other).
- Unify chatMLFetcher and networking.ts header paths via the helper, so
  X-Interaction-Type is now set unconditionally on every request (panel,
  inline, edits, etc.) instead of only for subagent/background/agent.
- Telemetry property 'requestKind' now emits the resolved interaction
  type (matches the wire header value exactly) instead of the internal
  enum string. GDPR comments updated.
- Flip newNotebookIntent (3 sites) and setupTestsInvocation (2 sites) —
  which run on the user's main agent model — from the Background default
  to MainAgent so they no longer report as background traffic.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RequestKind is now { MainAgent, Subagent, Background } — 1:1 with the
server's X-Interaction-Type taxonomy. Summarization and NES sites use
MainAgent and rely on location to resolve to conversation-agent /
conversation-other respectively.

- summarizer.ts: location Panel -> Agent
- summarizedConversationHistory.tsx: location Other -> Agent
- agentIntent inline summarization: kind Summarization -> MainAgent
- xtabProvider / xtabNextCursorPredictor: kind Nes -> MainAgent

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@kevin-m-kent
Copy link
Copy Markdown
Contributor Author

kevin-m-kent commented May 1, 2026

Summary

Goal

Classify every Copilot chat request with a requestKind for telemetry and align the X-Interaction-Type header sent to CAPI with the server's documented taxonomy, so each request surface — primary user turns, subagent calls, background utility work (title gen, commit messages, intent detection, language-model passthroughs, tool-call healers, etc.), summarization, and NES — is independently distinguishable in CAPI logs and our telemetry.

Changes

Telemetry (chatMLFetcherTelemetry.ts)

  • Added two properties to the chat response telemetry events:
    • requestKind — the resolved conversation-* value (matches what's sent on the wire).
    • parentRequestId — links a sub-call (subagent invocation, summarization, intent detection, healer retry, etc.) to the user turn that spawned it. Sourced from IChatMLFetcher options threaded through IMakeChatRequestOptions.
  • These were already on panel.request for the user-facing turn but were missing from response.success, response.cancelled, and response.error. Coverage is now uniform across all three terminal events, so a turn with a cancellation or filter/error mid-cascade can still be reconstructed.

RequestKind (networking.ts)

  • Introduced a RequestKind const object — three values that map 1:1 to the server's X-Interaction-Type taxonomy:
    • MainAgent → resolved from ChatLocation (conversation-panel / conversation-inline / conversation-edits / conversation-agent / conversation-other / etc.)
    • Subagentconversation-subagent
    • Backgroundconversation-background
  • Added a resolveInteractionType(kind, intent) helper used by both the websocket and HTTP fetcher paths so X-Interaction-Type is always sent and always matches the telemetry value.

Inverted default (chatMLFetcher.ts)

  • Requests that don't explicitly opt in to a kind now classify as Background (previously fell through to mainagent/normal). This correctly labels ~40 utility callers in telemetry without per-site annotation.
  • Primary user-initiated paths explicitly opt in to MainAgent: defaultIntentRequestHandler, inlineChat2Intent, newNotebookIntent (×3), setupTestsInvocation (×2).

Subagent coverage

  • executionSubagentToolCallingLoop now opts in to Subagent (was missing; companion searchSubagentToolCallingLoop already had it).

Summarization & NES rely on ChatLocation (no dedicated kind)

  • Summarization (summarizer.ts, summarizedConversationHistory.tsx, agentIntent inline summarization): MainAgent + ChatLocation.Agentconversation-agent. (summarizer.ts and summarizedConversationHistory.tsx had their location updated from Panel/OtherAgent so the fallthrough lands on the right value.)
  • NES (xtabProvider, xtabNextCursorPredictor): MainAgent with existing ChatLocation.Otherconversation-other.

Net effect

  • Every request emits a stable requestKind, parentRequestId, and conversationId on its response telemetry event (success / cancelled / error) and a matching X-Interaction-Type header on the wire.
  • Utility/helper callers that previously sent no header (websocket path) or a misleading conversation-panel (HTTP path) now correctly send conversation-background.
  • Main-agent, subagent, summarization (conversation-agent), inline-edit, NES (conversation-other), and panel/ask requests are each distinguishable in CAPI logs and in our telemetry, and sub-calls can be joined back to their parent turn / conversation regardless of outcome.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants