Harden Language Model Tool telemetry against PII leaks#1644
Merged
Conversation
Centralise all LMT telemetry through src/lmToolTelemetry.ts so user-supplied strings (target, expression, sessionName, file paths, class names, JVM stack traces, etc.) can no longer reach the telemetry pipeline. The new module exposes a typed sanitizedSend choke point that only accepts enums, booleans, numbers and opaque session IDs. Telemetry changes: - Drop sendError(error) on debug_java_application failure (stack trace leaked user class / method names). - Strip PII fields from every existing event: target, sessionName, currentFile, currentLine, simpleClassName, detectedClassName, error: String(error), input.reason. - Replace bare String(error) propagation with classifyError() -> ErrorCategory enum (mainClassMissing, classpathUnresolved, buildFailure, projectNotDetected, sessionAlreadyRunning, timeout, lsNotReady, noActiveSession, noSuspendedThread, noStackFrame, cancelled, other). - Add per-invoke recording for all 10 tools with outcome, errorCategory, durationMs, and a tool-specific enum (targetType / breakpointKind / stepKind / scopeType / evalContext / removeScope). The previous build only emitted telemetry on the launch tool and the session-info tool. - Add chatActivationSnapshot one-shot at registration time so we can measure adoption of the chat surfaces without per-turn cost (counts only). - evaluate_debug_expression: the expression text is NEVER logged. Only the evalContext enum and outcome are emitted. Policy: - src/lmToolTelemetry.ts is now the only file in the LMT code path allowed to call sendInfo. The top-of-file policy comment is the single source of truth for what may be logged. - The recorder is typed against ToolInvocationRecord so excess raw strings are rejected at compile time. Validated with: npm run tslint, npm run compile.
wenytang-ms
requested review from
chagong,
jdneo and
testforstephen
as code owners
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Centralizes Language Model Tool (LMT) telemetry behind a new sanitized sender to reduce risk of PII leaking into telemetry, while expanding per-tool invocation metrics (outcome/error category/duration and tool-specific enums).
Changes:
- Added
src/lmToolTelemetry.tswith classification helpers and arecordToolInvocation“choke point” for telemetry. - Replaced ad-hoc
sendInfo/sendErrorusage in LMT tools with sanitized recording and error classification. - Added a one-shot chat activation snapshot during extension activation.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| src/lmToolTelemetry.ts | Introduces telemetry policy, classifiers, and sanitized recording APIs. |
| src/languageModelTool.ts | Routes tool telemetry through recordToolInvocation and removes raw string properties from events. |
| src/extension.ts | Emits a one-shot “chat activation snapshot” telemetry event with counts only. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- classifyStep: unknown step operations now report 'unknown' instead of being silently mislabeled as 'over'. Also adds a runtime guard in debug_step_operation so an unknown operation no longer reaches commandMap[op]/executeCommand(undefined) or session.customRequest with an arbitrary string.
- recordToolInvocation: introduces a private normalizeToolInvocationRecord that keeps 'outcome' and 'errorCategory' in lock-step for the six shared terminal values (cancelled / timeout / lsNotReady / noActiveSession / noSuspendedThread / noStackFrame). Fixes the case where debug_java_application returns {success:false,message:'Operation cancelled by user'} but outcome was 'failure' while errorCategory was 'cancelled'.
- get_debug_stack_trace: empty-stack-frame early return now sets errorCategory='noStackFrame' alongside outcome (was only setting outcome).
- recordLaunchInternal: signature is now a discriminated union (LaunchInternalEvent) instead of (operationName: string, properties: Record<string, SafeValue>). Unknown event names and unexpected property keys are now rejected at compile time. Updated all 8 call sites.
- elapsedTime (string from .toFixed) split from elapsedMs (number) so the telemetry value is numeric and aggregable.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
chagong
approved these changes
May 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Centralise all LMT telemetry through src/lmToolTelemetry.ts so user-supplied strings (target, expression, sessionName, file paths, class names, JVM stack traces, etc.) can no longer reach the telemetry pipeline. The new module exposes a typed sanitizedSend choke point that only accepts enums, booleans, numbers and opaque session IDs.
Telemetry changes:
Drop sendError(error) on debug_java_application failure (stack trace leaked user class / method names).
Strip PII fields from every existing event: target, sessionName, currentFile, currentLine, simpleClassName, detectedClassName, error: String(error), input.reason.
Replace bare String(error) propagation with classifyError() -> ErrorCategory enum (mainClassMissing, classpathUnresolved, buildFailure, projectNotDetected, sessionAlreadyRunning, timeout, lsNotReady, noActiveSession, noSuspendedThread, noStackFrame, cancelled, other).
Add per-invoke recording for all 10 tools with outcome, errorCategory, durationMs, and a tool-specific enum (targetType / breakpointKind / stepKind / scopeType / evalContext / removeScope). The previous build only emitted telemetry on the launch tool and the session-info tool.
Add chatActivationSnapshot one-shot at registration time so we can measure adoption of the chat surfaces without per-turn cost (counts only).
evaluate_debug_expression: the expression text is NEVER logged. Only the evalContext enum and outcome are emitted.
Policy:
src/lmToolTelemetry.ts is now the only file in the LMT code path allowed to call sendInfo. The top-of-file policy comment is the single source of truth for what may be logged.
The recorder is typed against ToolInvocationRecord so excess raw strings are rejected at compile time.
Validated with: npm run tslint, npm run compile.