Skip to content

Compress large telemetry properties with gzip chunked columns - #327490

Merged
vijayupadya merged 5 commits into
mainfrom
fun-albatross
Jul 27, 2026
Merged

Compress large telemetry properties with gzip chunked columns#327490
vijayupadya merged 5 commits into
mainfrom
fun-albatross

Conversation

@vijayupadya

@vijayupadya vijayupadya commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Fixes https://github.com/microsoft/vscode-internalbacklog/issues/8434

What

Adds gzip + base64 compression for oversized telemetry properties in both the Copilot extension and the agent host, replacing the previous plain string-splitting for known-large fields.

When a property exceeds MAX_PROPERTY_LENGTH (8192 chars) — or is one of the always-compressed keys (messagesJson, diffsJSON) — the original column now carries only the first uncompressed 8192-char slice, and the full value is gzip-compressed, base64-encoded, and emitted across <key>Chunk, <key>Chunk_2, … columns (up to MAX_CONCATENATED_PROPERTIES). Values under the limit that aren't force-compressed are unchanged, and when no compressor is available the code falls back to the original plain continuation family.

Why

Large prompt/message/diff payloads previously inflated telemetry volume as raw split strings. Compressing them significantly reduces the emitted size while keeping a uniform, backend-readable chunk layout.

How

  • multiplexProperties is now async and awaits a pluggable compressor. The extension registers a gzip (via util.promisify(zlib.gzip)) + base64 compressor through setTelemetryPropertyCompressor; the agent host uses the same approach via compressTelemetryValue.
  • All call sites that build telemetry properties now use a fire-and-forget pattern (void multiplexProperties(...).then(send).catch(() => { /* best-effort telemetry */ })) so gzip runs off the event loop and never blocks hot paths (request dispatch, tool calls, inline edits).
  • The agent host telemetry reporter and GitHub telemetry router were made async to await multiplexing before routing.

Testing

  • Updated telemetry.spec.ts compression tests to async and added coverage for the chunked-column layout.
  • Updated agent host tests (agentHostGitHubTelemetryRouter, agentHostRepoInfoTelemetry, agentHostTelemetryReporter) for the async flow.
  • All telemetry unit tests pass; no compile errors.

Copilot AI review requested due to automatic review settings July 25, 2026 21:13

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds gzip/base64 compression and chunking for oversized telemetry properties across Agent Host and Copilot extension telemetry paths.

Changes:

  • Introduces asynchronous compressed property multiplexing.
  • Updates telemetry senders and routers for asynchronous processing.
  • Adds compression and round-trip tests.
Show a summary per file
File Description
agentHostTelemetryReporter.test.ts Updates reporter tests for async compression.
agentHostRepoInfoTelemetry.test.ts Updates reporter mocks to async.
agentHostGitHubTelemetryRouter.test.ts Tests compressed routing and round-tripping.
agentHostTelemetryReporter.ts Compresses restricted telemetry properties.
agentHostRestrictedTelemetry.ts Implements Agent Host gzip chunking.
agentHostGitHubTelemetryRouter.ts Makes telemetry routing asynchronous.
telemetryServiceImpl.ts Registers the Node gzip compressor.
telemetry.spec.ts Adds compression behavior tests.
telemetry.ts Implements shared compressed multiplexing.
chatStream.ts Sends multiplexed telemetry asynchronously.
multiFileEditQualityTelemetry.ts Adapts edit telemetry to async multiplexing.
applyPatchTool.tsx Adapts patch telemetry to async multiplexing.
abstractReplaceStringTool.tsx Adapts replacement telemetry to async multiplexing.
codeMapper.ts Adapts successful-edit telemetry.
repoInfoTelemetry.ts Adapts repository telemetry.
chatMLFetcher.ts Adapts tool-option telemetry.
nextEditProviderTelemetry.ts Adapts next-edit telemetry.
continuousEnhancedTelemetrySender.ts Adapts continuous telemetry.

Review details

  • Files reviewed: 18/18 changed files
  • Comments generated: 4
  • Review effort level: Medium

Comment thread extensions/copilot/src/platform/telemetry/common/telemetry.ts
Comment thread src/vs/platform/agentHost/node/agentHostRestrictedTelemetry.ts
Comment thread src/vs/platform/agentHost/node/agentHostTelemetryReporter.ts Outdated
Comment thread src/vs/platform/agentHost/node/agentHostGitHubTelemetryRouter.ts Outdated
# Conflicts:
#	src/vs/platform/agentHost/node/agentHostGitHubTelemetryRouter.ts
#	src/vs/platform/agentHost/node/agentHostTelemetryReporter.ts
#	src/vs/platform/agentHost/node/copilot/copilotAgent.ts
#	src/vs/platform/agentHost/node/copilot/copilotAgentSession.ts
#	src/vs/platform/agentHost/test/node/agentHostGitHubTelemetryRouter.test.ts
#	src/vs/platform/agentHost/test/node/agentHostTelemetryReporter.test.ts
@vijayupadya
vijayupadya marked this pull request as ready for review July 27, 2026 17:18
zhichli
zhichli previously approved these changes Jul 27, 2026
@vijayupadya
vijayupadya marked this pull request as draft July 27, 2026 18:02
# Conflicts:
#	src/vs/platform/agentHost/node/agentHostTelemetryReporter.ts
#	src/vs/platform/agentHost/node/copilot/copilotAgentSession.ts
#	src/vs/platform/agentHost/test/node/agentHostTelemetryReporter.test.ts
@vijayupadya
vijayupadya marked this pull request as ready for review July 27, 2026 19:16
@vijayupadya
vijayupadya enabled auto-merge (squash) July 27, 2026 19:33
@vijayupadya
vijayupadya merged commit 7d6473c into main Jul 27, 2026
29 checks passed
@vijayupadya
vijayupadya deleted the fun-albatross branch July 27, 2026 19:41
@vs-code-engineering vs-code-engineering Bot added this to the 1.132.0 milestone Jul 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants