Experiment with terminal output deltas for repeated polls

[kevin-m-kent]

This adds an experimental chat.tools.terminal.outputDeltas setting for repeated get_terminal_output polling.

When the experiment is enabled:

• The first poll for a terminal execution returns the full output, preserving current context.
• Repeated polls with identical output return a short unchanged marker instead of replaying the full terminal buffer.
• Polls with appended output return only the new suffix.
• Non-prefix changes, such as screen rewrites or truncated scrollback, fall back to returning the current full output.

This change is stateful across explicit get_terminal_output calls and targets prompt/tool-result replay from terminal polling loops.

[Copilot]

Pull request overview

Adds an experimental, stateful “output delta” mode for get_terminal_output polling to reduce repeated prompt/tool-result size when terminal output is unchanged or only appended.

Changes:

• Introduces chat.tools.terminal.outputDeltas (experimental, auto-enabled experiment flag) configuration setting.
• Updates GetTerminalOutputTool to optionally return unchanged markers or appended-output suffixes on repeated polls (falling back to full output for non-prefix changes).
• Adds browser tests covering unchanged output, appended output, and non-prefix rewrite fallback behavior.

Show a summary per file

File	Description
src/vs/workbench/contrib/terminalContrib/chatAgentTools/test/browser/getTerminalOutputTool.test.ts	Adds tests validating unchanged-marker, delta-suffix, and fallback-to-full-output behaviors when the experiment is enabled.
src/vs/workbench/contrib/terminalContrib/chatAgentTools/common/terminalChatAgentToolsConfiguration.ts	Registers the new chat.tools.terminal.outputDeltas experimental setting.
src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/getTerminalOutputTool.ts	Implements stateful output snapshotting and delta formatting behind the new configuration flag.

Copilot's findings

• Files reviewed: 3/3 changed files
• Comments generated: 0

[kevin-m-kent]

Validated that this is working as expected locally with the flag. Everything after the initial terminal poll returns a message like when the terminal hasn't updated since the last poll:

Output of terminal f7a744dc-de40-46e7-a631-776c4d73ea22 unchanged since previous poll (161 characters already shown). No new output.

[meganrogge]

The map stores full output strings (potentially large) for up to 100 terminals, with no cleanup on terminal disposal. Stale entries only get evicted when the cap is hit via FIFO.

Suggestions:

1. Clear on dispose — override dispose() to call this._lastOutputByTerminalId.clear().
2. Listen for terminal disposal — register a listener (e.g. onDidDisposeInstance) to delete the entry for that terminal ID, so we don't hold output strings for terminals that no longer exist.
3. Consider storing a hash + length instead of the full string — if we only need the "unchanged" and "startsWith" checks, we could store a hash of the previous output and the length. We'd lose the ability to do the prefix check cheaply, but it would drastically reduce memory. Alternatively, keep the current approach but lower the cap or add a per-entry size limit.

[meganrogge]

Thanks for this! See my comment

[kevin-m-kent]

The map stores full output strings (potentially large) for up to 100 terminals, with no cleanup on terminal disposal. Stale entries only get evicted when the cap is hit via FIFO.

Suggestions:

1. Clear on dispose — override dispose() to call this._lastOutputByTerminalId.clear().
2. Listen for terminal disposal — register a listener (e.g. onDidDisposeInstance) to delete the entry for that terminal ID, so we don't hold output strings for terminals that no longer exist.
3. Consider storing a hash + length instead of the full string — if we only need the "unchanged" and "startsWith" checks, we could store a hash of the previous output and the length. We'd lose the ability to do the prefix check cheaply, but it would drastically reduce memory. Alternatively, keep the current approach but lower the cap or add a per-entry size limit.

Thanks @meganrogge - I think these are all great suggestions. I made some updates that I believe addresses your feedback and tested again locally. For #3 I went with the hash approach.

[bhavyaus]

Probably not a big deal but worth checking if it works across compaction boundaries since we lose the full tool output in that case if the model reuses terminals/kicks off async tasks, hits compaction, and then resumes tasks

[kevin-m-kent]

Probably not a big deal but worth checking if it works across compaction boundaries since we lose the full tool output in that case if the model reuses terminals/kicks off async tasks, hits compaction, and then resumes tasks

I was chatting about this with @isidorn as well. I'm not sure how we solve this scenario, other than relying on the compaction call to preserve what is relevant from the previous tool result.

[isidorn]

relying on the compaction call to preserve what is relevant from the previous tool result.

I think this is a fair assumption to make.

[meganrogge]

/requires-eval-assessment terminalbench2 gpt-5.4,claude-opus-4.6,claude-opus-4.7

[meganrogge]

Seeing how this impacts evals

[vs-code-engineering]

⏳ Queued vscode build for f0bd87449265b04584b1d082863a7bd1ef6d01b6 (step 1/2).

• Build: https://dev.azure.com/monacotools/Monaco/_build/results?buildId=438325
• When this succeeds, the eval-assessment publish build will be queued automatically.

[isidorn]

Pretty cool! I did not know we have this evals integration!

Also consider doing evals for GPT-5.5 as well, since that is the latest OpenAI model.

[meganrogge]

/requires-eval-assessment terminalbench2 gpt-5.4,claude-opus-4.6,claude-opus-4.7,gpt-5.5

[vs-code-engineering]

⏳ Queued vscode build for f0bd87449265b04584b1d082863a7bd1ef6d01b6 (step 1/2).

• Build: https://dev.azure.com/monacotools/Monaco/_build/results?buildId=438656
• When this succeeds, the eval-assessment publish build will be queued automatically.

[vs-code-engineering]

🚀 Queued eval-assessment publish build for 858d4286b1e2e7ae6586ed44dad2c11af46cab92 (step 2/2).

• Pipeline run: https://dev.azure.com/monacotools/Monaco/_build/results?buildId=438695
• On success, publishes @vscode/vscode-copilot-evaluation-agent@0.0.0-dev.858d4286b1 on the dev tag.

[vs-code-engineering]

🔬 Queued eval-assessment benchmark for 95df7cf309.

• Package: @vscode/vscode-copilot-evaluation-agent@0.0.0-dev.858d4286b1 (dev tag)
• Benchmark: terminalbench2
• Tracking issues:
  • terminalbench2 / gpt-5.4: https://github.com/github/evald/issues/22421
  • terminalbench2 / claude-opus-4.6: https://github.com/github/evald/issues/22422
  • terminalbench2 / claude-opus-4.7: https://github.com/github/evald/issues/22423
  • terminalbench2 / gpt-5.5: https://github.com/github/evald/issues/22424

Results will be posted back here when the run completes.

[vs-code-engineering]

✅ Eval-assessment build published.

• Package: @vscode/vscode-copilot-evaluation-agent@0.0.0-dev.858d4286b1 (tag: dev)
• Install: npm install @vscode/vscode-copilot-evaluation-agent@0.0.0-dev.858d4286b1
• Pipeline run: https://dev.azure.com/monacotools/Monaco/_build/results?buildId=438695

[isidorn]

@jukasper I tried to access the https://github.com/github/evald/ links but I do not seem to have access. Can you add me please?

[meganrogge]

Repeating as we saw a regression on Claude Opus 4.6, but it was not huge.

[vs-code-engineering]

⏳ Queued vscode build for f0bd87449265b04584b1d082863a7bd1ef6d01b6 (step 1/2).

• Build: https://dev.azure.com/monacotools/Monaco/_build/results?buildId=438904
• When this succeeds, the eval-assessment publish build will be queued automatically.

[vs-code-engineering]

🚀 Queued eval-assessment publish build for 537b6f0cc8115f78a903d0e1b2ab2706423aa2cc (step 2/2).

• Pipeline run: https://dev.azure.com/monacotools/Monaco/_build/results?buildId=438928
• On success, publishes @vscode/vscode-copilot-evaluation-agent@0.0.0-dev.537b6f0cc8 on the dev tag.

[vs-code-engineering]

🔬 Queued eval-assessment benchmark for 641f97f4c9.

• Package: @vscode/vscode-copilot-evaluation-agent@0.0.0-dev.537b6f0cc8 (dev tag)
• Benchmark: terminalbench2
• Tracking issues:
  • terminalbench2 / gpt-5.4: https://github.com/github/evald/issues/22519
  • terminalbench2 / claude-opus-4.6: https://github.com/github/evald/issues/22520
  • terminalbench2 / claude-opus-4.7: https://github.com/github/evald/issues/22521
  • terminalbench2 / gpt-5.5: https://github.com/github/evald/issues/22522

Results will be posted back here when the run completes.

[vs-code-engineering]

✅ Eval-assessment build published.

• Package: @vscode/vscode-copilot-evaluation-agent@0.0.0-dev.537b6f0cc8 (tag: dev)
• Install: npm install @vscode/vscode-copilot-evaluation-agent@0.0.0-dev.537b6f0cc8
• Pipeline run: https://dev.azure.com/monacotools/Monaco/_build/results?buildId=438928

[meganrogge]

Thanks! Let's see how it goes 😄 . We'll want to kick off evals with and without this setting too.

[vs-code-engineering]

📊 Eval-assessment benchmark complete.

• Tracking issue: https://github.com/github/evald/issues/22522
• Publish build: https://dev.azure.com/monacotools/Monaco/_build/results?buildId=438928

🧪 Results

[vs-code-engineering]

📊 Eval-assessment benchmark complete.

• Tracking issue: https://github.com/github/evald/issues/22521
• Publish build: https://dev.azure.com/monacotools/Monaco/_build/results?buildId=438928

🧪 Results

[vs-code-engineering]

📊 Eval-assessment benchmark complete.

• Tracking issue: https://github.com/github/evald/issues/22520
• Publish build: https://dev.azure.com/monacotools/Monaco/_build/results?buildId=438928

🧪 Results

[vs-code-engineering]

📊 Eval-assessment benchmark complete.

• Tracking issue: https://github.com/github/evald/issues/22519
• Publish build: https://dev.azure.com/monacotools/Monaco/_build/results?buildId=438928

🧪 Results