Reinforce timeout + do-not-poll guidance in terminal tool descriptions#320141
Merged
Conversation
Contributor
📬 CODENOTIFYThe following users are being notified based on files changed in this PR: @anthonykim1Matched files:
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the terminal chat agent tool descriptions to more strongly discourage polling for terminal output and to steer agents toward using sufficiently generous timeout values so sync commands don’t get promoted to background unexpectedly.
Changes:
- Strengthen
run_in_terminal.timeoutparameter guidance with concrete “generous timeout” examples and explicit warning about short timeouts causing background promotion/polling behavior. - Expand
get_terminal_output’smodelDescriptionwith “do not poll; wait for notification” guidance at the tool decision point.
Show a summary per file
| File | Description |
|---|---|
| src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/runInTerminalTool.ts | Updates the timeout parameter description to steer agents toward longer timeouts and avoid timeout-driven background promotion/polling. |
| src/vs/workbench/contrib/terminalContrib/chatAgentTools/browser/tools/getTerminalOutputTool.ts | Updates the tool description to discourage polling and instruct waiting for completion notifications. |
Copilot's findings
- Files reviewed: 2/2 changed files
- Comments generated: 2
- getTerminalOutput: clarify valid use case (async output inspection) before the do-not-poll guidance - timeout: restore 'Optional' label, add human-readable durations (600000 = 10 min, 900000 = 15 min), and explain background promotion in plain terms the model can reason about Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- getTerminalOutput: cover both async-mode and timed-out-sync use cases instead of restricting to async-only - timeout: restructure to lead with 'if you set a timeout, be generous' so the numeric recommendations are clearly conditional, avoiding contradiction with the omit-entirely guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
/requires-eval-assessment terminalbench2 gpt-5.4,claude-opus-4.6,claude-opus-4.7 |
meganrogge
added
the
~requires-eval-assessment
Contributor
|
⏳ Queued vscode build for
|
Contributor
|
🔄 First vscode build failed; retried failed stages: https://dev.azure.com/monacotools/Monaco/_build/results?buildId=445312 |
vs-code-engineering
Bot
removed
the
~requires-eval-assessment
lramos15
approved these changes
Jun 5, 2026
meganrogge
added
the
~requires-eval-assessment
Contributor
|
⏳ Queued vscode build for
|
Contributor
|
🚀 Queued eval-assessment publish build for
|
Contributor
|
✅ Eval-assessment build published.
|
vs-code-engineering
Bot
removed
the
~requires-eval-assessment
Contributor
|
🔬 Queued eval-assessment benchmark for
Results will be posted back here when the run completes. |
Contributor
|
📊 Eval-assessment benchmark complete.
🧪 Results |
Contributor
|
📊 Eval-assessment benchmark complete.
🧪 Results |
Contributor
|
📊 Eval-assessment benchmark complete.
🧪 Results |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses microsoft/vscode-internalbacklog#7870.
The agent has been observed polling
get_terminal_outputfor ~27 steps across threenpmcommands when a too-short timeout (120s for a 3–5 minute install) caused sync→background promotion. The system prompt preamble already says "do NOT poll", andrunInTerminal'smodelDescriptionalready recommends generous timeouts — but neither of those signals attaches to the spot where the model is actually making the decision (thetimeoutparameter description forrunInTerminal, and the top-level description forget_terminal_output).This PR reinforces both signals at the point of use:
runInTerminal→timeoutparameter descriptionBefore:
After:
getTerminalOutput→modelDescriptionBefore:
After:
Why not pull these from the system prompt preamble?
The preamble already contains both pieces of guidance, but models attend most strongly to tool parameter descriptions at call time, and the preamble has to compete with many other paragraphs. Restating the rule at the decision point is what gets it followed.
Risks / follow-ups
Tests
No assertions reference these exact strings (verified via grep for
"Optional hard cap in milliseconds"and"Get output from an active terminal"). No test changes required.