🤖 Add truncation: auto to OpenAI Responses API #87
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ammario
force-pushed
the
oai-truncate
branch
2 times, most recently
from
45d44e4 to
6168efc
Compare
Member
Author
SDK Investigation ResultsAfter investigating the Vercel AI SDK source code ( v2.0.40), I discovered that the What I Found:
Code Evidence:// From @ai-sdk/openai/dist/index.js line 2586-2588
...modelConfig.requiredAutoTruncation && {
truncation: "auto"
}The SDK uses explicit mapping for provider options - it doesn't spread the entire Current Status:
Next Steps:We should file an issue/PR with the Vercel AI SDK to add
Generated with |
Member
Author
Update: ForceImplemented a fetch wrapper that injects into every Key safeguards:
All lint/type checks pass. _Generated with _ |
Enables automatic conversation truncation for OpenAI Responses API to prevent context overflow errors. When set to 'auto', the API will automatically drop input items in the middle of the conversation to fit within the model's context window. This prevents failures when the conversation history exceeds the available context size and allows long conversations to continue seamlessly. _Generated with `cmux`_
Clarifies that the /truncate command does not work with OpenAI models due to the Responses API's server-side conversation state management. Explains that OpenAI uses automatic truncation instead and provides workarounds for users (/clear, /compact, or relying on auto-truncation). _Generated with `cmux`_
Extended OpenAIResponsesProviderOptions type to include the truncation parameter since the SDK types don't yet include it, but it's supported by the OpenAI Responses API. _Generated with `cmux`_
Wrap the OpenAI provider fetch to inject on every Responses API call. This ensures automatic conversation truncation works immediately, regardless of @ai-sdk/openai support. _Generated with _
Add disableAutoTruncation flag to SendMessageOptions for testing context overflow behavior. Test verifies: 1. Context limit exceeded when auto-truncation disabled 2. Successful recovery with auto-truncation enabled Test sends large messages (~10k tokens each) to trigger 128k context limit, then verifies truncation:auto allows continuation. Will run in CI with API keys. _Generated with `cmux`_
ammar-agent
force-pushed
the
oai-truncate
branch
from
d18a38b to
eb6ff33
Compare
_Generated with `cmux`_
The test now properly waits for either stream-end or stream-error events during the context overflow phase, rather than assuming only IPC-level errors. This matches OpenAI's actual behavior where context exceeded errors are sent as stream-error events. _Generated with `cmux`_
The existing token limit error test was failing because auto-truncation is now enabled by default for OpenAI, preventing the expected context errors from occurring. This commit adds disableAutoTruncation: true to that test to preserve its original behavior. _Generated with `cmux`_
ammario
commented
Oct 8, 2025
tests/ipcMain/sendMessage.test.ts
Outdated
| // Send up to 20 large messages (200k tokens total) | ||
| // Should exceed 128k context limit and trigger error | ||
| for (let i = 0; i < 20; i++) { | ||
| const result = await sendMessageWithModel( |
Member
Author
There was a problem hiding this comment.
use historyService to fill up context to reduce API costs... see sendMessage e2e tests for example
Collaborator
There was a problem hiding this comment.
✅ Done! Updated the test to use HistoryService to populate history directly, reducing from 20 API calls to just 2. Test timeout reduced from 180s to 60s.
Instead of making 20 API calls to fill up the context window, directly use HistoryService to populate the conversation history. This reduces API costs significantly while still testing the auto-truncation behavior. Changes: - Use HistoryService.appendToHistory() to inject 12 large messages - Reduced from 20 API calls to just 2 (one to trigger error, one to verify success) - Reduced timeout from 180s to 60s (much faster execution) - Follows same pattern as existing token limit error test _Generated with `cmux`_
Collaborator
|
✅ Review comment addressed in commit c5fdf66 - test now uses HistoryService to reduce API costs |
The test was only using 12 messages, which wasn't enough to exceed the context window. Updated to use 80 messages (4M chars) to match the token limit error test for OpenAI. _Generated with `cmux`_
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prepares support for automatic conversation truncation with OpenAI Responses API.
truncationparameter from provider options. See investigation comment below for details.Changes
truncation: "auto"parameter to OpenAI provider options inbuildProviderOptions()/truncatecommandCurrent Behavior
/clearor/compactcommands to manage conversation historyNext Steps
File an issue/PR with Vercel AI SDK to add
truncationto the provider options mapping.Generated with
cmux