🤖 Add integration test for OpenAI reasoning error #73
+462
−11
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ammario
force-pushed
the
openai-reasoning-test
branch
5 times, most recently
from
e82a50a to
6b32d5e
Compare
This test reproduces the intermittent error: "Item 'rs_*' of type 'reasoning' was provided without its required following item"⚠️ DISABLED BY DEFAULT - Opt-in via OPENAI_REASONING_TEST_RUNS env variable Key features (based on cmux-docs-style analysis): - Uses ONLY read_file tool (safety - no file modifications) - Sets reasoning effort to HIGH (maximize reasoning content) - Uses aggressive prompts that trigger extensive reasoning (3800+ parts observed in real error) - Tests pattern: user → assistant (reasoning + tools) → tool results → user (follow-up) Rationale for opt-in: - Makes real API calls (costs money) - Specifically for reproducing an intermittent error, not for validating functionality - Should be run manually when investigating the bug Usage: OPENAI_REASONING_TEST_RUNS=10 TEST_INTEGRATION=1 bun x jest tests/ipcMain/openaiReasoning.test.ts The test: - Runs N attempts to trigger the intermittent error - For each attempt, creates fresh workspace - Sends 3 messages with prompts designed to trigger extensive reasoning - Tool policy limits to read_file only (safety) - Thinking level set to 'high' for maximum reasoning - Checks for the specific error in stream events - Reports whether error was reproduced Generated with `cmux`
Added second test that replays actual chat history from cmux-docs-style workspace which reliably reproduces the OpenAI reasoning error. Benefits: - Deterministic reproduction (no random prompts) - Real-world conversation pattern that actually triggered the bug - Faster debugging since we know exactly what causes it The test: - Loads chat.jsonl from tests/testdata/openai-reasoning-error-repro.jsonl - Replays the 3 user messages in sequence - Tests against the actual conversation that had 3800+ reasoning parts - Should reliably trigger the error Both tests are opt-in (require OPENAI_REASONING_TEST_RUNS env variable). Generated with `cmux`
Generated with `cmux`
## Changes - Update AI SDK from 5.0.56 to 5.0.60 - Update @ai-sdk/openai from 2.0.40 to 2.0.44 - Remove include: ['reasoning.encrypted_content'] from OpenAI provider options ## Reasoning Based on OpenAI documentation and AI SDK guidelines, encrypted reasoning content is only required when store: false (for ZDR compliance). Since we use store: true (default) with previousResponseId, OpenAI manages reasoning state automatically via the response ID. ## Known Issue Despite this configuration, there is an intermittent Vercel AI SDK bug that causes 'reasoning without following item' errors with OpenAI reasoning models when using tool calls. This is tracked in: - vercel/ai#7099 (supposedly fixed) - vercel/ai#8031 (duplicate) The error still occurs occasionally in production. Future work may include implementing retry logic or additional workarounds if the SDK fix is incomplete. Generated with `cmux`
ammario
force-pushed
the
openai-reasoning-test
branch
from
6b32d5e to
34140a0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds integration tests to reproduce the intermittent OpenAI reasoning error and updates dependencies to attempt to fix it:
Changes
1. Integration Test (Test Coverage)
File:
tests/ipcMain/openaiReasoning.test.tsTwo test approaches:
cmux-docs-styleworkspace that had 3800+ reasoning parts and 17 tool callsThe deterministic test successfully reproduces the error with 100% consistency on Turn 1.
Run with:
2. Dependency Updates & Configuration Fix
include: ['reasoning.encrypted_content']from provider optionsRationale: Per OpenAI docs, encrypted reasoning content is only needed when
store: false(ZDR compliance). We usestore: true(default) withpreviousResponseId, so OpenAI manages reasoning state automatically.Investigation Results
Key Findings
Error is a Vercel AI SDK bug, tracked in:
Error occurs during streaming on FIRST turn (no history), when:
[stream-start, reasoning-delta..., reasoning-end, tool-call-start, stream-error]Configuration doesn't fix it: Tested both WITH and WITHOUT
include: ['reasoning.encrypted_content']- error occurs in both casesSDK updates don't fix it: Error persists even with latest AI SDK 5.0.60
Debug Dumps
Test creates detailed dumps in
~/.cmux/debug_obj/:1_original_messages.json- Messages from history1b_openai_stripped.json- After stripping reasoning2_model_messages.json- After convertToModelMessages3_final_messages.json- Final messages sent to APIKnown Limitation
Testing Status
Next Steps
If monitoring shows the error persists in production:
Generated with
cmux