fix(serve): emit all stream_chunk deltas to fix concurrent tool-call streaming#4622
Merged
Conversation
Resolve api_server streaming conflict by combining multi-delta stream_chunk emission from fix-parser with OpenAI-aligned usage chunk handling from fix-parser-main. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates streaming response parsing so a single engine chunk can emit multiple ordered deltas, addressing concurrent tool-call streaming where reasoning/content/tool-call segments could previously remain buffered.
Changes:
- Changes
ResponseParser.stream_chunkto return a list of parsed deltas and updates OpenAI/Anthropic streaming consumers. - Updates GPT-OSS Harmony and default parser behavior for empty/no-op chunks under the new contract.
- Adjusts parser and endpoint tests to use the new list-return API and adds coverage for multi-delta ordering.
Reviewed changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
lmdeploy/serve/parsers/response_parser.py |
Changes the base parser streaming contract and emits all parsed deltas from one chunk. |
lmdeploy/serve/parsers/_openai_harmony.py |
Wraps GPT-OSS Harmony streaming output in the new list-return format. |
lmdeploy/serve/openai/api_server.py |
Iterates over parsed deltas and attaches finish/logprob/token metadata to the last delta per engine chunk. |
lmdeploy/serve/anthropic/streaming.py |
Iterates over parsed deltas for Anthropic SSE streaming. |
lmdeploy/version.py |
Bumps version to 0.14.0a0. |
tests/test_lmdeploy/serve/parsers/helpers.py |
Adds a helper for tests that still assert first-delta behavior. |
tests/test_lmdeploy/serve/parsers/test_qwen3_parser.py |
Updates Qwen3 parser tests and adds direct multi-delta assertions. |
tests/test_lmdeploy/serve/parsers/test_qwen3_5_parser.py |
Updates Qwen3.5 parser tests for the list-return contract. |
tests/test_lmdeploy/serve/parsers/test_llama3_parser.py |
Updates Llama3 parser tests for the list-return contract. |
tests/test_lmdeploy/serve/parsers/test_interns1_parser.py |
Updates InternS1 parser tests for the list-return contract. |
tests/test_lmdeploy/serve/parsers/test_gpt_oss_parser.py |
Updates GPT-OSS parser tests for the list-return contract. |
tests/test_lmdeploy/serve/parsers/test_glm47_parser.py |
Updates GLM4.7 parser tests for the list-return contract. |
tests/test_lmdeploy/serve/parsers/test_deepseek_v3_parser.py |
Updates DeepSeek V3 parser tests for the list-return contract. |
tests/test_lmdeploy/serve/anthropic/test_endpoints.py |
Updates fake Anthropic parsers to return lists of deltas. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The following is test script
test_concurrent_tools.py