server: move msg diffs tracking to HTTP thread #17740
Merged
+165
−92
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ngxson
commented
Dec 3, 2025
|
|
||
| llama_token sampled; | ||
|
|
||
| common_chat_format chat_format = COMMON_CHAT_FORMAT_CONTENT_ONLY; |
Collaborator
Author
There was a problem hiding this comment.
chat_format seems to be unused, so I removed this. no idea why it was added in the first place
ngxson
changed the title
Collaborator
Author
|
Server tests should be OK now, running from the mirror PR: ngxson#49 |
ngxson
commented
Dec 4, 2025
tools/server/server-context.cpp
Outdated
| states.emplace_back(task.params.oaicompat_chat_syntax); | ||
|
|
||
| tasks.push_back(std::move(task)); | ||
| states.push_back(task.params.oaicompat_chat_syntax); |
Collaborator
Author
There was a problem hiding this comment.
task is already moved at this point
ggerganov
reviewed
Dec 4, 2025
tools/server/server-context.cpp
Outdated
| // send the results | ||
| json res_json = result->to_json(); | ||
| if (result->is_error()) { | ||
| res->next = [res_this = res.get(), res_type, &should_stop, states = std::move(states)](std::string & output) mutable -> bool { |
Member
There was a problem hiding this comment.
Instead of making the lambda mutable, wouldn't it be cleaner to maintain the states in the server_res_generator instance (i.e. res)?
Collaborator
Author
There was a problem hiding this comment.
Yes that's good idea. I moved it to server_response_reader instead. update() is now called automatically whenever the result is received.
Also added a safe-guard ba2af58 to avoid someone forget to call update()
ngxson
force-pushed
the
xsn/server_improve_msg_diff
branch
from
aad20a6 to
1c30c28
Compare
gabe-l-hart
added a commit
to gabe-l-hart/llama.cpp
that referenced
this pull request
Dec 4, 2025
* origin/master: server: strip content-length header on proxy (ggml-org#17734) server: move msg diffs tracking to HTTP thread (ggml-org#17740) examples : add missing code block end marker [no ci] (ggml-org#17756) common : skip model validation when --help is requested (ggml-org#17755) ggml-cpu : remove asserts always evaluating to false (ggml-org#17728) convert: use existing local chat_template if mistral-format model has one. (ggml-org#17749) cmake : simplify build info detection using standard variables (ggml-org#17423) ci : disable ggml-ci-x64-amd-* (ggml-org#17753) common: use native MultiByteToWideChar (ggml-org#17738) metal : use params per pipeline instance (ggml-org#17739) llama : fix sanity checks during quantization (ggml-org#17721) build : move _WIN32_WINNT definition to headers (ggml-org#17736) build: enable parallel builds in msbuild using MTT (ggml-org#17708) ggml-cpu: remove duplicate conditional check 'iid' (ggml-org#17650) Add a couple of file types to the text section (ggml-org#17670) convert : support latest mistral-common (fix conversion with --mistral-format) (ggml-org#17712) Use OpenAI-compatible `/v1/models` endpoint by default (ggml-org#17689) webui: Fix zero pasteLongTextToFileLen to disable conversion being overridden (ggml-org#17445)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix #17726
The proposed approach: Delegate state tracking to HTTP thread
task_result_statethat holds the state of the current generation "session". This object will be owned by the HTTP threadresult.update(state); this allow the result to populate partial returns, at the same time also update thestateobject