server/common: Fix response_format: json_schema & prefill parsing bug#23993
Draft
roj234 wants to merge 1 commit into
Draft
server/common: Fix response_format: json_schema & prefill parsing bug#23993roj234 wants to merge 1 commit into
response_format: json_schema & prefill parsing bug#23993roj234 wants to merge 1 commit into
Conversation
- 听闻我在完全不了解生命周期的情况下直接使用裸指针,然后看程序是否崩溃来调试,项目的维护者发出尖锐爆鸣
Contributor
|
Tell your little AI buddy to simply prepend the backticks when constructing the generation prompt on continuation and report back the LOC difference. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
The problem happens when you prefill json schema on Gemma4 (maybe other model),
bnf =
... root ::= start (thought | )? "```json" space response-format-schema space "```" ...generation_prompt =
<|turn>model <|channel>thought <channel|>{"name":It could not parse because 3-backtick + json is missing
Fixes #23990
Automatically insert const string in BNF grammar
thoughttoresponse-format-schemaAlso changed struct
server_taskandtask_result_stateto sharechat_parser_paramsgeneration_prompt_fixed =
<|turn>model <|channel>thought <channel|>```json {"name":Additional information
This is a draft because
chat_parser_paramsmight cause concurrent bugs, andcommon/json_schema_prefill_support.hppmostly written by AI, this BNF string parser is very naive.But IT RUNS now, if you (viewer, not contributor or owner) need this, cherry-pick it!
Requirements