Skip to content

server/common: Fix response_format: json_schema & prefill parsing bug#23993

Draft
roj234 wants to merge 1 commit into
ggml-org:masterfrom
roj234:json_schema_prefill
Draft

server/common: Fix response_format: json_schema & prefill parsing bug#23993
roj234 wants to merge 1 commit into
ggml-org:masterfrom
roj234:json_schema_prefill

Conversation

@roj234
Copy link
Copy Markdown
Contributor

@roj234 roj234 commented Jun 1, 2026

Overview

The problem happens when you prefill json schema on Gemma4 (maybe other model),

bnf = ... root ::= start (thought | )? "```json" space response-format-schema space "```" ...
generation_prompt = <|turn>model <|channel>thought <channel|>{"name":

It could not parse because 3-backtick + json is missing

Fixes #23990

Automatically insert const string in BNF grammar thought to response-format-schema
Also changed struct server_task and task_result_state to share chat_parser_params

generation_prompt_fixed = <|turn>model <|channel>thought <channel|>```json {"name":

Additional information

This is a draft because

  • share chat_parser_params might cause concurrent bugs, and
  • common/json_schema_prefill_support.hpp mostly written by AI, this BNF string parser is very naive.

But IT RUNS now, if you (viewer, not contributor or owner) need this, cherry-pick it!

Requirements

- 听闻我在完全不了解生命周期的情况下直接使用裸指针,然后看程序是否崩溃来调试,项目的维护者发出尖锐爆鸣
@aldehir
Copy link
Copy Markdown
Contributor

aldehir commented Jun 1, 2026

Tell your little AI buddy to simply prepend the backticks when constructing the generation prompt on continuation and report back the LOC difference.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Eval bug: Failed to initialize samplers on Gemma4 with response_format: json_schema prefill

2 participants