unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regr…#22110
Conversation
…ession tests
- Add unicode_regex_split_custom_qwen35() to [src/unicode.cpp](src/unicode.cpp), a non-backtracking handler for Qwen3.5's [\p{L}\p{M}]+ regex (letters + combining marks).
- Register the handler in the custom tokenizer dispatch table to prevent stack overflows on long inputs (fixes ggml-org#21919).
- Add [models/ggml-vocab-qwen35.gguf](models/ggml-vocab-qwen35.gguf) (test vocab), [models/ggml-vocab-qwen35.gguf.inp](models/ggml-vocab-qwen35.gguf.inp) (test cases), and [models/ggml-vocab-qwen35.gguf.out](models/ggml-vocab-qwen35.gguf.out) (expected output) for regression testing.
- Update [tests/CMakeLists.txt](tests/CMakeLists.txt) to include the new test entry.
This mirrors the Qwen2 fix (commit 0d049d6), but adapts for Qwen3.5's regex. Ensures robust Unicode tokenization and prevents std::regex stack overflows.
Closes ggml-org#21919.
|
CI test failure - Issues Identified
The test called common_params_parse() 38 times (once for each LLAMA_EXAMPLE_*).
The test was running 3 models × 4 parallel contexts × 128 tokens on a debug Vulkan build.
Two separate issues: Fixes Applied
Added std::once_flag to ensure backends are loaded exactly once.
Connection, read, and write timeouts are now only applied when params.timeout > 0.
Reduced the number of tokens and parallelism for CI runs.
Replaced the std::regex call in normalize_newlines() with a simple character loop (identical semantics, no regex engine involved). |
github-actions
Bot
added
the
ggml
There was a problem hiding this comment.
Please revert the CI related changes, they're unrelated to this PR. Also, please take a look at the AI usage policy in CONTRIBUTING.md: https://github.com/ggml-org/llama.cpp/blob/master/CONTRIBUTING.md
Kabir08
force-pushed
the
fix/qwen35-tokenizer-stack-overflow
branch
from
a56a577 to
a103403
Compare
ggerganov
added
the
merge ready
ggml-org#22110) * unicode,test: add Qwen3.5 non-backtracking tokenizer handler and regression tests - Add unicode_regex_split_custom_qwen35() to [src/unicode.cpp](src/unicode.cpp), a non-backtracking handler for Qwen3.5's [\p{L}\p{M}]+ regex (letters + combining marks). - Register the handler in the custom tokenizer dispatch table to prevent stack overflows on long inputs (fixes ggml-org#21919). - Add [models/ggml-vocab-qwen35.gguf](models/ggml-vocab-qwen35.gguf) (test vocab), [models/ggml-vocab-qwen35.gguf.inp](models/ggml-vocab-qwen35.gguf.inp) (test cases), and [models/ggml-vocab-qwen35.gguf.out](models/ggml-vocab-qwen35.gguf.out) (expected output) for regression testing. - Update [tests/CMakeLists.txt](tests/CMakeLists.txt) to include the new test entry. This mirrors the Qwen2 fix (commit 0d049d6), but adapts for Qwen3.5's regex. Ensures robust Unicode tokenization and prevents std::regex stack overflows. Closes ggml-org#21919. * fix: enhance regex handling for Qwen3.5 tokenizer to include accent marks * cont : remove trailing whitespace --------- Co-authored-by: Kabir <kabir@example.com> Co-authored-by: Alde Rojas <hello@alde.dev>
4 participants
This mirrors the Qwen2 fix (commit 0d049d6), but adapts for Qwen3.5's regex. Ensures robust Unicode tokenization and prevents std::regex stack overflows.
Closes #21919.
Overview
Additional information
Requirements