Repository navigation
splintr 0.21.0
Added
QWEN35_PATTERN, the split for llama.cpp'sqwen35pre-tokenizer (Qwen 3.5, Bonsai 2), exported from Rust and Python. A GGUF vocabulary declaringqwen35loads with it instead of being refused. It differs fromQWEN2_PATTERNonly in keeping a combining mark in the same piece as the letter before it. It splits on a direct scanner, asQWEN2_PATTERNdoes, not on the regex engine.
Changed
- Requires
regexr0.6. Patterns that run on the regex engine instead of a scanner pick up its fixes: leftmost-first priority on the DFA engines, assertions inside a match, and linear-time search where some searches were quadratic. - Requires
base640.23.VocabError::Base64ErrorandVocabError::SpmBase64carrybase640.23'sDecodeError, so code that matches on that type needs the samebase64version. - Dependencies are updated to their latest releases.
Full diff: v0.20.0...v0.21.0