Skip to content

splintr 0.21.0

Choose a tag to compare

@github-actions github-actions released this 28 Sep 02:48
· 8 commits to main since this release

Added

  • QWEN35_PATTERN, the split for llama.cpp's qwen35 pre-tokenizer (Qwen 3.5, Bonsai 2), exported from Rust and Python. A GGUF vocabulary declaring qwen35 loads with it instead of being refused. It differs from QWEN2_PATTERN only in keeping a combining mark in the same piece as the letter before it. It splits on a direct scanner, as QWEN2_PATTERN does, not on the regex engine.

Changed

  • Requires regexr 0.6. Patterns that run on the regex engine instead of a scanner pick up its fixes: leftmost-first priority on the DFA engines, assertions inside a match, and linear-time search where some searches were quadratic.
  • Requires base64 0.23. VocabError::Base64Error and VocabError::SpmBase64 carry base64 0.23's DecodeError, so code that matches on that type needs the same base64 version.
  • Dependencies are updated to their latest releases.

Full diff: v0.20.0...v0.21.0