Skip to content

splintr 0.22.0

Latest

Choose a tag to compare

@github-actions github-actions released this 12 Oct 00:57
· 2 commits to main since this release

Removed

  • HfJsonError::InvalidByteLevel. No load returns it any more. A match that names it no longer compiles.

Fixed

  • A byte-level tokenizer.json whose vocabulary holds literal entries outside added_tokens now loads. Cohere's holds 1403, such as ™ and emoji pieces. Encode never emits such an entry, and decode renders its literal text, as tokenizers does. The load failed with InvalidByteLevel before, and that error variant is removed.

Full diff: v0.21.1...v0.22.0