Repository navigation
v1.0.0-rc.5 Release candidate
Pre-releasev1.0.0-rc.5
Another week, another Release Candidate 💥
This release candidate is mostly about correctness: a dozen encode and decode differences with 0.23 are fixed, along with crashes on malformed or unusual inputs.
It also drops the numpy dependency from the Python bindings, and freezes the shape of the Rust public API ahead of 1.0.
Install it with:
pip install --pre tokenizers==1.0.0rc5Breaking changes from v1.0.0-rc.4
If you built something on a previous release candidate, here are the breaking changes from v1.0.0-rc.4:
-
Python:
numpyis no longer a dependency.Encoding.ids_array,type_ids_arrayandattention_mask_arrayare replaced byids_dlpack,type_ids_dlpackandattention_mask_dlpack, which any DLPack consumer reads as anint64copy (#2709)encoding = compiled.encode("Hello, World!") torch.from_dlpack(encoding.ids_dlpack()) # int64 tensor numpy.from_dlpack(encoding.ids_dlpack()) # int64 array
decodeanddecode_tokensno longer take numpy arrays as input: pass anEncodingor a list of ids. -
Python: Python 3.10 is no longer supported, it reached end of life. The minimum is now 3.11 (#2668)
-
Node and wasm:
encodereturns anEncodinginstead of aUint32Array, and takes a single text or a pair, like Python. Newtokenize,encodeBatchanddecodeTokens(#2603) -
Rust: the public API is trimmed down for 1.0.
- The
Pipelineprefix is gone:PipelineTokenizeris nowTokenizer,PipelineEncodingisEncoding,PipelineBPEisBPE, and so on (#2703) - Fewer
pubitems, legacy pre-pipeline code removed, open enums and structs are#[non_exhaustive], structs with an invariant have private fields (#2696, #2698, #2699, #2701) tokenizersno longer re-exports the trainers: thetrain,esaxx_fast,parity-aware-bpeandprogressbarfeatures are removed. Training API will be redesigned and come back after 1.0 (#2706)Tokenize::tokenize_spanstakes aPreTokens, built byPreTokenizer::pre_tokens;PatternMatcher::extract_nexttakes a&str(#2711, #2712)
- The
Performance
- Python: installing
tokenizersno longer pullsnumpy, and loading a tokenizer uses 22% less peak memory. Getting anint64numpy array from anEncodingis 28% faster (#2709) - RISC-V: added-token and literal matching use the vector extension (RVV) when the CPU supports it (#2650)
Fixes
Encoding now matches 0.23 on:
- DeepSeek, and other models split with bitcannon: characters were classified with Unicode 17 tables, not the ones 0.23 uses, so some recent characters (e.g. a space followed by U+1FAEA on DeepSeek-V3) were pre-tokenized differently (#2558)
- Kimi: Han characters that are also digits or punctuation (
1〇,!⺀) (#2567) - CLIP and other byte-level BPEs with an
end_of_word_suffixorcontinuing_subword_prefix: non-ASCII words returned different ids (#2592) - BPEs with an
end_of_word_suffix: single-character words did not get the suffix (#2484) - BPEs with affixes and
fuse_unk: consecutive unknown characters gave one unk id each instead of one in total (#2676) - Added tokens with
single_word: the check looked at the wrong character, and panicked on some multi-byte text (e.g.bert-base-multilingual-cased) (#2666) - Added tokens on tokenizers with many of them: the fast added-token scan could miss some tokens, and was wrong on aarch64. It now falls back to the scalar path in both cases (#2726)
Decoding:
- Fixes a bug causing
decodeto return byte-level garbage ('a�b'instead of'a�b') for some tokens on byte-level BPEs whose added tokens overlap the vocab, e.g.Olmo-3(#2734)
Loading:
- Fixes a bug causing byte-fallback BPEs that lack some
<0xNN>tokens (e.g. Gemma 2) to fail to load (#2461) - Fixes a bug causing a
Sequenceholding a singleMetaspaceto fail to load (#2591) - Fixes a bug causing loading to panic, about 1 time in 6, on tokenizers with many consecutive added-token ids (e.g.
openai/whisper-large-v3) (#2667) - More robust vocabulary builds for small vocabularies (#2695)
Crashes and safety:
- A
SplitorReplaceregex whose matches overlap (e.g.\Kinside a look-behind) now returns an error instead of crashing the process (#2671) - A BPE whose merges do not carry the
continuing_subword_prefixis now rejected at load instead of producing invalid strings (#2398) - Unigram no longer overflows the stack on a very long word (#2658)
- A process forked while another thread was building the thread pool no longer runs every encode serially in the child (#2656)
- Removed unsafe code reachable from the safe Rust API (#2711, #2712)
Internal
- Dependency updates, lint fixes, CI and Node bindings CI maintenance (#2678, #2692, #2694, #2704, #2705, #2739)
Contributors
Many thanks to all the contributors who made this release candidate possible!
- @SBrandeis
- @McPatate
- @patrasq made their first contribution 🎉
- @MohammadHijjawi97 made their first contribution 🎉
- @apollo-2006 made their first contribution 🎉
- @daaain made their first contribution 🎉
- @xiazhuozhao made their first contribution 🎉
- @Iamsohungrynow made their first contribution 🎉
- @Aurantium-47 made their first contribution 🎉
- @original4422 made their first contribution 🎉
- @0z5a made their first contribution 🎉
- @founderinaction made their first contribution 🎉
Thank you all!
Full Changelog: v1.0.0-rc.4...v1.0.0-rc.5
This changelog has been written with AI assistance