Skip to content

v1.0.0-rc.5 Release candidate

Pre-release
Pre-release

Choose a tag to compare

@SBrandeis SBrandeis released this 20 Nov 14:05
· 1 commit to main since this release
v1.0.0-rc.5
1915b16

v1.0.0-rc.5

Another week, another Release Candidate 💥

This release candidate is mostly about correctness: a dozen encode and decode differences with 0.23 are fixed, along with crashes on malformed or unusual inputs.

It also drops the numpy dependency from the Python bindings, and freezes the shape of the Rust public API ahead of 1.0.

Install it with:

pip install --pre tokenizers==1.0.0rc5

Breaking changes from v1.0.0-rc.4

If you built something on a previous release candidate, here are the breaking changes from v1.0.0-rc.4:

  • Python: numpy is no longer a dependency. Encoding.ids_array, type_ids_array and attention_mask_array are replaced by ids_dlpack, type_ids_dlpack and attention_mask_dlpack, which any DLPack consumer reads as an int64 copy (#2709)

    encoding = compiled.encode("Hello, World!")
    torch.from_dlpack(encoding.ids_dlpack())   # int64 tensor
    numpy.from_dlpack(encoding.ids_dlpack())   # int64 array

    decode and decode_tokens no longer take numpy arrays as input: pass an Encoding or a list of ids.

  • Python: Python 3.10 is no longer supported, it reached end of life. The minimum is now 3.11 (#2668)

  • Node and wasm: encode returns an Encoding instead of a Uint32Array, and takes a single text or a pair, like Python. New tokenize, encodeBatch and decodeTokens (#2603)

  • Rust: the public API is trimmed down for 1.0.

    • The Pipeline prefix is gone: PipelineTokenizer is now Tokenizer, PipelineEncoding is Encoding, PipelineBPE is BPE, and so on (#2703)
    • Fewer pub items, legacy pre-pipeline code removed, open enums and structs are #[non_exhaustive], structs with an invariant have private fields (#2696, #2698, #2699, #2701)
    • tokenizers no longer re-exports the trainers: the train, esaxx_fast, parity-aware-bpe and progressbar features are removed. Training API will be redesigned and come back after 1.0 (#2706)
    • Tokenize::tokenize_spans takes a PreTokens, built by PreTokenizer::pre_tokens; PatternMatcher::extract_next takes a &str (#2711, #2712)

Performance

  • Python: installing tokenizers no longer pulls numpy, and loading a tokenizer uses 22% less peak memory. Getting an int64 numpy array from an Encoding is 28% faster (#2709)
  • RISC-V: added-token and literal matching use the vector extension (RVV) when the CPU supports it (#2650)

Fixes

Encoding now matches 0.23 on:

  • DeepSeek, and other models split with bitcannon: characters were classified with Unicode 17 tables, not the ones 0.23 uses, so some recent characters (e.g. a space followed by U+1FAEA on DeepSeek-V3) were pre-tokenized differently (#2558)
  • Kimi: Han characters that are also digits or punctuation (1〇, !⺀) (#2567)
  • CLIP and other byte-level BPEs with an end_of_word_suffix or continuing_subword_prefix: non-ASCII words returned different ids (#2592)
  • BPEs with an end_of_word_suffix: single-character words did not get the suffix (#2484)
  • BPEs with affixes and fuse_unk: consecutive unknown characters gave one unk id each instead of one in total (#2676)
  • Added tokens with single_word: the check looked at the wrong character, and panicked on some multi-byte text (e.g. bert-base-multilingual-cased) (#2666)
  • Added tokens on tokenizers with many of them: the fast added-token scan could miss some tokens, and was wrong on aarch64. It now falls back to the scalar path in both cases (#2726)

Decoding:

  • Fixes a bug causing decode to return byte-level garbage ('a�b' instead of 'a�b') for some tokens on byte-level BPEs whose added tokens overlap the vocab, e.g. Olmo-3 (#2734)

Loading:

  • Fixes a bug causing byte-fallback BPEs that lack some <0xNN> tokens (e.g. Gemma 2) to fail to load (#2461)
  • Fixes a bug causing a Sequence holding a single Metaspace to fail to load (#2591)
  • Fixes a bug causing loading to panic, about 1 time in 6, on tokenizers with many consecutive added-token ids (e.g. openai/whisper-large-v3) (#2667)
  • More robust vocabulary builds for small vocabularies (#2695)

Crashes and safety:

  • A Split or Replace regex whose matches overlap (e.g. \K inside a look-behind) now returns an error instead of crashing the process (#2671)
  • A BPE whose merges do not carry the continuing_subword_prefix is now rejected at load instead of producing invalid strings (#2398)
  • Unigram no longer overflows the stack on a very long word (#2658)
  • A process forked while another thread was building the thread pool no longer runs every encode serially in the child (#2656)
  • Removed unsafe code reachable from the safe Rust API (#2711, #2712)

Internal

Contributors

Many thanks to all the contributors who made this release candidate possible!

Thank you all!

Full Changelog: v1.0.0-rc.4...v1.0.0-rc.5

This changelog has been written with AI assistance