v1.0.0-rc.4 Release candidate

v1.0.0-rc.4

Release Candidate number 4 is out!
It brings back the required Python APIs to use tokenizers in other python libraries, for example transformers, vLLM, ...

Install it with:

pip install --pre tokenizers==1.0.0rc4

We're making steady progress towards feature parity with 0.23 on encoding and decoding, the main milestone left before a stable 1.0.0!

Breaking changes from v1.0.0-rc.3

If you built something on a previous release candidate, here are the breaking changes from v1.0.0-rc.3:

• Python: The Tokenizer class no longer encodes, you need to call tokenizer.compile() first. A Tokenizer now holds the tokenizer.json config and can be edited. compile() builds a frozen CompiledTokenizer from it. CompiledTokenizer can encode and decode. (#2562)

  tokenizer = Tokenizer.from_pretrained("openai-community/gpt2")
  tokenizer.truncation = Truncation(max_length=512)
  compiled = tokenizer.compile()
  compiled.encode("Hello, World!").ids

Benefits:

• Encoding takes no lock and releases the GIL
• Copying a Tokenizer is cheap: copies share the config until one of them is edited
• Padding and truncation live in the config and become the encode defaults; each encode call can still override them

New

• Python: composition API. Build a tokenizer from components, or swap one out on an existing tokenizer:
  • tokenizer.normalizer, .pre_tokenizer, .model, .post_processor and .decoder can be read and set. Components are shared like in 0.23: editing one updates every tokenizer using it (#2579)
  • Models: BPE, Unigram, WordLevel, WordPiece, and Tokenizer(model) (#2569)
  • Post-processors: TemplateProcessing, RobertaProcessing, BertProcessing, ByteLevel and Sequence (#2573)
  • Sequence for normalizers (#2577)
  • Component classes are now generated from the Rust declarations, so adding the remaining ones is cheap (#2629)
• Python: serialization. Tokenizer.from_str, to_str and save (#2563)
• Python: vocab APIs. get_vocab, get_vocab_size, token_to_id and id_to_token (#2614)
• Python: added tokens. AddedToken, Tokenizer.add_tokens, Tokenizer.add_special_tokens and CompiledTokenizer.added_tokens. Adding tokens one at a time is ~1000x faster than 0.23.1 (5k calls: 4 ms vs 3.85 s) (#2615)
• Python: DecodeStream, compatible with vLLM's usage (#2551)

Performance

• Added tokens use ~1000x less memory: gpt-oss goes from 1.71 MiB to 1.6 KiB. Decode is up to 20% faster (#2612)
• Many threads encoding at once no longer build a throwaway thread pool when the shared one is busy: up to 3x the throughput with 32 concurrent callers (#2633)
• Loading a BPE model builds its vocabulary once instead of three times (#2613)

Fixes

• Fixes a bug causing forked child processes to hang when they loaded or unpickled a tokenizer after the parent had loaded one. This is the pattern behind datasets.map(num_proc=...) and PyTorch DataLoader fork workers (#2581, #2632)
• Fixes a bug causing byte-level BPE tokenizers whose vocabulary lacks some bytes to fail to load. They load again with a warning, and encoding drops the missing bytes, like 0.23 (#2550)
• Fixes a bug causing WordPiece tokenizers with a "\0" token in their vocabulary (e.g. ai4bharat/IndicBERTv2-MLM-only) to fail to load (#2661)
• Fixes a bug causing Split pre-tokenizers read from tokenizer.json to ignore the fancy-regex backend when it is enabled (#2547)

Internal

• Dependency updates, CI and README maintenance (#2575, #2582, #2587, #2606, #2609, #2610, #2621, #2659)

Contributors

• @SBrandeis
• @McPatate
• @ArthurZucker
• @dlowzzxx made their first contribution 🎉 Thank you!

Full Changelog: v1.0.0-rc.3...v1.0.0-rc.4

This changelog has been written with AI assistance