Skip to content

Release candidate v1.0.0-rc.3

Pre-release
Pre-release

Choose a tag to compare

View all tags
@SBrandeis SBrandeis released this 19 Oct 17:08
· 0 commits to main since this release
v1.0.0-rc.3
This tag was signed with the committer鈥檚 verified signature.
SBrandeis Simon Brandeis
SSH Key Fingerprint: ruT0quYs2nek2A3cjKQwliurwzxI62SrKeaEMbnPqLY
Verified
Learn about vigilant mode.
f8f4181
This commit was signed with the committer鈥檚 verified signature.
SBrandeis Simon Brandeis
SSH Key Fingerprint: ruT0quYs2nek2A3cjKQwliurwzxI62SrKeaEMbnPqLY
Verified
Learn about vigilant mode.

v1.0.0-rc.3

1.0.0-rc.3 is mainly a bindings release: it brings tokenizers to the browser with WebAssembly bindings!

Install them with:

npm install tokenizers-wasm@next

We're making steady progress towards feature parity with 0.23, the main milestone left before a stable 1.0.0.

New

Fixes

  • Fixes a bug causing byte-level BPE tokenizers whose vocabulary lacks bytes that can never appear in UTF-8 (e.g. ModernBERT) to fail to load. The error now names the missing byte (#2466)
  • Fixes a bug causing WordPiece tokenizers with an empty-string token in their vocabulary to fail to load (#2538)
  • Fixes a bug causing the Python, Node and WASM bindings to reject any tokenizer.json with a regex Split pre-tokenizer or Replace normalizer, which covers NLLB and every model transformers converted from SentencePiece (#2536)
  • Fixes a bug causing data races when Python's Encoding.*_array getters were used concurrently; they now return an owned copy instead of a view (#2464)
  • Fixes a bug causing a 64-bit hash collision between two tokens to silently drop one of them when loading a vocabulary; it is now an error. Duplicate tokens log a warning instead of printing the whole vocab to stdout (#2467)
  • Fixes a bug causing ParityBpeTrainer to produce a different output than the regular BPE trainer; it now trains into Vocab + Merges like the others (#2474)

Python

  • huggingface_hub 2.0 is allowed (#2486)
  • One unnecessary Encoding copy at the Rust/Python boundary is gone (#2452)

Internal

  • Umbrella crate features: depend on tokenizers alone and pick components by feature, e.g. cargo add tokenizers -F train or cargo add tokenizers -F bpe --no-default-features. The tk-* crates still work on their own (#2476, #2477, #2479)
    • The bindings now depend on the umbrella crate (#2478, #2552)
    • The crate README and docs.rs front page now explain how to install the crate and pick its features (#2481)
  • Dependency updates, CI hardening and other maintenance (#2446, #2448, #2449, #2465, #2471, #2472, #2473, #2475, #2480, #2492, #2495, #2510, #2537, #2548)

Thank you

Full Changelog: v1.0.0-rc.2...v1.0.0-rc.3

This changelog has been written with AI assistance

Contributors

musicinmybrain, SBrandeis, and paulinebm
Assets 2
Loading