Bump tokenizers crate from 0.14.0 to 0.22.2#7
Merged
Anush008 merged 2 commits intoAnush008:mainfrom Mar 19, 2026
Merged
Conversation
Updates the Rust tokenizers dependency to pick up two years of upstream improvements. Adds ahash dependency required by newer crate versions. Source changes adapt to API changes in the with_* methods (now take Option<T>), BPE vocab type (AHashMap), and metaspace prepend_scheme parameter. Verified building on aarch64-unknown-linux-gnu with Rust 1.94.0.
Signed-off-by: Anush008 <mail@anush.sh>
Owner
|
Now available with |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updates the Rust
tokenizerscrate dependency from 0.14.0 (September 2023) to 0.22.2 (December 2025), picking up over two years of upstream improvements from huggingface/tokenizers.Changes
Cargo.toml:
tokenizers0.14.0 -> 0.22.2ahash = { version = "0.8", features = ["serde"] }(required by newer tokenizers crate for vocab hash maps)Source changes (required by API changes in tokenizers 0.15+):
with_pre_tokenizer,with_decoder,with_post_processor,with_normalizernow takeOption<T>instead ofTVocabtype changed from re-export toHashMap<String, u32>, requiresAHashMapconversionadd_prefix_space: boolreplaced byprepend_scheme: String+split: boolparametersformat!("{}", e)replaced withformat!("{e}")(Rust 2021 edition style)Verification
aarch64-unknown-linux-gnuwith Rust 1.94.0Context
The source changes follow the approach proven by @inference-net/tokenizers (which bumped to 0.21.4) combined with the platform targets from v0.5.0. This PR goes further to 0.22.2.