python3Packages.tokenizers: 0.8.1 -> 0.9.2 #100700
Motivation for this change
Changes in the derivation:
Draft PR, because Huggingface transformers is not yet compatible with this version of tokenizers.
The text was updated successfully, but these errors were encountered:
Changelog: https://github.com/huggingface/tokenizers/releases/tag/python-v0.9.0 https://github.com/huggingface/tokenizers/releases/tag/python-v0.9.1 https://github.com/huggingface/tokenizers/releases/tag/python-v0.9.2 Changes in the derivation: * Tokenizers uses a newer version of pyo3 that does not require Rust nightly anymore. So, we do not have to use any cheat codes anymore. * Tokenizers is now a mixed Rust/Python project. The way it is set up does not work with Maturin, so switch to setuptools-rust instead. * Add additional data files needed for tests. * Use `pytestCheckHook`.