Release v2.0.3: transformers v5, Python 3.15 and alignment fix · explosion/spacy-transformers

✨ New features and improvements

transformers v5 support. transformers>=4.45.0,<6.0.0 is now supported. CI tests both v5 and the v4 series.
Python 3.15 wheels. Wheels are built for Python 3.11–3.15 on Linux (x86_64, aarch64, glibc and musl), macOS (Intel and Apple silicon) and Windows (x64 and ARM64).
Safer serialization. Transformer weights are now saved in the safetensors format instead of a pickled torch state dict. Weights saved in the old format still load, but only through torch.load(weights_only=True), so pickled checkpoints can no longer run arbitrary code on load.
Warn when use_fast=False is requested but only a fast tokenizer is available. transformers v5 no longer provides slow tokenizers, and a fast tokenizer can split some texts into different wordpieces.
Building from source no longer requires numpy.

Fix the wordpiece alignment for fast tokenizers whose offsets include the preceding space, such as DeBERTa-v2/v3 (and, with older transformers versions, XLM-R-style tokenizers like Multilingual-MiniLM). Each wordpiece was also aligned to the previous spaCy token, so pooled token vectors were mixed with the next word's representation. Whitespace-only wordpieces (e.g. GPT-2's Ġ) are still aligned to the preceding token. See explosion/spaCy#13730 and #429.
Keep attention outputs when reloading a saved pipeline with output_attentions enabled under transformers v5.

Python 3.11+ is required. Python 3.10 is no longer supported.
transformers 4.45.0+ is required.
Pipelines saved with v2 can't be loaded with v1.x. Pipelines saved with v1.x still load with v2, but once saved with v2 their weights are in safetensors format, which v1.x can't read.
If you trained a pipeline with an affected tokenizer (e.g. DeBERTa-v3) using an earlier version, the alignment fix slightly changes the token features it sees. Retrain or re-evaluate those pipelines after upgrading. Pipelines based on RoBERTa or BERT, such as en_core_web_trf v3.6, are not affected.