1.4.0

Notes and Migration Guide

Tokenizer decoding compatibility change

The decoding cleanup policy now matches Transformers 5.7.0:

• clean_up_tokenization_spaces defaults to false when absent from the tokenizer config.
• BPE tokenizers skip cleanup even when that setting is true, preserving spaces around punctuation and contractions. For example, x != y now stays x != y instead of becoming x!= y.

This can change decoded text, including generated output, for existing models. Encoding and token IDs are unchanged. Affected models include Llama 3.x configurations with cleanup enabled, and tokenizers that omit the setting. Configurations with cleanup explicitly disabled retain their behavior.

To retain cleanup, set clean_up_tokenization_spaces to true in tokenizer_config.json. For BPE tokenizers, also set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output to true.

See swift-transformers #391 and the related upstream Transformers change.

Regex compilation errors and throwing initialization

Tokenizers now throw TokenizerError.invalidRegex during loading when a Split pre-tokenizer or Replace normalizer/decoder regex cannot be compiled, including inside a Sequence. Previously, invalid Split patterns silently left the input unsplit, while invalid Replace patterns terminated the process. The new TokenizerError.invalidRegex includes the pattern and underlying Foundation error. Patterns accepted by upstream may still fail if their syntax is unsupported by Foundation’s regex engine.

Source compatibility: PreTokenizer.init(config:), Normalizer.init(config:), and Decoder.init(config:) now declare throws. Calls through these protocol metatypes or generic constraints (for example, T: Normalizer) must use try and handle or propagate initialization errors:

// Before
let tokenizer = T(config: config)

// After (inside a throwing function)
let tokenizer = try T(config: config)

Existing nonthrowing implementations still conform to these protocols. Ordinary AutoTokenizer.from(...) calls already require try and need no source changes. Valid Split and Replace regex patterns retain their behavior.

See swift-transformers #407 and #409.

What's Changed

• Register Qwen3_5Tokenizer by @mattt in #403
• Match Rust tokenizers on code points in Replace, Metaspace and BPE merges by @zm2231 in #395
• Honor ignore_merges in BPE by @MBark286 in #397
• 🔴🔴🔴 Skip clean_up_tokenization_spaces when decoding with BPE models by @negativetime in #391
• 🔴🔴🔴 Clean up tokenization spaces is now set to false by default by @pcuenca in #405
• Match Split regex patterns on code points by @MBark286 in #398
• Respect addSpecialTokens in RobertaProcessing by @wanjinhao1 in #400
• More expressive splits by @pcuenca in #406
• 🔴 🔴 🔴 Throw on invalid regex rather than silently ignoring by @pcuenca in #407
• Register GPTNeoXTokenizer as a BPE tokenizer by @JamesPriceZV in #382
• Support yyjson 0.13.0 on Swift 6.1 and newer by @SeoFood in #392
• Tokenizer benchmarks by @jkrukowski in #393
• 🔴 Throw on invalid Replace regex rather than crashing by @apocryphx in #409
• Fuse unknown tokens per word and read fuse_unk from the model by @apocryphx in #411
• Strip combining marks in StripAccents normalizer by @RaphaelFakhri in #413
• Use precompiled_charsmap in the Precompiled normalizer by @apocryphx in #410
• Allow swift-jinja 3.0 by @mattt in #415
• fix: flush byte fallback runs at end of input by @TheTom in #414
• BPETokenizer: build vocab and merge strings natively to speed up load by @jeethu in #416
• [CI] Test offline tokenizers on Linux by @pcuenca in #417
• Fix WordPieceDecoder crash on empty input and ByteFallbackDecoder trailing byte drop by @john-rocky in #358
• Use BinaryDistinctString for consistent hashing on Linux and macOS by @pcuenca in #418
• fix(Tokenizers): avoid quadratic regex matching for added tokens without whitespace padding by @shubhransh-gupta in #386
• Fuse unknown tokens in linear time by @jeethu in #420
• BPETokenizer: faster UTF-16 decoding and merge-rank construction at load by @jeethu in #419
• Update README by @pcuenca in #421

New Contributors

• @zm2231 made their first contribution in #395
• @MBark286 made their first contribution in #397
• @negativetime made their first contribution in #391
• @wanjinhao1 made their first contribution in #400
• @JamesPriceZV made their first contribution in #382
• @SeoFood made their first contribution in #392
• @RaphaelFakhri made their first contribution in #413
• @TheTom made their first contribution in #414
• @jeethu made their first contribution in #416
• @shubhransh-gupta made their first contribution in #386

Full Changelog: 1.3.5...1.4.0