Starlings.Tokenizers 0.3.5

Added

• Split pre-tokenizer: the o200k regex (tiktoken's o200k_base pattern, as microsoft/Phi-4-mini-instruct declares it), on a new hand-written scanner over Unicode scalar values with reference-probed \p{Lu}, \p{Ll}, \p{Lt}, \p{Lm} and \p{Lo} tables. It loads as Isolated with invert false or as Removed with invert true (the regex leaves no gaps, so both give the same pieces); other combinations fail at load. Previously it was rejected at load. Checked against huggingface/tokenizers 0.23.3 on 12,420 split cases (O200kSplitPatternTests).
• Profile phi4-bpe-chat (microsoft/Phi-4-mini-instruct at cfbefacb), experimental: byte-level BPE with the o200k regex and ten rstrip added tokens, 137/137 cases exact against huggingface/tokenizers 0.23.3, with and without special tokens, truncation to 16 tokens, and padded batches of single texts and pairs, plus 1,500 randomized strings with the Phi-4 role and tool tags exact (starlinghq/starling#109).
• WhitespaceSplit pre-tokenizer (upstream pre_tokenizers/whitespace.rs): splits on the Unicode White_Space characters and removes them. Tokenizers that use it now load instead of failing with UnsupportedTokenizerComponentException (for example pre_tokenizer.pretokenizers[0].type=WhitespaceSplit). Component reference cases from huggingface/tokenizers 0.23.3 (WhitespaceSplitComponentTests).
• Profile xlmr-unigram-paraphrase-multilingual (sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 at e8f8c211), experimental: XLM-R Unigram with Sequence[WhitespaceSplit, Metaspace], 49/49 cases exact against huggingface/tokenizers 0.23.3, with and without special tokens, the file's truncation to 128 tokens and its padding, in padded batches of single texts and pairs. Needed by starlinghq/starling#266.
• reference/tokenizer-json/run_tokenizer_json.py --clear-asset-truncation-padding: accepts an asset that declares truncation or padding, clears both before encoding, and records them in environment.json. ProfileConformanceTests clears them the same way (a no-op for every earlier profile).
• Profile modernbert-bpe-gte-reranker (Alibaba-NLP/gte-reranker-modernbert-base at f7481e60), experimental: the ModernBERT tokenizer (NFC, GPT-2 byte-level BPE with use_regex: true, [CLS] $A [SEP] / [CLS] $A [SEP] $B [SEP] templates, [MASK] with lstrip), 137/137 cases exact against huggingface/tokenizers 0.23.3, with and without special tokens, truncation to 16 tokens, and padded batches of single texts and pairs. No library change. The file declares truncation 8000 and Fixed 8000 padding, which Tokenizer.FromFile applies as Tokenizer.from_file does.

Documentation

• docs/supported-components.md: ByteLevel pre-tokenizer with use_regex true (the GPT-2 split regex, used by whisper-bpe and modernbert-bpe-gte-reranker) is listed as supported; the table still listed it as rejected.
• CHANGELOG.md and the library: lines of qwen3-bpe-embedding, qwen3-bpe-chat and lfm3-bpe-chat: 0.3.3 was never published, so its section says so and those profiles now name 0.3.4, the first published version that has them.

Behavior

• A tokenizer with an rstrip added token and, in the same matching pass (normalized or not), an added token that starts with whitespace now fails at load: upstream resumes matching inside the absorbed whitespace and gives overlapping splits, which this library did not reproduce. No pinned asset has that combination. No change in output for any existing profile.
• No change in output for any profile that shipped in 0.3.4 or earlier from WhitespaceSplit or the asset truncation/padding clearing: none of them uses WhitespaceSplit, and none declares truncation or padding in its file. Every existing fixture passes unchanged. Of the profiles new in this release, xlmr-unigram-paraphrase-multilingual and modernbert-bpe-gte-reranker do declare truncation and padding in their files (see above).

Starlings.Tokenizers 0.3.4

Added

• Profile clover-bpe-chat (Diffuspace/Clover-35B-A3B-Preview at 4b0402b6), experimental: byte-level BPE with the Qwen2 regex and the diffusion mask token <|mask|> at vocabulary id 151669, 137/137 cases exact against huggingface/tokenizers 0.23.3, with and without special tokens, truncation to 16 tokens, and padded batches of single texts and pairs. No library change.

Behavior

• No change in output for any existing profile; no library change.

Starlings.Tokenizers 0.3.2

The code of 0.3.2-preview.1 released without a preview suffix, so that a non-prerelease ai-dotnet package can pin it (NuGet NU5104). Only the version and documentation changed.

Behavior

• No change in output for any existing profile; the whisper-bpe profile and the GPT-2 split behave exactly as in 0.3.2-preview.1.

Starlings.Tokenizers 0.3.2-preview.1

Added

• ByteLevel pre-tokenizer use_regex: true: the GPT-2 split regex ('s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+), matched over Unicode scalar values with the reference's character classes (a new Gpt2SplitPattern). Previously use_regex: true was rejected at load.
• Profile whisper-bpe: OpenAI Whisper's GPT-2 byte-level BPE tokenizer (vocab 50257 + 1609 added tokens, ByteLevel(add_prefix_space false, use_regex true) + TemplateProcessing).

Behavior

• Tokenizers whose ByteLevel pre-tokenizer declares use_regex: true now load and split on the GPT-2 regex instead of failing with UnsupportedTokenizerComponentException. No change for profiles that declare use_regex: false.

Starlings.Tokenizers 0.3.1

The code of 0.3.1-preview.1 released without a preview suffix, so that a non-prerelease ai-dotnet package can pin it (NuGet NU5104). Only the version and documentation changed.

Behavior

• No change in output for any existing profile: none declares lstrip or rstrip (0.3.0 refused to load such tokenizers), and the same conformance fixtures pass unchanged.

Starlings.Tokenizers 0.3.1-preview.1

Added

• lstrip and rstrip added-token options: an added token can absorb the whitespace on its left/right, matching upstream AddedVocabulary semantics (space_leftmost_at_end / space_rightmost_at_start). single_word remains rejected at load time.

Behavior

• Tokenizers whose added tokens declare lstrip or rstrip now load instead of failing with UnsupportedTokenizerComponentException; matching absorbs adjacent whitespace. No change for profiles that do not use these options.

Starlings.Tokenizers 0.3.0

The first release without a preview suffix, so that the ai-dotnet model packages (0.2.0, the first release with Stable model profiles) can depend on a non-prerelease tokenizer. The code is the code of 0.3.0-preview.1; only the version and documentation changed.

Changed

• Version scheme: releases are 0.MINOR.PATCH; -preview.N is kept for candidates. The API can still change between minor versions (0.x). Install without --prerelease.

Behavior

• No change in output for any profile: the same conformance fixtures pass unchanged.

Starlings.Tokenizers 0.3.0-preview.1

Changed

• Package ID and namespaces renamed from StarlingHq.Tokenizers to Starlings.Tokenizers (namespaces Starlings.Tokenizers, Starlings.Tokenizers.Pipeline). Replace the package reference and update using directives; the API and tokenizer output are otherwise unchanged. StarlingHq.Tokenizers 0.2.0-preview.1 is deprecated in favor of this package.

Behavior

• No change in output for any profile: the same conformance fixtures pass unchanged.

StarlingHq.Tokenizers 0.2.0-preview.1

First release from this repository and under this package ID. Versions up to 0.1.0-preview.4 were built under a planning-era package ID in the Starling planning repository and never published (docs/repository-history.md).

Changed

• License: dual MIT OR Apache-2.0 (previously MIT), copyright StarlingHq. Third-party portions keep their own permissive licenses and notices (THIRD-PARTY-NOTICES.md).
• Package ID and namespaces renamed from the planning-era name to StarlingHq.Tokenizers (the Starling ID prefix is reserved on nuget.org by another owner). Update using directives; the API is otherwise unchanged.
• Test and tool environment variables use the STARLING_ prefix (STARLING_MODELS, STARLING_REQUIRE_ASSETS, STARLING_QWEN_DIFFERENTIAL_DIR, STARLING_GEMMA_DIFFERENTIAL_DIR).

Added

• BERT-family WordPiece components, experimental: the BertNormalizer (defaults only) normalizer, the BertPreTokenizer pre-tokenizer, and the WordPiece model and decoder. They are covered by component reference cases and by loading sentence-transformers/all-MiniLM-L6-v2, BAAI/bge-small-en-v1.5 and cross-encoder/ms-marco-MiniLM-L6-v2. No profile yet.
• Data-driven conformance: every profile with fixture-format-1 cases is tested by ProfileConformanceTests, and reference/tokenizer-json/run_tokenizer_json.py writes those fixtures for any tokenizer.json. It reproduces the existing e5 and Gemma fixtures byte for byte.
• tools/pinned-assets.txt as the single download list, checked against the test pins and profiles.

Behavior

• No change in output for any profile: the same conformance fixtures pass unchanged (xlmr-unigram-e5 49/49, qwen2-bpe-kev 137/137, qwen35-bpe-chat 137/137, gemma-bpe-embeddinggemma2 49/49, gliner2-base-v1-unigram partial).