Skip to content

Starlings.Tokenizers 0.3.4

Choose a tag to compare

View all tags
@MarkWalls MarkWalls released this 31 Oct 23:40
· 24 commits to main since this release
v0.3.4
e39c03a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Added

  • Profile clover-bpe-chat (Diffuspace/Clover-35B-A3B-Preview at 4b0402b6), experimental: byte-level BPE with the Qwen2 regex and the diffusion mask token <|mask|> at vocabulary id 151669, 137/137 cases exact against huggingface/tokenizers 0.23.3, with and without special tokens, truncation to 16 tokens, and padded batches of single texts and pairs. No library change.

Behavior

  • No change in output for any existing profile; no library change.
Assets 4
Loading