Skip to content

v0.1.17

Latest

Choose a tag to compare

@rybruscoe rybruscoe released this 05 Sep 10:22
· 6 commits to main since this release

Sentencepiece tokenisation now matches llama.cpp's space-prefix rule (Mistral / Llama-2 style vocabularies). Mistral-7B-Instruct verified under the exact profile: byte-identical across binaries, Go reference reproduces every row of its dump. Tokenizer conformance vectors for five models.