Add SentencePiece/LLaMA tokenizer path (#464)

[michalharakal]

Closes #464.

Adds the SentencePiece path that was deferred from the #463 byte-level BPE fix: LLaMA / TinyLlama / Gemma / Mistral-v0.1 GGUFs now tokenize correctly through the same TokenizerFactory that already routes Qwen/GPT-2 to byte-level BPE.

Summary

• SentencePieceTokenizer (commonMain) — llama.cpp-style SPM:
  • Whitespace escape (space → ▁ / U+2581) with configurable addSpacePrefix
  • Code-point symbol split (surrogate-pair safe)
  • Score-priority BPE: the adjacent pair whose concatenation is in the vocab with the highest score wins (the SPM rule, opposite of the merge-rank rule used by QwenByteLevelBpeTokenizer)
  • Byte fallback via <0x00>..<0xFF> hex tokens for any leftover symbol that isn't in the vocab
  • Decode that recognizes <0xNN> tokens, accumulates them into raw bytes, UTF-8 decodes, and strips the leading space prefix
• TokenizerFactory wiring: fromGguf now dispatches tokenizer.ggml.model == "llama"/"sentencepiece" to the new tokenizer; fromTokenizerJson dispatches model.type == "Unigram" likewise. Both branches previously threw UnsupportedTokenizerException.
• GGUF UInt cast bug fix (latent in Byte-level BPE broken for GPT-2/Qwen models (affects both GGUF and SafeTensors) #463): StreamingGGUFReader returns UINT32 fields as kotlin.UInt (a value class — not a Number subclass), so the previous as? Number cast was silently dropping bos_token_id / eos_token_id / unknown_token_id. New toIntFlexible helper handles every signed/unsigned numeric type GGUF can produce, and the same fix is applied to QwenByteLevelBpeTokenizer.fromGgufFields.

Verification

End-to-end against real TinyLlama-1.1B-Chat-v1.0 (download-on-demand fixture, skips cleanly when absent):

Test	Expected	Result
"Hello"	[15043]	✓
"The capital of France is"	[450, 7483, 310, 3444, 338]	✓
ASCII encode/decode round-trip	identity	✓
CJK ("日本") byte-fallback round-trip	identity	✓
bosTokenId / eosTokenId from GGUF	1 / 2	✓
GGUF dispatches to SentencePieceTokenizer	yes	✓
tokenizer.json dispatch + round-trip	yes	✓

Reference token IDs come from HuggingFace transformers for TinyLlama/TinyLlama-1.1B-Chat-v1.0.

The synthetic SentencePieceTokenizerCoreTest in commonTest exercises whitespace escape, merging to single tokens, byte fallback for unknown Latin / CJK characters, and interleaved-token decode without needing any fixture (so CI without network stays green).

QwenByteLevelBpeTokenizerFixtureTest (#463) is also re-run and stays green — the UInt cast fix didn't break the Qwen path.

Files

New (commonMain):

• SentencePieceTokenizer.kt — core + GGUF and tokenizer.json builders
• GgufFieldHelpers.kt — toIntFlexible helper

New (commonTest):

• SentencePieceTokenizerCoreTest.kt — 9 synthetic algorithm tests

New (jvmTest):

• SentencePieceTokenizerFixtureTest.kt — 7 end-to-end TinyLlama tests

Modified:

• TokenizerFactory.kt — wire llama/Unigram branches
• QwenByteLevelBpeTokenizer.kt — adopt toIntFlexible for BOS/EOS
• TokenizerFactoryDispatchTest.kt — assert dispatch where it used to assert throws
• build.gradle.kts — downloadTinyLlamaTokenizerFixtures task
• CHANGELOG.md

Test plan

• ./gradlew :skainet-io:skainet-io-core:jvmTest — full core suite green
• ./gradlew :skainet-io:skainet-io-core:downloadTinyLlamaTokenizerFixtures then re-run jvmTest — all 7 fixture tests pass against the real TinyLlama GGUF
• Cross-check sample encodings against HuggingFace transformers (already done locally; outputs above)
• Confirm Qwen Byte-level BPE broken for GPT-2/Qwen models (affects both GGUF and SafeTensors) #463 path still green after the UInt cast change
• Verify offline build (no network) stays green via the skip path

🤖 Generated with Claude Code