apr convert: APR file missing embedded tokenizer for SentencePiece models (internlm2.5)

[noahgift]

Bug

APR inference fails with PMAT-172: APR file missing embedded tokenizer for models that use SentencePiece (tokenizer.model) instead of tokenizer.json.

Reproduction

apr pull internlm/internlm2_5-7b-chat
apr convert ~/.apr/cache/hf/internlm/internlm2_5-7b-chat/model.safetensors.index.json --format apr
apr run output.apr -p "Hello" --max-tokens 10
# ERROR: APR file missing embedded tokenizer

Root Cause

The APR converter embeds tokenizer data from tokenizer.json (HuggingFace tokenizers format). Models using SentencePiece (tokenizer.model) don't have tokenizer.json — they have tokenizer.model + tokenizer_config.json.

Affected Models

• internlm/internlm2_5-7b-chat
• internlm/internlm2_5-20b-chat
• teknium/OpenHermes-2.5-Mistral-7B
• Potentially others using SentencePiece

Expected Behavior

APR converter should detect tokenizer.model (SentencePiece) as a fallback when tokenizer.json is absent and embed it in the APR file.

[noahgift]

Fixed in aprender@2e88310a + realizar@a181eaa.

Changes:

• aprender: Added SentencePiece protobuf parser in tokenizer_loader.rs — extracts vocabulary (92,544 tokens) and Viterbi scores from tokenizer.model
• aprender: Embeds scores in APR metadata as tokenizer.scores JSON array
• aprender: GgufTokenizer gains scores: Vec<f32> field
• realizar: Added get_embedded_scores() and load_embedded_sentencepiece_tokenizer() methods
• realizar: encode_text() falls back to SentencePiece after BPE fails
• realizar: Inference fallback also converts SentencePiece to BPE-compatible for decode

Verified:

• Parsed real internlm2.5-7b-chat tokenizer.model: 92,544 vocab entries ✓
• APR import embeds tokenizer correctly: [GH-366] Loaded SentencePiece tokenizer: 92544 vocab tokens ✓
• APR inference loads embedded tokenizer: [GH-366] Loaded embedded SentencePiece tokenizer: 92544 vocab tokens ✓
• PMAT-172 error ("APR file missing embedded tokenizer") no longer triggers ✓
• All tests pass: aprender 12,628 ✓, realizar 15,033 ✓
• Zero clippy warnings in both repos ✓