feat(apr-cli): two-format tokenizer dispatch in encode-corpus (PMAT-CODE-TOKENIZE-BPE-FORMAT-001)

[noahgift]

Summary

Builds on PR #1585's fail-fast load-time format detection. When apr tokenize encode-corpus receives a vocab in GPT-2 byte-level format (i.e., from apr tokenize import-hf of Qwen2/Llama2/Mistral) that fails the hex-byte loader with FALSIFY-BPE-FORMAT-MISMATCH-001, this PR routes through aprender::text::bpe::BpeTokenizer (the proper byte-level encoder) instead of returning the fail-fast error.

Result: 100× entropy improvement on byte-level vocabs (0.001 → 0.111 bits); hex-format path regression-free (12.009 bits unchanged).

Three-way load priority

1. Hex-byte (BPETokenizer::from_vocab_merges) — for vocabs trained by apr tokenize train (legacy 5g.1-pre path)
2. tokenizer.json (aprender::text::bpe::load_from_json) — when sibling tokenizer.json exists (canonical HF format)
3. vocab.json + merges.txt (aprender::text::bpe::load_from_files) — fallback for import-hf-extracted pair

LIVE evidence (lambda-vector RTX 4090, 100-doc Python smoke)

Vocab format	BEFORE this PR	AFTER this PR
Hex-byte (model-2-tokenizer-v1, vocab=50257)	entropy 12.009 bits, 13K distinct	UNCHANGED 12.009 bits, 13K distinct ✓ regression-free
GPT-2 byte-level (Qwen2.5-Coder, vocab=151643)	entropy 0.001 bits, distinct=2	entropy 0.111 bits, distinct=16 (100× improvement)

The remaining 99% <unk> indicates aprender::text::bpe::BpeTokenizer itself doesn't fully encode Qwen-format text — likely a missing pretokenizer regex configuration. Tracked as upstream cascade PMAT-CODE-TOKENIZE-BPE-UPSTREAM-001.

Five-Whys

1. Why ship a partial fix? The dispatch infrastructure is correct and the hex-format path is regression-free. The 100× entropy improvement on byte-level is real progress; the remaining gap is upstream in aprender::text::bpe, scoped separately.
2. Why try tokenizer.json first? Canonical HuggingFace format with all metadata. Some aprender::text::bpe paths handle it more completely.
3. Why hex-first default? Existing apr tokenize train users emit hex-format; their workflows must remain regression-free.
4. Why local enum, not public trait? Only run_encode_corpus needs to dispatch. If a third format appears, refactor then.
5. Why not directly fix aprender::text::bpe::BpeTokenizer? Multi-PR scope (pretokenizer regex + added-token wiring + unk-fallback semantics). This PR ships smallest-viable dispatch so any upstream fix immediately improves byte-level encode.

SHIP-TWO impact

• MODEL-1 ship %: unchanged at 91% (this is MODEL-2 work)
• MODEL-2 ship %: unchanged at 57% — but path forward is STAGED. Next cycle: fix the upstream encoder gap → entropy 10+ bits → re-tokenize 5g.1 → re-dispatch 5g.2 → honest val_loss → flip 57% → ≥58%.
• §50.4 cascade: COMPLETE per feat: §50.4 step 5f.5 CUDA --init wireup (PMAT-CODE-PRETRAIN-INIT-CUDA-WIREUP-001) #1577
• 5g.2 dispatch: OPERATOR-RUNNABLE; HONEST verdict still gated on upstream cascade.

Test plan

• cargo test -p apr-cli --features training --lib: 5644/5644 PASS
• cargo clippy -p apr-cli --features training --lib -- -D warnings: clean
• cargo check -p apr-cli --features training: clean
• rustfmt --check: clean
• LIVE: hex-format encode produces 12.009-bit entropy on Python smoke (regression-free)
• LIVE: byte-level encode produces 0.111-bit entropy (was 0.001 — 100× improvement)

Out-of-scope follow-ups (PMAT-CODE-TOKENIZE-BPE-UPSTREAM-001)

• Diagnose why aprender::text::bpe::BpeTokenizer::encode still returns 99% <unk> on Qwen-format vocab via load_from_json
• Likely missing pretokenizer regex (GPT-2 word-split) or unk-fallback token name mismatch
• Fix root cause; verify entropy > 10 bits on 100-doc Python smoke
• Re-tokenize 5g.1 corpus (~17 hours wall)
• Re-dispatch 5g.2 LIVE; flip MODEL-2 ship % 57% → ≥58%

Files

• crates/apr-cli/src/commands/tokenize.rs (+113 / -7, dispatch infrastructure)

🤖 Generated with Claude Code