Add tokenizer training pipeline and extract src/tokenizer.py

[ncylich]

Summary

• Extract src/tokenizer.py from src/data.py: all tokenizer code (constants, NeedleTokenizer, pre_tokenize(), train_tokenizer(), get_tokenizer(), SP config) now lives in its own module. data.py re-exports everything so existing callers are unaffected.
• Add src/tokenizer_train.py: standalone GCS corpus pipeline with prepare, train, and validate subcommands for running on a GCP VM.
• pre_tokenize() isolates ( ) { } [ ] " , with spaces so BPE never merges them into multi-char tokens.
• user_defined_symbols hardcodes <tool_call>=4, <transcribe>=5, and all 8 structural chars at fixed IDs 6–13.
• Trained tokenizer results: 8192 vocab, 4.63 chars/token compression on multilingual corpus, 0 isolated-char violations. Model/vocab files are gitignored (hosted on GCS).
• Gitignore docs/tokenization_plan.md (internal design doc).

Test plan

• All imports resolve: src.tokenizer, src.data re-exports, src.tokenizer_train
• pre_tokenize() correctly isolates all 8 chars
• Full pipeline ran on e2-standard-4 VM: prepare (3.3M examples, 5.3GB corpus) → train (8192 BPE) → validate
• Isolated char constraint: 0 violations across all 8192 tokens
• Compression ratio: 4.63 chars/token (corpus), 4.76 (English)
• Special token IDs match constants (PAD=0, EOS=1, BOS=2, UNK=3, tool_call=4, transcribe=5)
• Verify needle train --toy still works end-to-end with new tokenizer