fix(fast): training no longer mutates the global cluster cache

[AmitMY]

Fixes #56 — and surfaces something bigger found while gating it.

The bug

apply_merge_to_cluster_cache mutated the global cluster-graph memo as training progressed, turning it into trained state: the second training run in a process produced different merges than the first. Repro (now a shared test, test_repeated_training_gives_identical_merges, all four tokenizers): corpus café déjà …, 4 merges — cold run merges é's UTF-8 bytes, warm run finds them pre-merged in the cache and picks a different merge. The reference is stateless and matches fast's cold run.

The mutation served nothing: the reference has no cluster cache at all (cluster building is a pure function), so no behavior can legitimately depend on merged cached graphs. Deleted, along with its call sites and the now-empty on_merge hook. It was also the hidden cause of flakiness potential in test_node_count_with_chinese_ids (Han handler Trees cached and mutated between builds).

One digest legitimately changes: SuperBPE full-scale 2a6baff51d → 30929f0c51 (phase 2 previously built graphs from clusters mutated by phase 1's 250 merges). Proven correct against the reference at 20k rows / 500 merges: pre-fix fast ae6475b5ef ≠ reference; post-fix fast b308cdb768 = reference b308cdb768. All other digests unchanged; warm == cold now holds at full scale.

⚠️ The bigger find: the fast test gate was testing the reference

cd fast && pytest ../tests/ never loads fast/conftest.py under pytest 9 (conftests apply to their directory subtree; a cwd conftest is not an initial conftest for sibling paths) — complex_tokenization resolved to the reference package, so every recent "fast suite passed" validated the reference twice. Digest/benchmark verifications were unaffected (they import complex_tokenization_fast directly); only the pytest gate was hollow.

Running the suite properly (pytest ../tests/ -p conftest, now documented as REQUIRED in CLAUDE.md) exposed two unmirrored APIs, both added to the fast shim:

• GraphSettings.TRADE_MEMORY_FOR_SPEED (every aliased test errored in a fixture without it)
• Tokenizer(cache_maxsize=...) incl. the word-graph lru_cache dedup in _build_graphs and kwargs-forwarding subclasses

Verification

• New repro test: 4/4 fail pre-fix under the aliased suite, pass post-fix; reference passes throughout.
• Suites: fast (properly aliased) 145 passed, 2 skipped; reference 145 passed, 2 skipped.
• Cold full-scale digests: BPE 2af4c7edca ✓, BNE 967201e51c ✓, Boundless 10318b5e1c ✓, SuperBPE 30929f0c51 (reference-proven). 2000-row defaults: all four unchanged.
• Times improved as a side effect (the shim's word-graph dedup): full-scale Boundless 7.65→6.61s, SuperBPE 9.31→7.14s, BPE 3.5→2.1s, BNE 4.3→2.9s. No regressions.

🤖 Generated with Claude Code