Remove fastokens entirely

[hallerite]

Why

fastokens is a process-global monkey-patch on transformers.AutoTokenizer that swaps .backend_tokenizer for a faster Rust BPE shim. The shim has no offset-mapping support, so we kept a parallel vanilla tokenizer per model purely for body/scaffold attribution in attribute_text_segments. After the v3 collapse-or-fallback refactor (PR #87), most assistant messages render mixed-label (scaffold+content+scaffold) and run through the offset path where fastokens contributes nothing; fastokens only saves time on homogeneous-label runs and direct _encode/_decode helpers.

The complexity tax it was charging:

• FASTOKENS_INCOMPATIBLE frozenset (DeepSeek-V3 family workaround — only exists because fastokens lacks Metaspace pre-tokenizer support)
• _FASTOKENS_PATCH_LOCK + _FASTOKENS_ANNOUNCED global state
• _patched_load() with contextlib.redirect_stdout to swallow [fastokens] patch_transformers: ... prints under thread contention
• The unpatch/restore race-safe path inside _get_offset_tokenizer (PR fix: _get_offset_tokenizer immune to global fastokens patch (concurrent-pool race) #86)
• Two tokenizers per model resident in memory
• Tests/CI gates around every one of the above

…wasn't earning its keep.

What's removed

Surface	Lines
fastokens>=0.2.0 dep + [tool.uv.exclude-newer-package].fastokens exemption	pyproject.toml
FASTOKENS_INCOMPATIBLE, _FASTOKENS_PATCH_LOCK, _FASTOKENS_ANNOUNCED, _patched_load	base.py
use_fastokens kwarg on load_tokenizer (signature → 1 positional arg)	base.py
Fastokens-failure-retry-vanilla branch in load_tokenizer	base.py
Orphan imports: top-level contextlib, io	base.py
_offset_tokenizers cache + lock + entire unpatch-and-reload block	base.py
tests/test_load_tokenizer_fastokens.py (211 lines, all fastokens-specific)	tests/

What replaces it

_get_offset_tokenizer collapses to its essential contract — probe and return, or raise:

def _get_offset_tokenizer(tokenizer):
    try:
        tokenizer("a", add_special_tokens=False, return_offsets_mapping=True)
    except (NotImplementedError, ValueError, TypeError) as exc:
        raise RuntimeError(
            "Hand-coded renderers require a fast tokenizer with "
            "``return_offsets_mapping=True`` support …"
        ) from exc
    return tokenizer

Tokenizers from load_tokenizer are PreTrainedTokenizerFast and satisfy this trivially. BYO tokenizers without return_offsets_mapping support fail loudly at construction time instead of silently triggering a reload from name_or_path that only existed to paper over the fastokens shim's missing offsets.

The single test that exercised the BYO-no-offsets reload path is replaced with one asserting the new error contract (test_get_offset_tokenizer_rejects_offsetless_byo).

Perf impact (honest)

Encode/decode now go through HuggingFace's stock tokenizers Rust BPE instead of fastokens (claimed ~10x faster on synthetic benchmarks). The mixed-label path — which is most assistant turns post-PR-#87 — was already off fastokens before this change, so the realistic-workload regression is bounded to homogeneous-label runs and direct encode/decode helper calls.

Not measured. If it matters in real training pools, revisit by adding fastokens back as the offset tokenizer once it supports offset_mapping upstream (filed: TODO link if a fastokens issue exists). Until then, the simpler tree wins.

Tests

• Before (origin/main): 2259 passed, 88 skipped, 1 xfailed
• After (this PR): 2248 passed, 88 skipped, 1 xfailed
• Delta: 11 deleted fastokens-specific tests, no regressions

Stats

 pyproject.toml                         |  12 --
 renderers/base.py                      | 243 ++++-----------------------------
 tests/test_load_tokenizer.py           |  46 ++-----
 tests/test_load_tokenizer_fastokens.py | 213 -----------------------------
 uv.lock                                |  36 +----
 5 files changed, 44 insertions(+), 506 deletions(-)

Net −462 lines.

Followups

• PR v3 collapse closures + tokenizers.Encoding offset API #87 (drop transformers from offset path) needs a rebase onto this; most of its _get_offset_tokenizer rewrite becomes simpler/obsolete since the path-4 AutoTokenizer fallback existed for fastokens-induced reasons.
• "Make transformers an optional dependency" (issue Is transformers necessary or tokenizers is enough? #31, plan at .claude/plans/enchanted-sniffing-scott.md) becomes simpler — load_tokenizer is now the only transformers user on the construction path.

🤖 Generated with Claude Code

---

Note

Medium Risk
Behavior change for callers passing use_fastokens or relying on silent offset-capable tokenizer reload; encode performance may regress on homogeneous-label paths. Core rendering security policy is unchanged.

Overview
Removes the fastokens optional dependency and all tokenizer monkey-patching from renderers.base, shrinking the load path to vanilla AutoTokenizer / PreTrainedTokenizerFast only.

load_tokenizer no longer accepts use_fastokens; it always goes through _load_tokenizer_via_auto with the existing security policy (pinned Kimi revisions, Llama unsloth mirrors). Deleted surfaces include FASTOKENS_INCOMPATIBLE, _patched_load, patch locks, stdout suppression, and fastokens failure fallback.

_get_offset_tokenizer is simplified to a probe-and-return contract: the supplied tokenizer must support return_offsets_mapping=True, or a RuntimeError is raised. The per-model vanilla offset cache and fastokens race-safe reload path are gone—callers with BYO tokenizers must pass a fast tokenizer explicitly.

Tests and lockfile drop tests/test_load_tokenizer_fastokens.py, update test_load_tokenizer.py (including test_get_offset_tokenizer_rejects_offsetless_byo), and remove fastokens from pyproject.toml and uv.lock.

Trade-off: encode/decode use stock HuggingFace tokenizers instead of the fastokens shim (~10x encode claim); mixed-label / offset attribution paths were already on vanilla HF for offsets.

^{Reviewed by Cursor Bugbot for commit 33c3f39. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Remove fastokens dependency and all related tokenizer patching logic

• Removes the fastokens package from dependencies and deletes all integration code, including _patched_load, the use_fastokens parameter on load_tokenizer, and the _offset_tokenizers cache.
• load_tokenizer now always loads a vanilla fast tokenizer via AutoTokenizer with no retry or fallback logic.
• _get_offset_tokenizer now raises RuntimeError instead of transparently loading a separate offset-capable tokenizer when the supplied tokenizer lacks offset mapping support.
• Deletes tests/test_load_tokenizer_fastokens.py and updates tests/test_load_tokenizer.py to match the new behavior.
• Risk: callers relying on the automatic offset-tokenizer fallback or the use_fastokens parameter will get a TypeError or RuntimeError at runtime.

^{Macroscope summarized 33c3f39.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR removes the fastokens dependency which provided ~10x tokenization speedup. While the code simplification is clean, removing a significant performance optimization is an architectural decision that warrants human review to confirm the tradeoff is intentional.

^{You can customize Macroscope's approvability policy. Learn more.}

[hallerite]

this is mostly because we currently need offset renderers everywhere. we will switch back to fasttokens once they support it.