v2.0.0 — the language list stops being hand-written

The final TokenDamper release. C and C# reduce under a new deep engine, which ships as its own package, tokendamper-deep. --mode now selects that engine. Nothing is scheduled after this release; DECISIONS §89 closes every held item, saying why it was held and what would have unblocked it.

Breaking

• --mode selects the engine: fast (the default) or deep, on optimize only.
  • --mode bench and --mode optimize are now errors that name tokendamper bench and tokendamper optimize.
  • --engine-mode is withdrawn, and the error names --mode fast|deep.
• app.mode and TOKENDAMPER_APP_MODE are ignored, with a one-line notice on stderr. Nothing ever read them. So do the variables withdrawn earlier (TOKENDAMPER_RISK_TOLERANCE, _MAX_OUTPUT_TOKENS, _MAX_LATENCY_MS, _TRACE_OUTPUT), which until now loaded silently.
• The engine can also be set with engine.mode or TOKENDAMPER_ENGINE_MODE. If either resolves to deep, bench and mcp refuse to start, because neither runs the deep engine. The error names TOKENDAMPER_ENGINE_MODE=fast as the per-command override.

New: C and C# under --mode deep

npm install -g tokendamper tokendamper-deep
tokendamper optimize src/server.c --mode deep --target-reduction-ratio 0.3

At ratio 0.3, each corpus with its own fallback rate:

corpus	source: saved (fell back)	tests: saved (fell back)
redis (C)	6.76% (34.3%)	32.04% (6.1%)
curl (C)	9.59% (30.8%)	35.76% (17.8%)
Newtonsoft.Json (C#)	20.12% (5.5%)	26.69% (0.6%)
jellyfin (C#)	21.46% (6.2%)	25.05% (11.7%)

A fallback is fail-open: you get your input back. C falls back most because TokenDamper refuses to drop a comment phrased as an instruction (always, must, do not), and C's block comments are full of them. In fast mode, C and C# are now validated by new bracket and quote lexers but never reduced. Those lexers flag 2 of 7,914 real C files and 0 of 6,230 C# files.

The default path moves for Python and TypeScript

• Fast Python reads wrapped and async def headers (§83). On pip, the file route goes 17.75% → 19.53%. It costs fallbacks on async-heavy code: across three corpora, 2 recovered and 9 new. That trade was accepted on purpose.
• A compound statement is one span (§86). Elision no longer leaves an else, catch, finally or do's while attached to a marker. On the main corpus, 29 rows change and none newly falls back. Adherence is slightly worse: TypeScript rows above 50% go 8 → 10.
• The Python validator reads \ line continuations (§88). It had falsely flagged 1,110 of the 13,897 .py files in a CPython 3.12 install, against 2 now. On CPython's standard library, the file route goes 7.24% → 8.60% with no new fallbacks.

Not established

• An install of tokendamper-deep on a platform without prebuilt grammar bindings. Prebuilds cover darwin, linux and win32 on x64 and arm64.
• Deep through bench or MCP. It is refused rather than implemented.
• Stdin for C or C# without --language.

The full record is DECISIONS §84–§89 and the CHANGELOG.