toks 0.3.3

toks 0.3.3

toks is Actual Computer's tokenizer: give it the tokenizer.json a model ships with and it returns exactly the ids
Hugging Face tokenizers 0.23.2 returns, from a small C library with hand-written asm kernels for arm64 (NEON) and
x86-64 (AVX2), no runtime and no dependencies, plus a Python package with an hf-style Tokenizer API. Apache-2.0.
The release commit is f97143b301a248319c62fb460f4914407a5eb41e. ABI 0.4, as 0.3.2.

What's new since 0.3.2

• SPEC §3.6, structural control-token isolation, certified per tokenizer (#60). toks_info.control_isolation
  is now proven at load. It means no added token's id, special or not, comes out of a TOKS_ADDED_NONE encode of
  any bytes. A positional chat encoder relies on this: it encodes client text with no added-token recognition and
  places the template's control tokens by id. src/core/reach.c excludes every way byte-level bpe can emit an added
  id:

  • not a vocab id, or a decode-only string;
  • not one byte, and no merge makes it;
  • not the unk of a byte the vocab lacks;
  • under ignore_merges / tiktoken ranks, never a whole piece of the chain's last split. The generic engine's own
    program decides this: a match through exactly its atoms with assertions relaxed, or a gap with no match start.

  The proof and the census are in docs/algorithms/reach.md. Every byte-level file in the tokenizer cache is
  certified but gx-spark-x25: its added <unk> is the unk of the bytes its vocab lacks, and "\xFF" encodes to it.
  Nemotron-family files keep their added tokens in the vocab under ignore_merges, and the piece test certifies
  them. spm, unigram and wordpiece report 0 (not analyzed). tests/c/test_reach.c checks it:

  • red files, each uncertified, with a text whose NONE encode gives the added id;
  • green files;
  • each certified file searched;
  • mutants of every branch.

• Exactness: the generic engine's byte domain (#60). In a Split after a ByteLevel, bytes 7F..A0 had the wrong
  alphabet chars (byte + 210 instead of gpt-2's byte + 162; 0x96 is U+0138, a lowercase letter, not U+0168). The
  breadth fixture gen_bytes carries hf's expectations.

Exact

Every result is checked against hf tokenizers 0.23.2:

• the release commit's CI on f97143b301a248319c62fb460f4914407a5eb41e:
  • test.yml run 40429344902: linux x86-64 and arm64, both tiers;
  • macos.yml run 40429344911;
  • windows.yml run 40429344899. Attempt 2 passed. Attempt 1 failed only test_par's thread count (3 threads after
    destroy on the shared runner; the scalar pass of the same attempt was clean). The same code passed windows.yml
    at 67316dc.
• the nightly full-parity run dispatched on the release commit, run 40429363251:
  • parity: 4 jobs (x86_64 and arm64, auto and scalar), 21 suites, 3,964,440 compared each, 0 failed;
  • norm: 313,357 compared, 0 failed;
  • bpe: 4 suites, 4,626,610 compared, 0 failed;
  • wheels: 113 suites, 1,303,805 compared, 0 failed.
• tests/parity/run.py on five large byte-level files (GLM-5.3, Qwen3.8, Nemotron 3.5, DeepSeek-V4.1-Flash and a
  private harmony-format file), on a linux x86-64 lab box: gen_cases.py's quick set (~12.4M cases each) plus a
  real-text corpus. 0 diffs on every file; control_isolation 1 on all five.
• The C core and header differ from 0.3.2's in 8 files under src/ and include/ (170 insertions, 7 deletions):
  reach.c, gen.c's piece test and byte-domain fix, the config's last-split record. The asm kernels are unchanged.

Where the artifacts were built

• linux x86-64 and linux arm64: the release-artifacts workflow at the tag (run 40429347575), clean Blacksmith
  runners with the pinned clang 21.1.8. tools/release.sh runs make test first, then the bundle, then the five
  wheels with their tests (cp313 runs the whole Python suite: 325 passed, 44 skipped on each). The bundles'
  libtoks.so floor is GLIBC_2.34; the wheels are manylinux2014.
• macOS arm64: tools/release.sh at the tag on an Apple-silicon Mac (Apple clang 21.0.0, the m2ultra machines'
  Xcode), with the M2 Ultra release machines unreachable from this cut. cp313: 327 passed, 46 skipped.

Known limits

The 0.3.0 report's UNMET table (docs/release/0.3.md) still applies, but control-token isolation is now met for
byte-level bpe. The speed table stands at 245cc5c (0.3.0's candidate); 0.3.3 does not re-measure it.

Pinning (docs/release.md, "How a consumer pins toks")

toks 0.3.3 commit f97143b301a248319c62fb460f4914407a5eb41e
linux-x86_64 sha256 dc3a47c61e6a61b6d417a722d007a7d7e86beacf8547801be2d8deedd4c362ca
linux-arm64  sha256 e0c2b20877454a9fd7ad7e46ff18bdb42dbaaf0df4b474443743ba4ac60b07f7
macos-arm64  sha256 d048b61b52f059778f7b7dec61341e7ff1672ca50bccef18f326c3b9abb9c0fb

Integration flags: one scratch per worker thread, kept across requests, initialized with flags 0; encode calls with
flags 0. Every file's sha256: SHA256SUMS. The wheels are attached here and not published to PyPI.