toks 0.3.2

toks 0.3.2

toks is Actual Computer's tokenizer: give it the tokenizer.json a model ships with and it returns exactly the ids
Hugging Face tokenizers 0.23.2 returns, from a small C library with hand-written asm kernels for arm64 (NEON) and
x86-64 (AVX2), no runtime and no dependencies, plus a Python package with an hf-style Tokenizer API. 0.3.2 is the
first release under the Apache License, Version 2.0; the release commit is 55a5230b08a75916a2b36f92c320f057376833ea.

License

From this release on, toks is open source under the Apache License, Version 2.0
(SPDX Apache-2.0): use it, modify it, vendor it and ship it inside anything, closed products included, at any
scale, keeping the LICENSE and NOTICE files with your copies. 0.3.0 and 0.3.1 shipped under the Business Source
License 1.1 and their tags keep that text. LICENSING.md is the plain-language page. The wheels declare
License-Expression: Apache-2.0 and carry LICENSE, NOTICE, LICENSING.md and THIRD_PARTY_NOTICES.md under
licenses/; the bundles carry the same four files, each named in their MANIFEST with its sha256.

What's new since 0.3.1

• ABI 0.4: the hf and tiktoken primitives (#13). toks_template (the post-processor's single-sequence
  template), toks_added (the added tokens in id order, with their lstrip / rstrip / single_word / normalized
  flags), TOKS_NO_TRUNCATE / TOKS_NO_PAD (encode without the file's truncation and padding) and
  TOKS_DECODE_RAW (the bytes the ids spell, never repaired); toks_info reports the file's truncation, padding
  and the template's shape, and an abi 0.3 caller's toks_info size is still answered. TOKS_ABI_MINOR 3 -> 4;
  TOKS_ABI_MAJOR stays 0.
• Python: hf's Encoding and tiktoken's Encoding on toks.Tokenizer, over abi 0.4 (#34), and
  Tokenizer.encode_bound(n), the C library's toks_encode_bound (#20).
• Exactness: a SentencePiece pending unk stays pending across byte-fallback chars, as hf's merge_word keeps
  it (#36); config.c refuses what hf refuses before any model reads the file (#22); the last id, 2^21 - 2, loads
  (#24).
• Memory geometry: make test-guard puts every table and every scratch region on its own pages, and a fault on
  a guard page says where (#14, #38, #39, #40, #47); no writable data in the C core (#24); toks_plat_arena's first
  touch is a write, so toks_par's scratches keep their first 2 MiB frame huge on a kernel that splits the huge
  zero page (#15, #29); huge pages for a default scratch, measured (#46).
• The stall screen (#35): test_stall names suspects and fails growth at x8 then x12 from 128 KiB to 2 MiB;
  test_par's parallel floor (#7).
• Measured and not taken, with their raw logs in the tree: the memo's keyed check and bytes-then-check records
  (#28, #41), the low-id pair grid on avx2 (#6), K6's round one in two passes (#23).
• Documentation and tooling: the gigatoken gate's picture on the default scratch, its m6 paragraph generated
  from the probe's log (#45, #49); receipts that name their commits (#30); the README's tiktoken range by the hf
  clause's rule (#42, #43); two more x86-64 machines in docs/machines.md (#27); the external pull request policy
  decided by repository permission (#33); the release-artifacts workflow (#26); the API audit's severity-3 tests
  (#21).

Exact

Every result is checked against hf tokenizers 0.23.2 (Kimi K3 against its own reference):

• the release commit's CI: make test on both tiers on linux x86-64 and arm64, macOS arm64 and Windows x86-64
  (the Actions runs on 55a5230b08a75916a2b36f92c320f057376833ea: test.yml run 37710610274 for linux x86-64
  and arm64 on both tiers, windows.yml run 37710610279, macos.yml run 37710610281);
• the nightly full-parity run dispatched on the release commit: run 37710682088: x86_64-scalar: 21 suites, 3964440
  compared, 0 failed; arm64-auto: 21 suites, 3964440 compared, 0 failed; x86_64-auto: 21 suites, 3964440 compared, 0
  failed; arm64-scalar: 21 suites, 3964440 compared, 0 failed
  (4 jobs x 21 suites, 3,964,440 cases each, 0 failed); its norm job: 313,357 compared, 0 failed; its bpe job:
  4 suites, 4,626,610 compared, 0 failed.
• the 0.3.0 release report (docs/release/0.3.md) for the release machines' parity samples, sanitizer runs and
  wheels at 0.3.0. The C core and header of 0.3.2 differ from 0.3.1's in 24 files under src/ and include/
  (732 insertions, 173 deletions): the ABI 0.4 primitives (#13), spm_c.c's pending unk (#36), config.c's
  refusals (#22), no writable data and the last id's load (#24), the guard build's table and scratch pointers (#14)
  and the arena's first touch (#15); the asm kernels are unchanged.

Fast

The speed table (docs/bench/e2e.md) stands at commit 245cc5c, the 0.3.0 release candidate: 13-151x faster than
hf tokenizers in every cell on the asm tiers, 4-23x faster than tiktoken in every cell it can run, ahead of
gigatoken cold in 252 of 255 cells. 0.3.2 is a license and patch cut and does not re-measure it; the table is
re-measured at a later cut.

Where the artifacts were built

Each set is tools/release.sh at the release commit on the machine named (docs/machines.md), from a tree synced by
tools/remote.sh: make test on both tiers first (every suite 0 failures), then the bundle, then the five wheels
with their tests on each CPython (cp313 runs the whole Python suite).

• linux arm64: the GB10 release machine (gb10c), load 0.07 at the start; cp313: 324 passed, 46 skipped.
• macOS arm64: the M2 Ultra release machine (m2ultra2), load 1.2; cp313: 313 passed, 46 skipped.
• linux x86-64: the 9970X release machine (tr9970x), reachable again for this cut, load 2.7, the build pinned to
  taskset -c 0-7 (untimed); cp313: 314 passed, 46 skipped. The bundle's libtoks.so floor is GLIBC_2.34 (its
  MANIFEST's glibc line); the wheels are manylinux2014.

Known limits

The 0.3.0 report's UNMET table (docs/release/0.3.md) still applies: the fuzzing budget (24 cpu-hours per ISA per
entry point), the proof package, the gigatoken gate on the new default scratch, the integration package,
control-token isolation, the Python binding's per-target parity on the release machines, and the speed table at
the release commit.

Pinning (docs/release.md, "How a consumer pins toks")

toks 0.3.2 commit 55a5230b08a75916a2b36f92c320f057376833ea
linux-x86_64 sha256 e0780282bd8caabcee355e12404d2d3ead3ed5a52442fb73eb88ee2ba4b23b76
linux-arm64  sha256 d273392667d6d80f66cf61a61e3129a7dfca16176b0be990752a0d08e4fd2fb8
macos-arm64  sha256 410f4981ba76a1b473640d0fcd9bb63b457b841abf4c0f2027826badc61fe82b

Integration flags: one scratch per worker thread, kept across requests, initialized with flags 0; encode calls with
flags 0. Every file's sha256: SHA256SUMS. The wheels are attached here and not published to PyPI.