Skip to content

Tokenizer 3.6.4

Choose a tag to compare

View all tags
@vthorsteinsson vthorsteinsson released this 10 Jul 12:09
· 15 commits to master since this release
v3.6.4
7ce5d94
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Tokenizer 3.6.4 improves tokenization throughput without changing token output or the public API.

What changed

Raw whitespace-delimited tokens are now constructed directly with lazy origin offsets. Normal tokenization skips a redundant second rough-token pass; the additional pass is retained when HTML escape replacement is enabled because replacements such as   can introduce whitespace.

Performance

On test/toktest_large.txt (573 KB, 149,040 output tokens), median CPython tokenization time decreased from 1.905 seconds in 3.6.3 to 1.514 seconds in 3.6.4, a 20.5% reduction. Under cProfile, total calls decreased from 9.32 million to 7.77 million.

Compatibility and verification

  • Token kinds, text, values, original text, and materialized origin spans match 3.6.3 across the complete large corpus.
  • HTML escape and Unicode replacement behavior is preserved.
  • The full suite passes on CPython 3.10 through 3.14 and PyPy 3.11.
  • The source distribution and universal wheel pass twine check and a clean-venv installation test.

Upgrade with:

pip install --upgrade tokenizer

Full changes: #62

Assets 2
Loading