Skip to content

v0.12.1 — Vectorized BPE training

Choose a tag to compare

View all tags
@cattolatte cattolatte released this 07 Jul 01:32
v0.12.1
4bffbff
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

BPETokenizer.train is now vectorized with numpy — ~8× faster than the naive Python loop (gap grows with corpus size), byte-level semantics and losslessness unchanged. numpy is imported lazily so encode/decode stay dependency-light.

Assets 2
Loading