This is a Haskell implementation of
tiktoken, but just the tokenization
logic. In other words, given an existing encoding (like cl100k_base) you
can tokenize a string (into smaller strings or token ranks).
This means that you can't (yet) use this package to create your own new encodings, but you can use it to consume encodings. In particular, this comes in handy for prompt engineering where you want to use as much of the available prompt tokens as possible (which requires accurately counting tokens).
Encoding speed is ≈2.6-3.1 MB/s on an M1 MacBook Pro (using only one core since this package does not yet support parallel tokenization):
All
Encode 10 MB of Wikipedia
r50k_base: OK (23.88s)
3.356 s ± 151 ms
p50k_base: OK (10.39s)
3.445 s ± 31 ms
p50k_edit: OK (11.13s)
3.693 s ± 240 ms
cl100k_base: OK (11.16s)
3.685 s ± 143 ms
o200k_base: OK (11.01s)
3.648 s ± 134 ms