Skip to content

tokenizer v0.2.0

Choose a tag to compare

View all tags
@vinceswu vinceswu released this 11 Sep 02:52
· 1 commit to master since this release
tokenizer-0.2.0
34b4363
This commit was signed with the committer’s verified signature.
vinceswu Vince Swu
SSH Key Fingerprint: lWR69zo/uAip0DoEhaIsyJLVf02PRq7TJ8bOY3PlLnA
Verified
Learn about vigilant mode. 
tokenizer-0.2.0

Add full tiktoken preset parity: get_encoding, encoding_for_model, gp…
Assets 3
Loading