v1.0.0

tiktokenizer

An interactive playground for comparing how different language models tokenize the same text.

Getting started

See README.md in the repository for installation, configuration and usage examples.