"I built a tokenizer to learn how tokenizers work. It works. That is the only nice thing I can say about it."
A fully functional, from-scratch tokenizer built for one purpose and one purpose only: learning. Not speed. Not accuracy. Not usability. Just vibes and vocabulary lists.
If you were looking for tiktoken, sentencepiece, or literally any tokenizer maintained by people who know what they're doing — this is not that. Turn back now.
Because at some point I stared at a wall of subword tokens and thought "I bet I could build that." I could not, in fact, build that — not well, anyway. But I built a version of that, and along the way I actually learned how the real ones work. So mission technically accomplished, dignity technically lost.
- ✅ Splits text into tokens (the bar was on the floor and I still tripped over it)
- ✅ Implements basic → intermediate tokenization concepts (whitespace splitting, BPE-ish merging, vocab building)
- ✅ Fully documented, mostly for my own sanity
- ✅ Zero (0) production use cases
- ✅ Guaranteed to be slower than whatever you're currently using
- ❌ Not optimized
- ❌ Not benchmarked against anything real
- ❌ Not something you should put anywhere near a real ML pipeline
This repo is basically a diary of me going:
- "Tokenization is just splitting on spaces, right?" (wrong)
- "Okay it's about subwords." (closer)
- "Byte-Pair Encoding... I understand 60% of this." (progress!)
- "I built my own BPE merge logic and it kind of works?!" (victory lap)
- "...it is still useless." (acceptance)
git clone https://github.com/yourusername/very_useless_tokenizer.git
cd very_useless_tokenizer
# congrats, you now have a tokenizer you will never use in productionfrom very_useless_tokenizer import Tokenizer
tokenizer = Tokenizer()
tokens = tokenizer.encode("why did I build this")
print(tokens)
# output: proof that learning happened, utility did not| Tokenizer | Speed | Accuracy | Should you use it |
|---|---|---|---|
| tiktoken | ⚡⚡⚡⚡⚡ | ✅✅✅✅✅ | Yes |
| SentencePiece | ⚡⚡⚡⚡ | ✅✅✅✅✅ | Yes |
| very_useless_tokenizer | 🐌 | 🤷 | Absolutely not |
- How BPE merges work under the hood
- Why vocab size vs. sequence length is a real tradeoff and not just a config number I copy-pasted
- Why professional tokenizers are written in Rust and mine is not
- Humility
- Fast
- Correct in every edge case
- A replacement for anything
- A cry for help (it is a little bit a cry for help)
If you'd like to contribute, first ask yourself why. Then, if you still want to, PRs are welcome — bonus points if you can make it more educational without making it less useless.
MIT — do whatever you want with it, I certainly won't be held responsible.
Built with love, confusion, and an unreasonable number of print statements.