Tokenizer

You all know LLM takes tokens as input. But how to convert natural language into tokens? This is the job of tokenizer. In this repo, I provide a simple tokenizer for both Chinese and English.