| LLM | Tokenizer | Vocab Size | Avg Char/Token |
|---|---|---|---|
| shisa-v1 | augmxnt/shisa-7b-v1 | 120073 | 2.3074 |
| LLM-jp 3 | llm-jp/llm-jp-3-172b-instruct3 | 99574 | 2.12281 |
| Stockmark 100B | stockmark/stockmark-100b | 60032 | 2.04877 |
| CALM3 | cyberagent/calm3-22b-chat | 65000 | 1.99779 |
| Gemma 7B | google/gemma-7b | 256000 | 1.85274 |
| Gemma 2 | google/gemma-2-9b-it | 256000 | 1.85274 |
| Gemma 3 | google/gemma-3-27b-it | 262144 | 1.78502 |
| Orion 14B | OrionStarAI/Orion-14B-Base | 84608 | 1.7117 |
| Llama 4 | unsloth/Llama-4-Scout-17B-16E-Instruct-unsloth | 200000 | 1.64369 |
| Cohere Command-R+ | CohereForAI/c4ai-command-r-plus | 255000 | 1.64356 |
| Nemotron 4 | failspy/Nemotron-4-340B-Instruct-SafeTensors | 256000 | 1.62567 |
| LLM-jp v2.0 | llm-jp/llm-jp-13b-v2.0 | 96867 | 1.62129 |
| RakutenAI-7B | Rakuten/RakutenAI-7B | 48000 | 1.6117 |
| DeepSeek V3 | deepseek-ai/DeepSeek-V3-0324 | 128000 | 1.48097 |
| Qwen 2 | Qwen/Qwen2-7B-Instruct | 151643 | 1.47894 |
| Qwen 3 | Qwen/Qwen3-30B-A3B | 151643 | 1.47894 |
| GLM-4 | THUDM/glm-4-9b-chat | 151329 | 1.45818 |
| Llama 3 | NousResearch/Meta-Llama-3-8B | 128000 | 1.4549 |
| GPT-4o | Xenova/gpt-4o | 200000 | 1.3897 |
| Mistral Nemo (Tekken) | mistralai/Mistral-Nemo-Instruct-2407 | 131072 | 1.3366 |
| GPT-4 | Xenova/gpt-4 | 100263 | 0.998234 |
| DBRX | databricks/dbrx-instruct | 100278 | 0.998234 |
| Microsoft Phi 4 | microsoft/Phi-4 | 100352 | 0.998234 |
| Mistral Large 2 | mistralai/Mistral-Large-Instruct-2407 | 32768 | 0.946412 |
| Mistral v0.3 | mistralai/Mistral-7B-v0.3 | 32768 | 0.945937 |
| Swallow MX NVE | tokyotech-llm/Swallow-MX-8x7b-NVE-v0.1 | 32000 | 0.945937 |
| Jamba | ai21labs/Jamba-v0.1 | 65536 | 0.92805 |
| Microsoft Phi 3 | microsoft/Phi-3-medium-128k-instruct | 32000 | 0.883761 |
| Yi 34B 200K | 01-ai/Yi-34B-200K | 64000 | 0.832232 |
| Yi 1.5 | 01-ai/Yi-1.5-34B-Chat | 64000 | 0.831878 |
| Falcon 2 | tiiuae/falcon-11B | 65024 | 0.756638 |