tokencount

Fast, zero-dependency token and cost estimator for GPT, Claude, Gemini and other LLM prompts. No wasm binary, no downloaded vocab files, install size under 10kb — just a regex-based estimator that lands within a few percent of real tokenizer output for typical English/code prompts.

Works as a CLI (npx tokencount) and as a library you can require in your own code.

Why

Full tokenizers (tiktoken, gpt-tokenizer, @anthropic-ai/tokenizer) ship large wasm binaries or vocabulary tables just to answer "roughly how many tokens is this / what will it cost me?". tokencount skips the vocabulary entirely and gives you a fast approximate answer instead — good enough for budgeting, CI guardrails, and quick sanity checks before you hit an API.

Install

npm install -g @cryptoteep/tokencount
# or just run it once, no install:
npx @cryptoteep/tokencount "some text"

CLI usage

tokencount "Hello, world!"
# 4

echo "some text" | tokencount -m claude-3-5-sonnet

tokencount -f prompt.txt --cost -m gpt-4o --output 500
# {
#   "model": "gpt-4o",
#   "inputTokens": 128,
#   "outputTokens": 500,
#   "inputCost": 0.00032,
#   "outputCost": 0.005,
#   "totalCost": 0.00532
# }

tokencount --list-models

Use it as a pre-commit or CI check to catch prompts that quietly grew too large:

tokencount -f prompts/system.txt | awk '{ if ($1 > 4000) exit 1 }'

Library usage

const { estimateTokens, estimateCost, listModels } = require('@cryptoteep/tokencount');

estimateTokens('Hello, world!'); // 4
estimateTokens('Hello, world!', 'claude-3-5-sonnet'); // model-aware estimate

estimateCost('Explain quantum computing.', 'gpt-4o-mini', { outputTokens: 300 });
// { model, inputTokens, outputTokens, inputCost, outputCost, totalCost }

listModels(); // ['gpt-4o', 'gpt-4o-mini', 'claude-3-5-sonnet', ...]

TypeScript types are bundled (index.d.ts), no @types package needed.

Accuracy

tokencount uses the same GPT-2-style pre-tokenization regex real BPE tokenizers start from (splitting on words, numbers, punctuation and whitespace runs), then applies a small per-model-family correction factor. It will not match tiktoken/gpt-tokenizer exactly — real BPE merges push counts a bit lower for common words — but it's typically within a few percent for natural-language and code prompts, at a fraction of the install size and with no vocabulary file to keep in sync.

If you need exact counts, use the vendor tokenizer. If you need a fast estimate for budgeting or CI, this is for that.

Pricing data

The bundled price table is a small, illustrative snapshot and will go stale — LLM pricing changes often. For an up-to-date catalog across 100+ models, see the companion package llm-prices (npx llm-prices).

License

MIT © Cryptoteep