tōks ("tokes" as in "tokens") counts tokens for files and directories across LLM providers. See how much of a model's context window your content will consume before you spend the tokens. This is particularly useful for context engineering & token budgeting when building software using AI agents.
Install as a tool with uv:
uv tool install toksOR, if you're still a "pipper"
pip install toksThe ct ("count tokens") shorthand alias is also available — ct src/ is equivalent to toks src/.
toks setupThe interactive wizard walks you through selecting providers, entering API keys, and choosing models.
# Single file
toks src/main.py --for claude
# Entire directory with glob filter
toks src/ --for claude --glob "*.py"
# Quiet mode (just the number)
toks README.md --for claude -q
# Pipe from stdin
cat file.py | toks - --for claudeProvider: claude (claude-opus-4-6)
File Tokens Agent Web API
toks/ [17,911] 1.8% 9.0% 1.8%
├── providers/ [4,174] 0.4% 2.1% 0.4%
│ ├── __init__.py 324 <0.1% 0.2% <0.1%
│ ├── base.py 474 <0.1% 0.2% <0.1%
│ ├── claude.py 600 <0.1% 0.3% <0.1%
│ ├── gemini.py 643 <0.1% 0.3% <0.1%
│ ├── grok.py 1,315 0.1% 0.7% 0.1%
│ └── openai.py 818 <0.1% 0.4% <0.1%
├── __init__.py 33 <0.1% <0.1% <0.1%
├── __main__.py 28 <0.1% <0.1% <0.1%
├── cli.py 3,188 0.3% 1.6% 0.3%
├── config.py 1,338 0.1% 0.7% 0.1%
├── output.py 2,260 0.2% 1.1% 0.2%
├── registry.py 978 <0.1% 0.5% <0.1%
├── runner.py 1,146 0.1% 0.6% 0.1%
├── scanner.py 1,318 0.1% 0.7% 0.1%
└── setup.py 3,448 0.3% 1.7% 0.3%
Total: 17,911 tokens
Agent (1M): 1.8%
Web (200K): 9.0%
API (1M): 1.8%
Three percentage columns show context window usage for each provider's interfaces: coding agent, web chat, and API. The Agent and Web columns populate when you configure your plan and coding agent model via toks setup.
| Provider | Text | Images | Office Docs | |
|---|---|---|---|---|
| Claude (Anthropic) | Yes | JPEG, PNG, GIF, WebP | Yes | No |
| OpenAI | Yes | JPEG, PNG, GIF, WebP | Yes | DOCX, XLSX, PPTX |
| Gemini (Google) | Yes | JPEG, PNG, GIF, WebP + more | Yes | No |
| Grok (xAI) | Yes | JPEG, PNG (local calc) | No | No |
All token counting is done via each provider's REST API. No local tokenizer libraries are needed.
When scanning directories, toks respects .gitignore rules by default — files matching gitignore patterns are skipped. Nested .gitignore files are also respected. Use --no-gitignore to include all files regardless. Binary files are excluded by default (--include-binary to override), and files over 50MB are skipped (--max-size to adjust).
Config files live in ~/.config/toks/:
config.toml-- provider settings, default model, plan tier.env-- API keys
Re-run toks setup at any time to update your configuration.
toks <target> [options] # Count tokens (default command)
toks setup # Interactive configuration wizard
toks models --refresh # List known models / refresh registryct can be used anywhere in place of toks.
| Flag | Description |
|---|---|
--for <provider> |
Provider to use (claude, openai, gemini, grok) |
--model <model> |
Specific model (provider inferred from registry) |
--glob <pattern> |
Filter files by glob pattern |
--max-size <size> |
Exclude files larger than size (default: 50MB) |
--depth <n> |
Limit directory recursion depth (0 = target dir only) |
-q / --quiet |
Output only the total token count |
--summary |
Totals without the tree |
--no-progress |
Suppress progress bar |
--concurrency <n> |
Concurrent API requests (default: 10) |
--retries <n> |
Retry count for transient errors (default: 3) |
- Technical Documentation -- architecture, module details, tutorials
- Feature Specification -- complete design specification
Releases are cut from the project root with release X.Y.Z (or release for an interactive version prompt). The tool bumps pyproject.toml, commits, tags, pushes -- which triggers the PyPI publish workflow via GitHub Actions Trusted Publishing. Release notes auto-generate from commit messages between tags, so commits should read well as changelog entries.