Token Engine

Ultimate LLM Token Optimization Engine
Fast · accurate · model-agnostic · Cursor-native

token-engine · cursor-kit · Cursor env

---

Compress agent context while preserving errors, stack traces, and task-critical code.
Built for Cursor + MCP

Cursor Integration

Ponytail + Caveman + Token Engine + codebase-memory — see docs/CURSOR.md and docs/CURSOR-ENV.md.

Per-repo bootstrap of the full stack: /setup-project in cursor-kit.

pip install -e ".[cursor,dev]"
token-engine cursor-setup
token-engine benchmark --check-baseline

Install

macOS / Linux

git clone https://github.com/Bossmann007/token-engine.git
cd token-engine
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[cursor,dev]"

Optional symlink used by this machine’s Cursor env:

ln -sfn "$PWD" ~/token-engine

Windows

git clone https://github.com/Bossmann007/token-engine.git
cd token-engine
.\scripts\install-windows.ps1 -SkipClone

Quick Start

token-engine optimize benchmarks/fixtures/app_log.txt
token-engine optimize-context benchmarks/fixtures/agent_context.json --task "auth bug"
token-engine benchmark --check-baseline
token-engine serve   # REST on :8741

Example benchmark output:

╔══════════════════════════════════════════════════════════╗
║                      Token Engine                        ║
║                     Benchmark Report                     ║
╚══════════════════════════════════════════════════════════╝
Fixture                 Original     Saved   Ratio  Bar
────────────────────────────────────────────────────────
app_log                    1,234       890   72.1%  ███████████░░░░░

Features

Module	What it does
Analyzer	Token tiers, redundancy, relevance scoring
Optimizer	BM25 rank + knapsack budget
Compressors	JSON, logs, code, diffs, pytest/git/npm output
MCP	caveman_compress, session compress, schema compact
Harness	POST /optimize-context before each LLM turn

Fail-closed: never replaces content unless provably smaller and quality checks pass.

Documentation

Doc	Topic
ARCHITECTURE.md	System design
ALGORITHMS.md	Techniques + rejected approaches
API.md	REST endpoints
CURSOR-ENV.md	Ultimate Cursor Environment
BENCHMARKS.md	Baseline gates
cursor-kit	Templates, hooks, /setup-project

License

MIT