Your coding agent burns thousands of tokens reading files it only needed a few functions from. This is a local MCP server that gives it a compressed version instead — the parts that match what it's actually looking for, kept byte-for-byte, with everything else recoverable on demand.
No account. No API key. No network. Runs entirely on your machine. MIT.
npm install -g @tokz/compress
tokz-compress setupsetup finds every AI coding tool you have installed and wires this in
correctly — right scope, right spawn form for your OS, right workspace root.
Restart your editor and it's there.
Code this tool shows your agent is a byte-exact substring of the file on disk, with real line numbers. Not a summary. Not a paraphrase. Not "reconstructed".
That's checked programmatically for every span before it's emitted — spans that
fail are dropped and counted, never shown. It matters because an agent editing
against paraphrased code produces an old_string that doesn't match disk, so
the edit fails. Or worse, it silently reasons about code that never existed.
Anything elided comes back via tokz_retrieve, from a local cache, in about a
millisecond.
Verbatim output of compress <file> "verify token signature timing safe expiry"
on an auth module:
eval/golden/auth_service.ts (1,285 → 567 tokens, 44% kept, code/tree-sitter+bm25)
Shown lines are byte-exact. Recover any elided range:
tokz_retrieve(hash="3e69549e", lines="113-122")
1 import { createHmac, timingSafeEqual, randomBytes } from 'node:crypto';
2 import { readFile } from 'node:fs/promises';
3 import type { Redis } from 'ioredis';
4
5 export interface SessionRecord {
6 userId: string;
7 issuedAt: number;
8 expiresAt: number;
9 scopes: string[];
10 }
⋯ 11-18 elided (8)
19 const DEFAULT_TTL = 3600;
20 const REFRESH_WINDOW = 300;
⋯ 21-25 elided (5)
26 export function signSession(session: SessionRecord, config: AuthConfig): string {
⋯ 27-31 elided (5)
32 /**
33 * Verify a token's signature and expiry.
34 *
35 * Signature comparison uses timingSafeEqual so that a wrong token cannot be
36 * discovered byte by byte through response timing.
37 */
38 export function verifyToken(raw: string, config: AuthConfig, now = Date.now()): SessionRecord | null {
39 const parts = raw.split('.');
40 if (parts.length !== 2) return null;
41 const [payload, mac] = parts as [string, string];
42
43 const expected = createHmac('sha256', config.secret).update(payload).digest();
44 const provided = Buffer.from(mac, 'base64url');
45 if (expected.length !== provided.length) return null;
46 if (!timingSafeEqual(expected, provided)) return null;
47
48 let session: SessionRecord;
49 try {
50 session = JSON.parse(Buffer.from(payload, 'base64url').toString('utf8')) as SessionRecord;
51 } catch {
52 return null;
53 }
54
55 const skew = config.clockSkewSeconds * 1000;
56 if (session.expiresAt + skew < now) return null;
57 if (session.issuedAt - skew > now) return null;
58 return session;
59 }
⋯ 60-97 elided (38)
98 export function refreshIfNeeded(raw: string, config: AuthConfig, now = Date.now()): string | null {
⋯ 99-130 elided (32)
131 export class RateLimiter {
⋯ 132-160 elided (29)
Imports kept. The function the query was about, kept whole with its doc comment. Everything else reduced to a signature the agent can see exists — and retrieve if it turns out to matter.
npm install -g @tokz/compress
tokz-compress setupThat's it. setup detects Claude Code, Cursor, Antigravity, Windsurf and
Codex, and configures each one it finds:
Launching as: cmd.exe /c tokz-compress serve
(using the globally installed binary)
ok claude-code plugin installed at user scope (MCP server + hooks)
ok cursor configured
/home/you/.cursor/mcp.json
-- codex not installed
Restart your editor/CLI for the new server to appear.
tokz-compress setup --dry-run— show what it would change, change nothing.tokz-compress setup cursor— just one host (claude-code,cursor,antigravity,windsurf,codex).
It's idempotent, backs up any file before its first write, preserves other servers in your config, and refuses to touch a file that isn't valid JSON.
It also handles three things that are easy to get wrong by hand:
- Hooks on Claude Code. For Claude Code,
setupinstalls the bundled plugin (marketplace +claude plugin install --scope user) rather than a bareclaude mcp add, because the plugin carries thePreToolUse/PostToolUsehooks that make compression automatic. It also clears any stale plain-MCP registration that would shadow or duplicate it. - Windows spawning. Hosts can't launch
npxdirectly on Windows: barenpxfailsENOENT, andnpx.cmdfailsEINVALbecause Node refuses to spawn.cmdwithout a shell (CVE-2024-27980).setupwrites thecmd.exe /c …form that actually works. - Workspace roots. Claude Code gets no path argument (it passes
CLAUDE_PROJECT_DIR); every other host gets an absolute path, because.would resolve against whatever directory the host launched from.
Configuring by hand instead
Prefer the plugin — it bundles the MCP server and the hooks that make compression automatic:
claude plugin marketplace add <path-to-installed-package> # the npm package root
claude plugin install tokz-compress@tokz-compress --scope user(tokz-compress setup runs exactly this for you, with the right path.)
Registering only the MCP server also works, but you get no hooks — the agent has to choose to call the tool:
claude mcp add --scope user tokz-compress -- npx -y @tokz/compress serveNo path argument needed — Claude Code passes CLAUDE_PROJECT_DIR.
.cursor/mcp.json:
{
"mcpServers": {
"tokz-compress": {
"command": "npx",
"args": ["-y", "@tokz/compress", "serve", "/absolute/path/to/project"]
}
}
}codex mcp add tokz-compress -- npx -y @tokz/compress serve /absolute/path/to/projectKnown Codex limitation (not specific to this tool): current Codex CLI builds hide stdio MCP tools from the model's tool list — the tools are registered and callable, but the agent won't discover or reach for them on its own. Tracked upstream in openai/codex#16028. Until that's fixed, expect Codex to keep using its built-in read.
Put this in mcp_config.json — ~/.gemini/config/mcp_config.json covers every
surface, or use ~/.gemini/antigravity-cli/ (CLI only),
~/.gemini/antigravity/ (IDE only), or .agents/mcp_config.json (one project):
{
"mcpServers": {
"tokz-compress": {
"command": "npx",
"args": ["-y", "@tokz/compress", "serve", "/absolute/path/to/project"]
}
}
}npx -y @tokz/compress serve /absolute/path/to/projectPass an absolute path on every host except Claude Code. The server only reads inside the roots you give it, and
.resolves against whatever directory the host happened to spawn it in — which often isn't your project. If reads come back "outside the configured workspace roots", that's why.
On Windows, wrap the command:
"command": "cmd.exe","args": ["/c", "npx", "-y", "@tokz/compress", "serve", "C:\\path\\to\\project"]. Hosts can't spawnnpxdirectly there.
Then check it:
npx -y @tokz/compress doctorIt runs a real compression end to end and prints the workspace roots it resolved.
| tool | what it does |
|---|---|
compress_file(path, query?, rate?) |
read a large file, compressed toward query |
compress_text(text, query?, kind?) |
compress output you already have |
tokz_retrieve(hash, lines?) |
get the exact original back |
tokz_stats() |
what you've saved locally |
Four, deliberately. Every tool definition costs context on every turn, and tools that sound alike get misrouted.
Agents default to their built-in file read, and a tool description alone doesn't reliably change that.
On Claude Code, tokz-compress setup installs the bundled plugin and it
becomes automatic. A PreToolUse hook denies Read on files over ~2,500
tokens with a reason pointing at compress_file, so the agent reroutes
because the read failed — not because it chose to cooperate. A PostToolUse
hook compresses other MCP servers' oversized output (over ~4,000 tokens).
Both fail open: any crash, timeout or malformed input exits 0 with no decision and your read proceeds normally. Saving tokens is never worth breaking your workflow.
On other hosts, add a line to AGENTS.md or your system prompt:
For files over ~400 lines, use
compress_filewith a query instead of reading the whole file.
Produced by bun run eval in the repo and written to eval/results.json.
Re-run it yourself — nothing here is estimated.
Golden set: 24 (file, query, must-appear span) cases across TypeScript, Python and Go, 43 required spans. A span counts as found only if it appears verbatim.
| metric | value |
|---|---|
| recall (spans found verbatim) | 90.7% (39/43) |
| cases at full recall | 22/24 |
| mean compression | 50.9% of tokens kept |
| tokens across the set | 35,805 → 17,704 |
| latency | p50 14ms, p95 29ms |
| verbatim failures | 0 |
109 tests, including end-to-end runs against a real MCP client over JSON-RPC stdio, the CLI as a subprocess, and the hook scripts executed the way Claude Code executes them.
On the compression ratio, honestly: ~50% is lower than the headline numbers
some tools quote, and that's a deliberate trade. The skeleton an agent needs —
imports, class headers, every definition's signature — is itself 30–45% of a
typical 700–1,500 token file. Cut it and the agent can't tell "this was elided"
from "this doesn't exist", which is worse than not compressing at all. Larger
files do better: a 3,810-token file lands at 41–47%. If you want the other
trade, config set skeletonShare 0.25 exposes it.
Code → structural selection. Tree-sitter parses the file, definitions are ranked against your query with BM25, top-ranked ones are emitted in full and the rest as signatures. Imports always kept. LLMLingua-2 is deliberately never used here — it drops individual tokens, so its output isn't valid source.
Logs → clustering. Identical-shaped lines collapse to [×N]. Every error,
warning and stack frame is kept unconditionally.
JSON → shape plus samples. A 400-record array becomes its schema — including which fields are optional and in how many records — plus two examples.
Prose → block selection, or LLMLingua-2 if you opt in.
Under ~2,000 tokens → returned unchanged. And if compression wouldn't actually shrink the input, you get the original. This tool never charges you tokens for the privilege of being compressed.
The ONNX checkpoints are hundreds of megabytes, so downloading one on your first read without asking isn't acceptable. Prose uses extractive block selection by default — no model, no download, works offline.
npx -y @tokz/compress config set llmlingua true
npx -y @tokz/compress config set llmlinguaModel xlm-roberta # larger, better
npx -y @tokz/compress models # repos and cache locationWeights download on first use into ~/.tokz/models/. If loading fails the
prose path falls back to extractive selection and doctor tells you why.
- Path confinement. Both the lexical and the symlink-resolved path must sit
inside a configured root, so a symlink in your repo pointing at
~/.ssh/id_rsais refused rather than followed. - Denylist regardless of root:
.env*,*.pem,*.key,id_rsa*,.git/config,.npmrc,.netrc,.aws/**,.ssh/**,.gnupg/**,.kube/**, and more. Enforced before the file is opened. - Cached originals live in
~/.tokz/compress/objects/for 7 days. Stats append to~/.tokz/compress/stats.jsonl. - Nothing is uploaded, ever. There is no server to upload to.
tokz_stats reports a retrieve rate — retrievals divided by compressions.
High means compression is dropping things your agent actually needed. That's
the quality signal here, not the compression ratio, which is trivially gamed by
throwing away more.
npx -y @tokz/compress setup # wire into every host you have
npx -y @tokz/compress doctor # verify the install works
tokz-compress update # update + refresh the Claude Code plugin
npx -y @tokz/compress compress <file> [query...] # try it on a file
npx -y @tokz/compress retrieve <hash> [lines] # pull an original back
npx -y @tokz/compress stats # local savings
npx -y @tokz/compress config # show/edit settings
npx -y @tokz/compress clear-cacheSettings live in a file, so you don't have to thread environment variables through your host's MCP config:
npx -y @tokz/compress config # show everything + where it came from
npx -y @tokz/compress config set rate 0.2 # compress harder
npx -y @tokz/compress config set llmlingua true
npx -y @tokz/compress config unset rate # back to the defaultconfig prints the effective value of every setting and tags it with the layer
it came from, so there's never a mystery about why a value is what it is:
user config /home/you/.tokz/config.json
project config /home/you/repo/.tokz.json (not present)
sizeGate 2000
rate 0.2 [user]
skeletonShare 0.35
...
Precedence, highest first: environment variable → project .tokz.json →
user ~/.tokz/config.json → default. Add --project to config set to write
.tokz.json in the current directory, which you can commit so a team shares
the same settings.
| setting | default | effect |
|---|---|---|
rate |
0.3 | share of tokens to keep — lower compresses harder |
sizeGate |
2000 | tokens below which input is returned unchanged |
skeletonShare |
0.35 | share of the budget reserved for signatures |
roots |
auto | workspace directories the server may read |
maxFileBytes |
8388608 | refuse files larger than this |
cacheTtlDays |
7 | how long originals stay retrievable |
hookReadThreshold |
2500 | tokens above which Read is redirected |
hookMcpThreshold |
4000 | tokens above which MCP output is compressed |
llmlingua |
false | use the LLMLingua-2 model for prose |
llmlinguaModel |
bert | bert or xlm-roberta |
serverName |
tokz-compress | MCP server name used in hook messages |
Every setting also has a TOKZ_* environment variable that overrides the files
for one-off runs — config lists them. TOKZ_HOME (default ~/.tokz) is
env-only, since it's where the config file itself lives.
tokz-compress updateUpgrades the npm package and refreshes the Claude Code plugin in one step — the plugin's cached copy is version-keyed and never re-read implicitly, so skipping the refresh would leave hooks running the old version.
Interactive CLI commands (stats, doctor, setup, config) also print a
one-line notice on stderr when a newer version exists. That check hits the
npm registry at most once per day, is never made by the MCP server or the
hooks (the compression path stays fully offline), fails silently without a
connection, and is disabled with TOKZ_NO_UPDATE_CHECK=1.
Node ≥ 20. No native modules, no build step on install, no postinstall scripts.
MIT