Token Optimizer Adversarial Test Suite

An adversarial test suite exposing dangerous failure modes in AI token optimizer tools — specifically RTK (29K stars) and LeanCTX (673 stars).

What This Is

Token optimizers intercept shell commands from AI coding agents (Claude Code, Cursor, etc.) and compress their output before the AI sees it, claiming 60–90% token reduction. This repo contains an adversarial test harness that checks whether that compression is safe — or whether it silently replaces correct information with wrong information.

Result: RTK has 24 confirmed dangerous failure modes. LeanCTX has fewer, but introduces its own critical class of failure.

Read the Findings

• Technical findings — Full adversarial analysis: 24 RTK failure modes, 16-test head-to-head comparison against LeanCTX, statistical analysis, production impact scenarios
• LeanCTX dangerous commands — Which commands to avoid with LeanCTX, what goes wrong, and what to run instead

Quick Start

# Install RTK
bash install-rtk.sh

# Run RTK-only adversarial tests (11 categories, 60+ scenarios)
bash tests/run-all.sh

# Run RTK vs LeanCTX comparison (requires lean-ctx)
# Install LeanCTX:
curl -sL https://github.com/yvgude/lean-ctx/releases/download/v3.2.5/lean-ctx-x86_64-unknown-linux-musl.tar.gz \
  | tar xz && mv lean-ctx ~/.local/bin/

bash tests/compare/run.sh

Test Categories

Category	Tests	Key Finding
Git operations	15	DETACHED HEAD warning stripped, stash ambiguity
Log analysis	8	[CRITICAL] severity silently dropped
Test runners	12	Skipped/xfail tests invisible
File operations	6	.env hidden from ls, wc pipe returns zero
Docker	8	Crashed containers absent, health status stripped
Linters	6	File:line:col references removed
Package managers	4	pip shows 2 of 314 packages, pnpm parser failure
Comparative (RTK vs LeanCTX)	16	RTK 2/16 safe, LeanCTX 9/16 safe

Tested Versions

• RTK: v0.37.1
• LeanCTX: v3.2.5
• Platform: Linux x86_64 (WSL2 / Ubuntu)
• Date: April 2026

Requirements

git, bash, python3, pytest, ruff, node, docker

---

RTK source: https://github.com/rtk-ai/rtk
LeanCTX source: https://github.com/yvgude/lean-ctx