v11.1.0 — Token Economics Refinement (Phase 5.5.1)

Added

• Real BPE tokenizer: tiktoken cl100k_base integration via src/utils/token-estimator.ts. Replaces chars/3.5 heuristic for token cost estimation. Heuristic preserved as fallback (zero-downtime). Empirical verification: 72-file benchmark 276,516 raw → 60,732 compressed = 78% savings sostained con real BPE tokenizer.
• Defense-in-depth conditional compression: compressor-level bypass for files <100 lines OR <1024 bytes (src/compressor-foveal.ts:102-123). Complements existing handler-level bypass (src/handlers/code/read.ts:84). Zero overhead instances confirmed empirically across all file-size buckets (72 files tested).
• bySize SessionReport field: per-file-size tracking buckets (<100L, 100-299L, 300-999L, ≥1000L) in engine.getSessionReport() output.
• Empirical benchmark scripts: scripts/benchmark-token-economics.ts + scripts/simulate-claude-session.ts for ongoing verification.
• Token economics empirical verify documentation: docs/token-economics-empirical-verify.md.

Changed

• Template adelgazamiento:
  • templates/CLAUDE.md 3,965 → 1,670 bytes (495 BPE tokens, ~58% reduction)
  • templates/AGENTS.md 3,885 → 1,253 bytes (372 BPE tokens, ~68% reduction)
  • Combined 867 BPE tokens (~61% reduction, ~8% over Apr 2026 target of 800 combined tokens, honest disclosure).
• Token cost estimation accuracy: real BPE replaces direccional-but-imprecise chars/3.5 heuristic. Heuristic validated within ~10% magnitude accuracy of real BPE empírico.

Deferred

• Phase 5.5.2: Hybrid runtime integration → shipped in v11.2.0.
• Phase 5.5.3: Comparative head-to-head benchmarks (NREKI vs Corsa, Zilliz Claude Context, codebase-memory-mcp, GitNexus).