v1.2.0 — BM25 Compression

What's new

Terminal output now uses BM25 query-focused compression instead of blind head truncation. Agent context cost drops from ~17,500 tokens to ~1,500 tokens per search — a 10-13x reduction.

BM25 Pipeline

regex match → sentence split → tokenize (bigrams + stemming + stop words)
            → BM25 score → select top chunks → adaptive budget → dedup

Feature	Effect
Sentence-level splitting	3-5x more facts per budget
Suffix stemming	deploy matches deployed, deploying, deployment
Stop word filtering	60+ words removed from scoring
Bigram scoring	pip install boosts chunks with adjacent words
Adaptive budget	3 matches = 2000 chars each, 50 = 300 each
Content dedup	~35% fewer duplicate results
Near-miss hints	Empty results suggest simpler queries

Other

• --json output preserves full uncompressed text (unchanged)
• 10 new tests for BM25 pipeline
• Fixed splitChunks fallback for single-newline text
• Full docs in README

Install

go install github.com/evoleinik/claude-grep@v1.2.0