tok

Cut LLM token costs by 60–90%. A Go library for prompt compression, output filtering, token estimation, and secrets scanning — built for AI coding agents.

---

What tok Is

tok is a library, not a CLI. It exposes token-efficiency primitives as a clean Go API:

• Prompt compression — tok.PromptCompress / tok.Compress shrink verbose prompts 20–70% (six modes).
• Output filtering — a 31-layer pipeline (entropy pruning, perplexity filtering, AST-aware compression, H2O heavy-hitter, …) that strips noise from command output before it re-enters an LLM context.
• Token estimation — tok.EstimateTokens* and tok.EstimateCost give model-aware token counts and pricing.
• Secrets scanning — tok.SecretDetector and tok.IsSensitiveFilename catch credentials before they leak into prompts.
• Usage limits & tracking — thread-safe token/cost windows (tok.NewUsageTracker) plus persistent SQLite-backed gain tracking (tok.NewTracker).
• Runtime graph projection — privacy-safe compression, usage, budget, and redaction summaries (runtimegraph.Build).

It is consumed directly as a Go module, and it powers the tok commands inside Hawk (hawk tok ...), which imports it as a library.

Ecosystem Boundaries

tok is a Hawk support engine. Keep the dependency edge one-way:

• use eagle for any cross-repo shared contracts
• do not import hawk/internal/*
• do not import removed legacy path hawk/shared/types; use eagle/types
• do not import other engines (eyrie, yaad, trace, sight, inspect) — engines are peers, not dependencies

---

Install

go get github.com/GrayCodeAI/shrike

import "github.com/GrayCodeAI/shrike"

tok ships no standalone tok CLI binary. Its CLI surface is exposed through Hawk, which embeds this library: hawk tok compress, hawk tok estimate, and hawk tok scan.

---

Quick Start

Compress a prompt

out, _ := tok.PromptCompress("Please implement authentication", tok.IntensityUltra)
// → "Implement auth."

Filter command output

out, _ := tok.Compress(verboseOutput, tok.Aggressive)
// 200 lines → a few lines: pass/fail + failures

Estimate tokens & cost

n    := tok.EstimateTokensForModel(text, "gpt-4o")
cost := tok.EstimateCost(text, "gpt-4o")

Scan for secrets

d := tok.NewSecretDetector()
findings := d.DetectSecrets(text)
redacted := d.RedactSecrets(text)

Track usage and enforce a budget

usage := tok.NewUsageTracker()
usage.SetLimits(tok.UsageLimits{
    HourlyTokens:  100_000,
    SessionTokens: 500_000,
    CostUSD:       10,
})

if allowed, _ := usage.CanProceed(); allowed {
    usage.Record(promptTokens+completionTokens, requestCostUSD, provider, model)
}

allowed, reason := usage.CanProceed()
snapshot := usage.GetUsage()

SetLimits and GetLimits are safe to use while requests are being recorded. Callers own request deduplication and decide whether a failed decision blocks the current request or the next one.

---

Library API Highlights

• tok.PromptCompress(text, intensity) — prompt compression (Lite / Full / Ultra). ~150 phrase substitutions, drop-lists for articles / filler / pleasantries, and auto-clarity (security/destructive segments pass through verbatim). intensity is monotonic: len(ultra) <= len(full) <= len(lite).
• tok.Compress(text, opts...) — the full output pipeline with options: WithCustomFilters, WithCodeAware(lang), WithPerplexityGuided(scorer, ratio), profile options, and more.
• tok.IsSensitiveFilename(path) — 3-layer filename detection (exact basename, sensitive directory, name token). Companion to the content-based SecretDetector. Catches .env, id_rsa, ~/.ssh/..., test_credentials.json, etc.
• tok.SmartTruncate(content, maxLines, lang) — code truncation that preserves function signatures and always reports the exact drop count (kept + dropped == total).
• tok.ExtractJSON / ExtractJSONArray / ExtractAllJSON — brace-balanced JSON extraction from LLM output with surrounding prose, markdown fences, and unterminated objects.
• tok.NewTracker(ctx) — persistent gain tracker (SQLite + WAL, 90-day retention, pure-Go via modernc.org/sqlite). Aggregate, Recent, Prune queries.
• tok.NewUsageTracker() — in-memory hourly, daily, session, and cost accounting with atomic limit snapshots and threshold decisions.
• tok.EstimateTokensFast / WithEncoding / ForModel — model-aware token estimation.
• filter.CompressWithRetry — validate-fix-retry loop: caller supplies a Validator and AdjustFunc; the loop escalates mode/intensity and retries up to N times.
• filter.NewTOMLFilter / LoadTOMLFilterFile — full 8-stage TOML pipeline as a pluggable Filter.

Full reference: pkg.go.dev/github.com/GrayCodeAI/shrike.

---

Compression Modes

Mode	Style	Savings
lite	Drop filler, keep grammar	~20%
full	Drop articles, fragments OK	~40% (default)
ultra	Telegraphic, abbreviations	~60%
wenyan-lite	Classical Chinese light	~30%
wenyan	Classical Chinese standard	~50%
wenyan-ultra	Classical Chinese max	~70%

Output Filtering (31-Layer Pipeline)

Research-backed algorithms: entropy pruning, perplexity filtering, AST-aware compression, H2O heavy-hitter, attention sink preservation, semantic chunking, and 25+ more.

Custom Filter DSL

Define regex find/replace rules in a TOML file and plug them into the pipeline. Opt-in — no rules, no change.

# filters.toml
[[rule]]
name        = "collapse-uuids"
pattern     = "[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"
replacement = "<uuid>"
priority    = 10

rules, _ := tok.LoadFilterRules("filters.toml")
out, _ := tok.Compress(text, tok.WithCustomFilters(rules))

Team / Shared Compression Profiles

Bundle mode + tier + budget into a named, versioned TOML profile teams can share. Built-ins: default, aggressive, code-safe.

p, _ := tok.LoadProfile(".tok/profiles/code-safe.toml")
out, _ := tok.Compress(text, p.Options()...)

Code-Aware (Symbol-Preserving) Compression

tok.WithCodeAware(lang) marks input as source code and guards function/type/export signatures so compression can never strip them. Defaults to a dependency-light regex symbol provider; swap in an LSP-backed one via WithSymbolProvider.

Perplexity-Guided Token Dropping

tok.WithPerplexityGuided(scorer, ratio) (LLMLingua-style) drops the lowest-importance tokens first. Default HeuristicPerplexityScorer is zero-dependency; plug in your own PerplexityScorer, or tok's experimental OllamaScorer when built with -tags experimental_ollama. Opt-in.

---

Benchmarks

Measured on this repo via benchmarks/run.sh (raw vs tok.Compress(..., tok.Aggressive)):

fixture	raw bytes	raw tokens	tok bytes	tok tokens	saved
git log	2,873	718	298	74	89 %
git diff	385,051	96,262	1,117	279	99 %
ls -la	66,341	16,585	148	37	99 %
find .go	19,145	4,786	147	36	99 %

Profile the hot paths with ./scripts/profile.sh [compress|tokens|filter|secrets|all].

---

Architecture

tok
├── tok.go, *.go         Public library API (top-level package)
├── runtimegraph/        Privacy-safe operations/policy/quality projection
├── internal/
│   ├── compress/        Input compression engine (6 modes)
│   ├── filter/          Output pipeline (31 layers)
│   ├── secrets/         Secret detection + redaction
│   ├── tracking/        SQLite token-usage database
│   ├── fastops/         Hot-path primitives (entropy, etc.)
│   └── core/            Token estimation & cost
├── benchmarks/          Token-savings benchmarks
└── evals/               Eval harness

Pure-Go, zero CGO, no runtime dependencies.

runtimegraph.Build projects completed compression statistics, usage snapshots, budget decisions, and redaction summaries into tok.graph/v1. Input text, model names, budget reasons, secret types, and secret values are digest-only. Tok remains the source of truth for compression, limits, and redaction; graph consumers receive immutable summaries.

---

Contributing

git clone https://github.com/GrayCodeAI/shrike.git && cd tok
make test && make lint
./scripts/build.sh        # verifies the library compiles (go build ./... + go vet)

See CONTRIBUTING.md.

---

License

MIT