Add cache breakpoint markers and pyramid budget#30
Merged
Conversation
* feat: update root and home view * style: fix gofmt and goimports across codebase * chore: go mod tidy
- Wire CoordinatorPool into fallback.go applyPipeline to eliminate 20+ filter allocations per command - Replace all unsynchronized PipelineStats.LayerStats map writes with AddLayerStatSafe across pipeline_early_exit, core_filters, content_route_strategy, six_layer_pipeline, pipeline_process - Add layer_names.go constants and replace magic strings in buildLayers and all stat recorders to fix index coupling - Remove unbounded goroutine spawn from StatusLine.Publish; render synchronously instead - Cache tee config in GetTeeDir to avoid repeated disk reads - Reset processedLayers in coordinator pool reset() - Add PipelineCoordinator.UpdateConfig for safe pool reuse
Fixes: - isPathSafe EvalSymlinks fallback for non-existent paths - looksLikeJSON array detection + SQL injection prevention - LayerCache race window (single Lock), parallel result mismatch - Wenyan regex pre-compilation, GetTokenBudget env cache - Security stubs: RBAC deny-by-default, RateLimiter mutex - 100MiB output cap, pipe leak fixes, error shadowing - LayerIdx constants, LayerBitset uint64, CI lint re-enable Tests: - 45+ new test files across 20 packages - cache 23%→46%, compression 35%→82%, security 39%→71% - simd 12%→73%, ml 30%→100%, build 5%→40% - container 3%→18%, linter 10%→32% Verification: build, vet, fmt, test, race all clean
- Delete internal/app/ (superseded by commands/ structure) - Delete pkg/cli/cli.go (unused) - Fix module path references in Makefile, docs, config - Update .golangci.yml local-prefixes - Security scanner: add O(1) unicode lookup, sort findings - Tee: fix bubble sort → sort.Slice, add config caching - Tracking: truncate output, singleton tracker
.gomodcache/ is in .gitignore but was accidentally committed. Remove 10,660 tracked files from index.
…and migration versioning - parallel_executor: copy best output/tokens before pool.Put to fix use-after-return data race - tee: tighten dir/file permissions 0755/0644 → 0700/0600 (output may contain credentials) - security: PluginSandbox.Execute returns an explicit error instead of silently passing through; SecurityScanner documented as no-op stub - runner: wire LimitedWriter to cmd.Stdout/Stderr so 100 MiB cap is enforced during streaming, not after full allocation - tracking: replace flat InitSchema loop with RunMigrations backed by PRAGMA user_version; each migration stamped individually for crash-safe recovery; legacy un-versioned DBs handled via idempotent IF NOT EXISTS DDL - filter: remove unused SafePipelineStats (dead code) - golangci.yml: align go version to 1.25 to match go.mod Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Bugs fixed: - view.refresh action was a no-op (RequestRefresh return discarded); propagate cmd through actionResultMsg.Cmd so it actually reloads - refreshTickCmd never bootstrapped in Init(); periodic refresh now starts - filterOverlayMsg silently dropped for all sections; add SetFilter(q) delegate to every table section so `f` filter overlay works - logs `c` key bypassed confirm modal; now dispatches actionRequestMsg - compact tabs breadcrumb had two empty string placeholders; now shows prev/next section Short() names Visual improvements: - Toast, streak calendar, confirm buttons, PanelTitle all now read from the theme struct instead of hard-coded hex — all four themes (dark, light, high-contrast, colorblind) now render consistently - detectUTF8 defaults to true on darwin/linux/bsd so Braille charts work without LANG being set UX: - `s` key sorts by the accent (Savings) column in any table section; cycles asc → desc → off via new Table.SortAccent() - Sort key documented in FullHelp and KeyMap Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix tracker race: add doneCh for clean shutdown (prevent panic on send after close) - Fix global printer race: use atomic.Pointer[Printer] - Fix compressor: pre-compile regexes with consistent case-insensitive matching - Fix cache: use RWMutex for read-heavy paths - Fix SQL injection: add column name allowlist + sqliteQuoteIdentifier - Harden path validation: reject absolute paths, traversal, null bytes - Fix UTF-8 width: use runewidth.StringWidth for emoji-aware padding - Fix double token counting: compute savings against original input - Remove dead code: empty config parsing in hooks - Add language detection sampling: limit to first 4KB for performance - Remove standalone distribution (brew, aur, release workflows) - Update README: hawk is the exclusive distribution channel
…s across codebase - Fix race conditions in QueryCache (atomic counters), LimitedWriter (mutex), CircuitBreaker (TOCTOU), and metrics (atomic lastReset) - Add goroutine lifecycle management (done channels, WaitGroups) in cache, ttlcache, telemetry, and status_line - Harden security: ANSI escape filtering, path traversal prevention, file permissions (0700/0600), full secret masking, atomic file writes - Fix Process() signature mismatches across filter pipeline callers - Fix rate limiter busy-wait, regex/sketch cache unbounded growth, parallel semaphore acquisition, and division-by-zero in compare_cmd - Convert cache subcommands from os.Exit to RunE, fix CLI flag wiring - Remove dead tiktoken_estimator.go, fix isPathSafe symlink resolution on macOS - Fix config LoadFromFile to use toml.DecodeFile matching encoder output
…tion Remove standalone CLI, MCP server, TUI, telemetry, and all command infrastructure. tok is now a pure Go library imported as: import "github.com/GrayCodeAI/tok" compressed, stats := tok.Compress(text, tok.Aggressive) c := tok.NewCompressor(tok.Adaptive) Kept: filter pipeline (20 layers), core token estimation, cache, config, simd, utils. Removed 30+ packages that only served the CLI/external agents.
…boundaries with token counting
From Compresr-ai/Context-Gateway pattern: - StreamCompressor maintains always-ready compressed output - Background goroutine re-compresses when token threshold exceeded - Snapshot() never blocks — returns latest compressed version - Thread-safe (RWMutex), race-detector clean - 9 tests including concurrent access from 60 goroutines
- CacheBreakpointFilter: annotates output with [CACHE_BREAK] markers for Anthropic/Gemini prompt caching (90% cost savings) - PyramidBudget: layer-adaptive compression budgets (early layers permissive, later aggressive)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cache breakpoint filter + pyramid budget distribution for tok compression pipeline