feat(release): cache-aware compaction, LLM extract, and production hardening#14
Merged
Conversation
…rdening The next-release cut of context-guru. Ships the cache-aware compaction redesign and the LLM relevance-trimmer, then hardens the whole path for production based on a multi-agent review (security, generalizability, bugs, dead code, tests, docs). Presets & defaults - New default presets shipping the SWE-bench-winning config: `codesmart` (cache-aware `[format, dedup, failed_run, cmdfilter, extract_llm, extract, cacheinject]`, extract_llm routed to CHEAP_MODEL and gated so most turns make no model call) and `codesafe` (deterministic-only, zero model calls by policy). Presets can now carry tuned per-component config (presetConfigs); the proxy default is `codesmart`. Without a cheap model, extract_llm safely no-ops. - Removed the `phi_evict` component (superseded by `mask`). Cache stability (the release's core invariant) - `mask` now freezes its decision and replays it byte-identically every turn, so a masked output that slides into the cached prefix no longer reverts to full and churns the provider KV cache (previously only `failed_run` did this). - Every offloader now honors kept-verbatim + existing-marker skips uniformly, so content the agent just expanded is never re-compacted into an expand-loop. - GET /expand is scoped to the owning session (per-(session,key) ownership record), closing a cross-session/tenant disclosure of stashed originals. Correctness fixes - stripTerminalNoise no longer blanks CRLF-terminated tool output (the trailing CR is the line separator, not an in-line progress redraw). - AggregateSSE reconstructs extended-thinking blocks (thinking_delta + signature_delta), so the expand continuation builds a valid assistant turn. - UTF-8-safe truncation in headPeek and the dump trace (no split runes). - resolveCacheAware detects a real cache_control/cachePoint breakpoint structurally instead of substring-scanning the whole body (no false positives from tool text). Reliability / security hardening - Top-level fail-open recover around apply.BodyFull, the proxy pre-forward path, and emitter callbacks — a panic outside a component now forwards the original request. - Request bodies are bounded (http.MaxBytesReader) to prevent memory-exhaustion DoS. - Starlark sandbox gains an input-size cap and a heap-growth watchdog (the step/time limits can't preempt a native op); skeleton's tree walk gains a depth bound (an unbounded recursion is an uncatchable stack-overflow crash). - modelinfo resolves the model window via a single-flighted, non-blocking background fetch with negative caching (was a synchronous hot-path GitHub GET, refetch-on-fail). - Debug/dump files created 0600 (they persist raw payloads). Cleanup - Removed dead code (internal/extract/cache.go, RunExtractionBatch/buildBatchPrompt, lastUserText, overlap, writeBuffered, modelinfo.Default + unused `once`) and internal-only docs (docs/superpowers/). Retired stale phi_evict references and visualizations. Tests & docs - New regression tests: mask byte-stability across turns, kept-verbatim skip, CRLF preservation + interior-CR redraw, headPeek UTF-8, AggregateSSE thinking-block preservation, modelinfo single-flight/non-blocking, /expand session scoping, and rich-preset config resolution. Full suite + race detector green; mkdocs --strict clean. - README + docs overhaul with the three-way SWE-bench results (cheapest arm, beats headroom on cost/cache/steps/reward), per-component live-captured before/after examples, and corrected build/quickstart. Assisted-By: Claude (Anthropic) Signed-off-by: Osher Elhadad <osher.elhadad@live.biu.ac.il>
The build-test job failed on the gofmt check: three files edited in the release commit (p4_test.go, common.go, proxy.go) were left unformatted after the phi_evict removal and inline edits. gofmt -w applied. Adds a `make cover` target (race + cross-package coverage → coverage.out, prints the statement-weighted total; currently 74.0%) so coverage can be surfaced in CI. Assisted-By: Claude (Anthropic) Signed-off-by: Osher Elhadad <osher.elhadad@live.biu.ac.il>
The Test step now runs `make cover` and writes the statement-weighted total coverage to the run's job summary (visible on every PR), plus per-package coverage in the log. Assisted-By: Claude (Anthropic) Signed-off-by: Osher Elhadad <osher.elhadad@live.biu.ac.il>
github-project-automation
Bot
moved this from New/ToDo
to Done
in Rossoctl Issue Prioritization
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Next release: cache-aware compaction, LLM extract, and production hardening
This is the release cut. It ships the cache-aware compaction redesign and the cheap-model
relevance-trimmer, then hardens the whole request path for production based on a
multi-agent review (security, generalizability, correctness, dead-code, tests, docs).
Highlights
Presets & defaults
codesmart(the default — cache-aware[format, dedup, failed_run, cmdfilter, extract_llm, extract, cacheinject]) andcodesafe(deterministic-only, zero model calls by policy). Presets can now carrytuned per-component config.
extract_llmengages only when a cheap model is configured(
CHEAP_MODEL*) and safely no-ops to deterministic otherwise.phi_evictcomponent (superseded bymask).Cache stability (core invariant)
maskfreezes and replays its decision byte-identically every turn, so a masked outputthat slides into the cached prefix no longer reverts to full and churns the KV cache.
just expanded is never re-compacted into an expand loop.
GET /expandis scoped to the owning session, closing a cross-session/tenant disclosureof stashed originals.
Correctness
stripTerminalNoiseno longer blanks CRLF-terminated tool output.AggregateSSEpreserves extended-thinking blocks (+signature) so the expandcontinuation builds a valid assistant turn.
resolveCacheAwaredetects a realbreakpoint structurally instead of substring-scanning the body.
Reliability / security
outside a component forwards the original request instead of 500-ing.
watchdog; skeleton tree-walk depth bound (uncatchable stack-overflow crash); modelinfo
window resolution is now a single-flighted, non-blocking background fetch with negative
caching; debug/dump files created
0600.Cleanup, tests & docs
internal/extract/cache.go,RunExtractionBatch/buildBatchPrompt,lastUserText,overlap,writeBuffered,modelinfo.Default) and internal-only docs(
docs/superpowers/).-racegreen;mkdocs --strictclean.on cost/cache/steps/reward) and live-captured per-component before/after examples.
Verification
CGO_ENABLED=1 go build -tags cg_skeleton ./...andgo test -tags cg_skeleton ./...— green.go test -raceon the concurrency-touched packages — green.codesmartdefault and round-trips a real Anthropic request(3605 → 93 tokens on a repeated-output transcript);
mkdocs build --strictclean.