Skip to content

Releases: lucasmartins-ai/lcc

v1.2.0

Choose a tag to compare

@lucasmartins-ai lucasmartins-ai released this 20 Oct 06:26
c341568

Added

  • lcc measure-session <session.jsonl>: runs rules:lossless, rules:stashed and auto
    on one Claude Code session, offline, and prints counts only (messages, tool calls, tokens
    before/after, texts kept, time). --markdown is ready to paste into an issue, --json is
    lcc-session-report/1. It never writes to the real stash.
  • lcc compact --mode tool-calls --rules-mode <mode> (with --provider rules): the CLI can
    now select stashed, recoverable or lossy; before, only MCP and Python could.
  • lcc compact --mode tool-calls reads a Claude Code session file (JSONL) directly.
  • server.json for the official MCP registry (io.github.lucasmartins-ai/lcc), and a
    local-context-compiler console script (same entry point as lcc), so
    uvx local-context-compiler mcp starts the server.
  • demos/stash.gif and demos/make_session.py (a synthetic 91-message session), and
    docs/EVIDENCE.md: the aggregates of the private studies behind the
    README numbers.
  • Claude Code plugin: idle stash (opt-in, off by default). With idleStash: true, a
    session resumed after idleTtlSeconds of idle has its spent tool outputs stashed
    (provider: rules, rulesMode: stashed) at session.start, before the first request
    re-writes a prompt cache that has already expired. Every failure answers { skip }, and the
    classic PreCompact hook blocks Claude Code's own summary for an idle compaction, so it never
    becomes the built-in summary. idleTtlSeconds defaults to 7200, not 3600: Claude Code 2.1.296
    keeps the cache warm for 60 minutes after a turn (cache keepalive), so with a 1-hour TTL it is
    cold about two hours after the last turn. Resumed interactive sessions only: on 2.1.296
    $.session.compact is refused from prompt.submit and in -p/SDK sessions, and a timer's
    call skips the plugin's own hooks (all observed in a disposable session). Other knobs:
    idleMinRemovableTokens (10000), idleMaxPassesPerSession (8). The request log
    (LCC_REQUEST_LOG=1) gains trigger, gap_bucket and removed_tokens on idle passes.
    Evidence (INFERIDO) and limits: docs/CLAUDE_CODE.md#idle-stash-opt-in.
  • compact_transcript: ref_unchanged: true. Returns each message the pass left unchanged
    as {"ref": <input index>} and leaves per-call decisions out. Claude Code caps an MCP
    result (MAX_MCP_OUTPUT_TOKENS) and hands a plugin's $.mcp.call an error text in place of
    a larger one; a 12-message session with four 10 KB reads already crossed it (49 KB answer).
    The idle stash uses it; the /compact path is unchanged.

Changed

  • README: leads with /compact vs lcc on a real session, the plugin install with
    rulesMode=stashed, and how to send numbers.

Fixed

  • Claude Code plugin: /compact on a long session always fell back to the built-in summary.
    The session.compact hook asked compact_transcript for the whole compacted conversation.
    Claude Code caps an MCP result (MAX_MCP_OUTPUT_TOKENS, 25,000 tokens by default on 2.1.296)
    and hands a plugin's $.mcp.call a pointer text in place of a larger one, so on any session
    worth compacting the answer was not JSON and the hook stood down. The hook now asks for
    ref_blocks: true (below) and rebuilds the conversation from the session's own messages; the
    idle stash uses it too. A capped answer now toasts built-in summary (tool result is not JSON (N chars; over Claude Code's MCP output cap?)) instead of a JSON parse error.
  • compact_transcript: ref_blocks: true. ref_unchanged down to the block: a changed
    message comes back as {"ref": i, "toolUses": [kept indices], "toolResults": [kept index | {"ref", "tool_use_id", "text", "head"?, "tail"?}]}, a replaced result being the original's
    first head and last tail UTF-16 units around text. Only placeholders and trim notes
    travel. On a synthetic 40-output, 1 MB transcript (rules_mode: lossy) the whole answer is
    1.5 MB, ref_unchanged 10.6 KB and ref_blocks 2.1 KB; ref_unchanged grows with every
    trimmed result it sends whole (86 KB for 21 trims on another shape, against 6.3 KB).
  • recoverable / stashed disk check vs Claude Code's Read format. Claude Code 2.1.296
    numbers the empty line after a file's final newline (480\t for a 479-line file), so every
    whole-file read of a newline-terminated file failed the disk check (keep_disk_mismatch)
    and was never replaced or stashed. That empty trailing row is now ignored; a non-empty one
    still fails.

Limits. Every measured number in this release comes from the maintainer's own sessions and is labelled INFERIDO; see docs/EVIDENCE.md. Run lcc measure-session <session.jsonl> --markdown and post yours in Discussions → Results.

pip install -U "local-context-compiler[tiktoken]"

v1.1.0

Choose a tag to compare

@lucasmartins-ai lucasmartins-ai released this 19 Oct 09:04
596b18c

Added

  • Claude Code plugin: rulesMode: "stashed". The plugin accepted only lossless,
    recoverable and lossy, so the stashed mode could not be picked from Claude Code. Docs now
    recommend stashed with the recall tool, with the one measured run (INFERIDO) in
    docs/TOOL_CALLS.md.

  • rules_mode: "stashed" and lcc recall. Tool-call compaction can now remove what
    lossy drops or trims and what recoverable replaces without losing it: the exact bytes go
    to a local content-addressed stash (~/.cache/lcc/stash, 0700/0600, LCC_STASH_DIR)
    and the result becomes [removed: N tokens · lcc recall <sha12>]. lcc recall and the MCP
    tool recall re-hash on read. stdio only; lcc mcp --http refuses it. lossless stays
    the default. Measured offline on 847 real-session dossiers (owner-attested, private
    receipts): 30.1% pooled token reduction, 0 verification violations. Agent-in-the-loop eval on
    172 items (INFERIDO): non-inferior to full context on artifact recall with 36.6% fewer tokens,
    and the model recalled the right blob on 90% of probes about stashed content.

  • provider="specialist-v3": a local block ranker for agent transcripts. A frozen
    LR + boosted-tree model (183 KB, plain Python, no new dependency) ranks blocks and drops the
    lowest-ranked until specialist_drop_fraction (default 0.3) of candidate characters is gone.
    LCC's protection rules run first, and deterministic-protection and high-risk blocks are never
    dropped. Block metadata (message index, role, kind, tool) comes from trusted
    segments (render_transcript), never from text. auto uses it whenever segments are
    given (see Changed); plain text keeps the Jev/mechanical path.
    The model's sha256 is checked at load, and a mismatch falls back to mechanical. Held-out results and
    limits: docs/SPECIALIST.md.

Changed

  • Behaviour change: auto is now specialist-v3 whenever structured transcript blocks are
    present
    , ahead of Jev and Laya. That covers block compaction with segments and every
    tool-call path (Claude Code hook, MCP compact_transcript, lcc compact --mode tool-calls), which always have them. Jev runs only when named (provider: jev, the
    plugin's provider setting, --provider jev); a key alone no longer selects it. Plain-text
    auto is unchanged (Jev, else mechanical). The plugin and MCP compact_transcript default
    changes from jev to auto. To restore the old behaviour set provider to jev.
    Evidence (INFERIDO): held-out, v3 ties Opus on recall and beats Gemini Flash on all-kept;
    dev (43 sessions, optimistic) v3 0.886 vs Jev 0.818 recall AUC; Jev was not run on the
    held-out set. See docs/SPECIALIST.md.
  • Tool-call mode with specialist-v3: the transcript is rendered with trusted segments,
    block compaction runs unchanged, and a call is dropped only when every block of its call
    and result was dropped (trimmed when only the result was). Text is never touched. Any
    ranker error keeps every call (specialist_v3_failed_fail_safe). rules
    (lossless default) is unchanged.
  • The request log records provider and degradation_reason (labels only) for tool-call
    compaction, and lcc request-log counts entries by provider.

Deprecated

  • Laya (provider: laya). No automatic or fallback path selects it (none did for block
    mode; it is now stated). It still runs when named. Removal is planned for a later release.

Security

  • The TypeSafe bearer key could leave over plain HTTP or follow a redirect.
    TYPESAFE_ENDPOINT must now be https:// (plain http:// is accepted only for
    loopback). If it is not, Jev is reported as unavailable, with a note on stderr.
    Redirects are refused, because urllib re-sends Authorization to whatever host a
    301/302/303 names.

  • The request log was world-readable under the default umask. The log directory is
    now created 0700, and requests.jsonl (including the rewrite done on deletion) is
    0600. The goals are redacted, but they are still users' text.

  • lcc mcp --http let anonymous callers choose paid or heavy providers. Any client
    that reached the server could send provider: "jev" and spend the operator's TypeSafe
    key, or provider: "laya" and load model weights on the host. Without the new
    LCC_HTTP_TOKEN env var, HTTP callers are now limited to mechanical. With it set,
    POST /mcp requires Authorization: Bearer <token> and any provider is allowed. Each
    request also gets a 30 s socket timeout, so a stalled body no longer holds a thread.

  • The npm package would have published the whole checkout. package.json had no
    files field, so npm pack took 1054 files (45.8 MB), including
    .claude/settings.local.json and session worktrees. It now ships index.js,
    index.d.ts, README.md, LICENSE and NOTICE.

Fixed

  • Anonymous HTTP callers could not use compact_transcript at all. The provider gate
    allowed only mechanical, and compact_transcript refuses that. The offline rules
    provider is now allowed too.
  • lcc route from a pip install ran with risk detection off. configs/ is not in
    the wheel, and with no YAML in the working directory every risk keyword list was
    empty. The built-in defaults now match configs/router_policy.yaml.
  • A failed neighbourhood fingerprint was silent. The decision cache then fell back to
    content-only keys without saying so. It now adds a neighbourhood_fingerprint_failed
    warning to the report.

Changed

  • benchmarks/research/results/matrix.jsonl (16.9 MB) is now committed as
    matrix.jsonl.gz (227 KB). The file decompresses to the same bytes, run_matrix.py
    writes it with mtime=0 so identical rows produce identical bytes, and
    show_losses.py reads it. The old file stays in git history.

  • Package metadata names the author: Lucas Martins lucas@lookadev.com.

  • New Dependency audit workflow: pip-audit runs weekly and on demand. It is not run on
    PRs, so an advisory published against a dependency does not block unrelated merges.

  • Removed act2_router/, which held 17 star re-exports of lcc.router and lcc.agents
    and was never in the wheel. The 7 tests that still used it now import the real modules.

  • The supersession cue list is defined once in graph.SUPERSESSION_CUES, and the graph
    adds its two extra cues explicitly. Matching is unchanged.

  • fast-jev-compaction is a dev dependency, because only the benchmark arm uses it. CI
    now tests Python 3.11–3.14 (the classifiers already claimed 3.13 and 3.14), runs
    ruff check . on the whole repo, and builds both dists and runs twine check --strict
    on every PR.

  • Text containing <|endoftext|> was never counted exactly. tiktoken rejects
    special-token strings by default, and the counter silently fell back to the heuristic.
    Pasted chat logs then ran --require-exact-tokens and cache economics on estimates.
    These strings are now counted as plain text.

  • A malformed 200 from TypeSafe aborted compaction instead of falling back. A
    non-JSON body raised JSONDecodeError, and a non-object answer raised AttributeError.
    Both escaped the JevError fail-safe. They now raise JevMalformedResponseError, are
    written to the ledger, and degrade to mechanical scoring.

  • One malformed line ended the lcc mcp stdio session. A parse error now gets a
    -32700 reply and the server keeps reading. A JSON array gets -32600. Non-object
    params or arguments get -32602 on both transports, instead of an internal error
    (stdio) or a dropped connection (HTTP).

  • lcc mcp never completed the MCP handshake with real clients. The stdio server only
    read LSP-style Content-Length frames, while the MCP stdio transport (Claude Code, Codex,
    Cursor) sends one JSON message per line, so initialize hung until the client timed out.
    In Claude Code this left the plugin's lcc server disconnected and every /compact fell
    back to the built-in summary (no connected MCP tool "compact_transcript"). The server now
    reads newline-delimited JSON and still accepts Content-Length frames, replying in the
    framing of each request.

v1.0.0 — first stable release

Choose a tag to compare

@lucasmartins-ai lucasmartins-ai released this 07 Oct 23:07
c19c244

First stable release. The library API, the CLI surface and the MCP tool set are now covered by the compatibility promise.

pip install -U "local-context-compiler[tiktoken]"

Highlights

  • Verbatim tool-call compaction for agent sessions. On a real Claude Code session: 55,411 → 18,761 tokens (−66.1%) with all 42 user/assistant texts preserved byte for byte; on a real 337-message agent session: −81.7% with all 58 texts preserved (Jev judge; measurements and limits).
  • Local judge, no key: lcc compact --mode tool-calls --provider laya retires spent tool calls offline.
  • Claude Code plugin: claude plugin marketplace add lucasmartins-ai/lcc && claude plugin install lcc@lcc replaces the lossy /compact summary, and falls back to it on any failure.
  • Hosted MCP + ChatGPT plugin: six read-only tools over streamable HTTP (lcc mcp --http), with a published privacy policy.
  • MSI research release (sprints 1–10): lcc.msi.compile() with an audit receipt, Context IR (context-ir/0.1), bounded sufficiency restoration, audit findings R1–R3 resolved, paper and limitations.
  • Docs: new README with demos and a quick start, and a landing page at https://lucasmartins-ai.github.io/lcc/.

Boundaries

The default path is deterministic and offline: no API key, no model call, no network. Model judgment (jev, laya, verifier, agents, routing) is opt-in and outside the deterministic core. Token counts are exact only with local tiktoken assets and are labelled as estimates otherwise. Not included: RAG, embeddings, vector databases, semantic retrieval execution.

Full list of changes: CHANGELOG.md · v0.5.0...v1.0.0

v0.5.0

Choose a tag to compare

@lucasmartins-ai lucasmartins-ai released this 21 Sep 23:09

Release focus: Jev first, and enough of it shipped to prove it. The Jev provider is now the
headline path in the documentation (docs/JEV.md), the compiler learned a message-level mode
(lcc compact --mode tool-calls) that drops spent tool calls and results from an agent
transcript without ever summarizing, and a Claude Code plugin replaces that editor's built-in
compaction summary with the same pass. A head-to-head study against fast-jev-compaction 0.4.0
on the same transcripts with live Jev is published in benchmarks/research/TRANSCRIPT_AB.md:
this mode removed 13.3 / 39.2 / 55.7% of the transcript at fact recall 1.00 / 1.00 / 1.00, where
that library removed 70.5 / 78.9 / 83.4% at recall 0.00. Two defects the study found in the new
mode are fixed below.

Added

  • Tool-call compaction study: benchmarks/research/TRANSCRIPT_AB.md measures lcc compact --mode tool-calls against fast-jev-compaction 0.4.0 on the same three deterministic transcripts with matching options and live Jev. Our mode removed 13.3 / 39.2 / 55.7% of the transcript at fact recall 1.00 / 1.00 / 1.00; the npm arm removed 70.5 / 78.9 / 83.4% at recall 0.00 (every evidence pair dropped, reproducibly). New harness: make_transcripts.py (ground-truth corpus with a self-check), run_transcript_ab.py (both arms measured by the same ruler, per-pair outcomes derived from the emitted messages, arm selection in the artifact filename), fast_jev_arm.mjs; artifacts under benchmarks/research/results/. Canonical row in benchmarks/research/RESEARCH_STATUS.md.

Fixed

  • Tool-call compaction, state budget honesty: the mode scored the conversation with the stdlib heuristic, which undercounts JSON-heavy transcripts by roughly a third. On a then-497-message snapshot of a real session it judged a 22 936-token state inside its 25 000 budget, sent 38 batches, and had every one refused by the API (400 max_tokens_exceeded, 123 741 characters ≈ 31 000 real tokens) before degrading to keep-everything. _budget_tokens now counts with the real tokenizer when the environment has one and otherwise inflates the heuristic by the measured factor, so a session that cannot fit fails closed in ~50 ms with a typed error and zero requests spent. Regression tests: test_the_budget_is_never_optimistic, test_fitting_uses_the_budget_function.

  • Tool-call compaction: a drop decision reported chars_after equal to chars_before, so the report's tool_reduction_ratio read 0.0 while the transcript had shrunk by more than half. A result shorter than --trim-head-chars was also "trimmed" into a longer transcript, because the audit note outweighed the characters it removed; such a pair is now kept whole with reason short_result_kept_whole. Both were found by the new A/B study and both have regression tests.

  • Tool-call compaction, real sessions: benchmarks/research/REAL_SESSIONS.md adds the measurement the release was missing. A real Claude Code session in this repository (192 messages, 75 tool calls), in the exact shape the hook receives, goes from 55 411 to 18 761 tokens (−66.1%) with all 42 user/assistant texts preserved byte for byte; this release's own 337-message Hermes session (187 tool calls) goes from 199 717 to 36 645 tokens (−81.7%) with all 58 texts preserved. New harness: measure_session.py (Claude Code JSONL or the Hermes store → the session shape → the shipping code path).

  • Claude Code plugin (plugins/claude-code, .claude-plugin/marketplace.json): replaces the built-in compaction summary with verbatim tool-call compaction. The session.compact hook is a thin ESM adapter (no build step, no dependencies) that hands the transcript to the plugin's own lcc mcp server and maps the result back; it stands down to Claude Code's own summary on any failure, on a below-minReductionRatio pass, or when a message carries content it cannot rebuild. Install and limits: docs/CLAUDE_CODE.md; operator guide: plugins/claude-code/hooks/README.md. Verified by claude plugin validate plus test/hook-map.test.mjs (10 cases, run by npm test); a live head-to-head against the built-in summary is still pending.

  • Tool-call compaction: lcc compact --mode tool-calls (and the MCP tool compact_transcript) compacts a session transcript instead of a document. Each tool_use is paired with its tool_result by id; the first and newest --preserve-recent messages are pinned; every other pair gets two typed Jev questions (does the call still matter, does its result still need to be there verbatim) and is kept, trimmed to --trim-head-chars plus a note, or dropped whole. User and assistant text is never scored, trimmed or rewritten, results stay in the message they arrived in, and a history that cannot be shown inside --max-state-tokens fails closed instead of being judged blind. mechanical/laya are refused with exit 2 rather than pretending to judge semantics. The report is transcript-compaction-1.0 and reads through lcc explain unchanged. New: src/lcc/relevance/transcript.py, docs/TOOL_CALLS.md.

  • Jev path documentation: new docs/JEV.md (typed-decision contract, key resolution order, honest fallback table, measured behaviour and its limits), a Jev-first section at the top of README.md with a five-line --provider jev example and a TypeSafe badge, a Jev path in docs/QUICKSTART.md, and tests/test_docs.py assertions that keep the document linked from both entry points and pinned to the tested phrases.

Changed

  • README positioning: the compaction claim now leads with "Jev-powered relevance compaction that never summarizes", and the provider table's note points to docs/JEV.md. No behaviour change; the offline default path (mechanical/laya, no key, no network) is unchanged.

Boundaries and honesty (unchanged from previous releases)

  • The deterministic core stays deterministic: cleaning, dedupe, token budget, prompt builder,
    inspection and benchmarking make no network calls and run no models. Model judgment —
    lcc compact providers, the tool-call mode, agents, routing, the verifier — is opt-in and
    lives outside that core (ADRs 0010–0016).
  • lcc prepare uses deterministic lexical selection only: it never summarizes, rewrites,
    paraphrases, embeds or semantically ranks source content.
  • Token counts are exact only with local tiktoken assets; otherwise they are a clearly
    labelled approximate count, and the tool-call state budget inflates the heuristic rather than
    trusting it.
  • Not included: RAG, embeddings, vector databases, an API server, a hosted product, semantic
    retrieval execution.
  • The Claude Code plugin installs from this repository's marketplace and its backend is measured
    on real transcripts, but no /compact in a live session has been intercepted. What that
    means, and the two-minute recipe to close it, is in docs/CLAUDE_CODE.md.

Commands: lcc optimize, inspect, prepare, bench, intake,
compact (--mode blocks|tool-calls; providers mechanical/jev/laya), explain, agent,
route, mcp (stdio, six tools: compact, compact_transcript, inspect, prepare,
explain, intake).

v0.4.0

Choose a tag to compare

@lucasmartins-ai lucasmartins-ai released this 21 Sep 11:55

[0.4.0] - 2026-09-21

Added

  • Laya live validation (2026-09-21): the real backend runs end-to-end (verified offline, in-budget, judged, no fallback) and the pre-validation "Laya = Jev, −22.6% on XL" claims are corrected — measured with real weights the default checkpoint keeps essentially every block (0.0% reduction at small/medium/large, −0.5% at XL; Jev −52% / mechanical −70% at XL, all at 100% category recall). run_comparative_stress_test.py and run_laya_comparison.py run real backends by default (--mock opts into the labelled harness) and every result row now carries harness: real|mock. New run_laya_context_cases.py (budget cases A/B/C, offline) and gated live test tests/test_laya_integration.py (LCC_LAYA_INTEGRATION=1). Pre-validation mock archives preserved and labelled. Full rows: benchmarks/research/RESEARCH_STATUS.md.

  • Laya opt-in local semantic backend for lcc compact (ADR 0016, local-context-compiler[laya] extra): LayaClient judges typed noul/choice/score questions on-device with zero network and zero remote tokens. Strict context budgeting (laya 512 / laya-multilingual / laya-typed-decisions 1024, HEAD_RESERVATION_TOKENS = 192 → 319/831 state tokens, --laya-context-limit override); oversized blocks are kept whole with laya_context_limit_exceeded, never sliced. Temperature calibration (calibrate_noul, --laya-temperature, default 1.0) is pure stdlib and part of the decision-cache identity. New CLI flags --laya-model, --laya-device, --laya-context-limit, --laya-temperature; Python API LayaClient + provider="laya". Missing extra or failed batch falls back honestly to mechanical (laya+mechanical_fallback, degraded: true, typed reason) — never a fake judged. Operator guide: docs/LAYA.md.

  • Laya maturity: report schema relevance-compaction-1.2 (additive; new context_budget_used, laya_temperature alongside laya_model_requested/resolved, laya_context_limit, latency_ms), lcc compact --help provider decision guidance (mechanical vs laya vs jev vs auto), and fallback/observability tests including the missing-dependency path.

  • Discoverability: docs/QUICKSTART.md (5-minute offline tour), runnable examples/compact_providers.py (mechanical vs laya vs honest jev-skip on one dossier) and examples/long_session_cache.py (decisions-cache + prefix-marker + append-only session), both covered by tests/test_examples.py.

  • Deterministic boundary hardening: tests/test_deterministic_boundary.py fails any network/LLM/embedding capability import in cleaning, token_budget, inspection, pipeline, lexical_selection, benchmarking (sole exceptions: the fail-closed download guard and *_VERSION identity constants), fails any opt-in layer import into core, and proves core outputs are byte-identical with the network disabled. ADR 0010 clarified (decision unchanged): Laya, sufficiency, verifier, agents, and router are opt-in outside the core.

  • MCP server MVP (lcc mcp, src/lcc/mcp_server.py, docs/MCP.md): stdio JSON-RPC tools compact (defaults to offline mechanical), inspect, prepare, explain, intake. Stdlib only — zero new dependencies; covered by tests/test_mcp_server.py including a real framed-subprocess session.

  • Minimum-sufficient-context compilation (ADR 0014): lcc compact now answers two questions — "is this block relevant?" and "after these drops, can the objective still be solved from what remains?". New modules lcc.relevance.safety (independent relevance/necessity/dependency-risk/semantic-risk/redundancy axes), lcc.relevance.graph (typed SUPPORTS/QUALIFIES/CONTRADICTS/SUPERSEDES/DEPENDS_ON/DUPLICATES/DERIVED_FROM edges), lcc.relevance.sufficiency (verify-then-restore with --sufficiency/--no-sufficiency, --max-restorations), lcc.relevance.trim (type-aware TRIM: valid-syntax JSON/YAML/XML, row-aware tables, line-bound code, head+tail logs, TRIM→KEEP for high-stakes content), and lcc.benchmarking.metadata (reproducible benchmark identity). Decision cache keys bind the full policy identity (v1.1); reports add tokenizer identity, jev_model_resolved, sufficiency/relationship/marker accounting, per-decision confidence/relationships/policy/content-type, and compilation_ms. New CLI flags --sufficiency/--no-sufficiency, --max-restorations, --confidence-threshold. Adversarial suite grows 20→26 cases (scope quantifiers, coreference, paraphrase duplicates, set-level VAT chain, HTML/log injection); new benchmarks/research/run_answer_eval.py gates answer-level regressions (26/26 pass, 0 regressions mechanical). Node engine gains tokenizerIdentity()/estimateTokensWithMeta() with a documented parity boundary.

  • Relevance compaction v1.1 (2026-09-18 review of tamaratran/fast-jev-compaction, adapted to this boundary): three-way decisions — keep / trim / drop — where borderline blocks keep a bounded head plus an audit note (--trim-threshold, --trim-head-chars 0 restores strict keep/drop); --preserve-tail N pins the newest blocks untouched for live append-only contexts; scoring batches run concurrently (--max-workers, up to 8); reduction accounting (reduction_ratio, worth_it, --min-reduction) flags passes that save too little to justify a cache epoch. Report schema is relevance-compaction-1.1 (additive; decisions[].chars_after for trims). A failed scoring batch no longer aborts the remaining ones — missing blocks fall back mechanically, as before.

  • lcc compact (ADR 0013): opt-in instant relevance compaction. Drops context blocks that are irrelevant to an objective using narrow model judgment (TypeSafe System One / Jev) in batched calls, with a fully local mechanical fallback (zero-lexical-overlap blocks only) when no API key is configured. Fail-safe: provider failures keep content, never drop it.

  • Cache-aligned compaction: sticky decisions (--decisions-cache) pin (objective, block) outcomes so unchanged history keeps byte-identical output; --protect-prefix / --prefix-marker make early-prefix mutation impossible; reports expose first_mutation_offset, prefix_sha256, and output_sha256 for automated cache accounting (docs/CACHE_ALIGNMENT.md).

  • lcc intake --enable-relevance (with --relevance-threshold, --relevance-provider): runs relevance compaction on raw input before intake compilation; results appear in the intake report JSON and summary.

  • Deterministic relevance block segmentation (lcc.relevance.blocks) with content-addressed ids (blk_0001_<12 hex>), fenced-code-aware spans, and byte-exact reconstruction; shared-ledger audit trail for every scoring call (lcc_compact feature tag, matching the LookADev fleet ledger schema).

  • lcc agent local agent subsystem (lcc.agents) with native support for Gemma 4 e4b (turn formatting <start_of_turn>user...) and Qwen3.5-4B (ChatML <|im_start|>...), Ollama, llama.cpp / GGUF (e4b 4-bit), vLLM, MLX, and mock backends with health checks and zero remote token usage.

  • lcc route hybrid local/cloud routing subsystem (lcc.router) with conservative verification gates (RuleBasedVerifier, LocalLLMVerifier) and selective escalation to Fireworks AI.

  • lcc intake intelligent prompt intake engine (lcc.intake) with automated readiness triage (READY_TO_EXECUTE, NEEDS_LIGHT_REFINEMENT, NEEDS_INTAKE, BLOCKED), intent/assumptions extraction, and clarifying questions generation.

  • lcc speech and transcript cleaning (lcc.cleaning.speech) with deterministic speech disfluency stripping (English & Portuguese fillers), Whisper audio tag elimination ([Music], [Applause], etc.), hallucinated subtitle/video outro removal, and speaker-turn collapsing.

  • Code-block and Markdown-table safe normalization in lcc.cleaning.normalize protecting fenced code blocks (``` and ~~~) and Markdown tables against destructive inner-space collapsing and indentation disruption.

  • Fixed REMOTE_DIRECT router token leak in lcc.router so direct cloud model escalation always receives safe deterministic cleaning (normalization, boilerplate removal, and exact paragraph deduplication).

  • Universal TypeScript & Node.js client package exports (LccCompressor, LccIntake, processIntake).

  • lcc prepare deterministic inspect-first workflow. It runs inspection, skips prompt

    generation for skip/manual_review recommendations, and runs deterministic lexical
    selection plus the existing safe optimization path for optimize_safe/optimize_with_flags,
    with input-file overwrite protection for output and report paths.

  • Machine-readable inspection recommendation scoring signals with deterministic scores,
    thresholds, evidence values, and reason codes for duplication pressure, projected savings,
    token budget pressure, missing pricing, approximate token counting, and manual-review risk.

  • Additive chunk_inventory data in inspection reports, with deterministic chunk IDs,
    character offsets, line spans, paragraph counts, token counts and methods, structural
    labels, literal heading text, and exact duplicate markers.

  • Deterministic question-aware lexical chunk selection for lcc prepare optimize decisions,
    using keyword overlap, heading matches, rare-term matches, heading proximity, and exact
    duplicate markers from chunk_inventory; prepare reports explain selected chunks and nested
    optimization output.

  • Deterministic benchmark coverage for workflow: prepare cases, including bundled
    selection-applied and selection-not-applied fixtures with required evidence markers,
    forbidden irrelevant/duplicate markers, and prepare selection report assertions.

  • Phase 1.7 roadmap status for the implemented deterministic prepare and recommendation
    scoring slices, chunk-inventory slice, lexical-selection slice, and prepare benchmark slice.

  • ADR 0010 documenting the deterministic-first preparation and optional model-assistance
    boundary. Optional future model assistance is not impleme...

Read more

v0.2.0 — Deterministic Context Compiler & Prompt Intake Engine

Choose a tag to compare

@lucasmartins-ai lucasmartins-ai released this 28 Aug 22:49

What's Changed

  • Unified Engine: Single-command lcc intake and lcc optimize pipelines in Python and TypeScript.
  • 2026 Contract Templates: Full alignment with Anthropic Claude XML, Cursor, and OpenAI structured contracts.
  • Token Estimation: Built-in heuristic and exact tiktoken counting with projected cost savings.
  • Zero Telemetry: 100% local execution with no network overhead or API keys required.

Full Changelog: https://github.com/lucasmartins-ai/lcc/commits/v0.2.0

v0.1.0 — Deterministic MVP

Choose a tag to compare

@lucasmartins-ai lucasmartins-ai released this 19 Jun 22:37

v0.1.0 — Deterministic MVP

Initial public release of Local Context Compiler.

This release provides a deterministic, local-first Python toolkit and CLI for preparing text context before LLM calls. It supports safe cleaning, conservative deduplication, exact/approximate token counting, configurable cost estimates, evidence-aware prompt building, deterministic JSON reports, and a fixture-based benchmark harness.

LCC does not call LLMs, APIs, remote services, embedding models, vector databases, or audio/voice systems. Runtime network access is blocked by default, including indirect tokenizer downloads. If exact tokenizer assets are unavailable locally, LCC falls back honestly to approximate counting and reports that behavior.

Included:

  • lcc optimize
  • lcc bench
  • deterministic benchmark fixtures
  • tokenizer no-network guard
  • JSON reports with schema versioning
  • ADRs 0001–0008
  • CI workflow
  • open source contribution templates

Not included yet:

  • RAG
  • embeddings
  • local or remote LLM calls
  • API server
  • voice/audio
  • transcript ingestion
  • semantic evaluation
  • model routing
  • response verification