Skip to content

v2.0.0 — Multi-Scale Episodic Retrieval, 2-Hop Session Bridges & Official BEAM 10M Validation

Latest
Latest

Choose a tag to compare

View all tags
@kruschdev kruschdev released this 29 Oct 14:59
· 13 commits to main since this release
v2.0.0
694ff34

krusch-context-mcp v2.0.0 upgrades the retrieval and epistemic grounding engine from single-pass turn search to a multi-scale episodic memory architecture—while keeping the 5 canonical MCP verbs (retrieve, remember, revise, nudge, health), the 5-category closed taxonomy, and the zero-Docker Node 22 node:sqlite .agent/context.db layout 100% backward-compatible (zero manual migrations required).

Since v1.8.0 was the previous published GitHub Release, these notes cover both the v2.0.0 multi-scale retrieval release and the v1.8.1–v1.9.0 storage, provenance, and 8K embedding upgrades included in v2.0.0.

🚀 What's New in v2.0.0

1. Two-Level Hierarchical Episode Rollups (src/episode-rollup.js)

  • Zero-LLM Extractive Episode Panoramas: Automatically groups multi-session or 64-turn windows into compact extractive episode digests (<= 420 chars) inside ide_episode_rollups, paired with an external-content Porter-stemmed SQLite FTS5 index (ide_episode_rollups_fts) and AFTER INSERT / DELETE / UPDATE sync triggers.
  • Salience-Weighted Float32Array(1024) Centroids: Computes L2-normalized 4,096-byte binary centroids weighting standing directives (1.40x), session openers (1.35x), and explicit state transitions (1.30x), strictly sanitized via stripTurnCitations so macro panoramas never leak raw leaf Turn IDs.
  • Zero Small-Store Overhead: Remains a < 0.05 ms no-op below KRUSCH_ROLLUP_MIN_ROWS (240 active rows default) with O(1) SQLite pragma_data_version watermark caching on warm stores. Renders #### 🗂️ Hierarchical Episode Panorama on macro summarization/ordering queries, rescues deep child turns beyond KRUSCH_MAX_ACTIVE_SCAN, and surfaces rollupCount in krusch_context_health.

2. Gated 2-Hop Multi-Session Bridge Expansion (src/retrieval-bridge.js)

  • Zero-RPC Orthogonal Residual Bridging (< 5 ms): Connects entities and uncovered query facets across disjoint sessions in a single synchronous retrieve call (expandMultiHopBridges, computeResidualBridgeVector).
  • Damped Gram-Schmidt Projection: Projects out the dominant Hop-1 query direction (v_bridge = normalize(v_hop1 - 0.75 * <v_hop1, v_q> * v_q)) over in-SQLite Float32Array blobs and executes a targeted cross-session FTS5 + dense pass—gated on uncovered query stems or high-confidence multi-session entity anchors (rawScore >= 0.56) so unanswerable abstention queries never hop.
  • Rank-Bounded Packing: Enforces post-smoothing ceilings (0.84 on uncovered-facet bridges, 0.76 on entity-only bridges) so Hop-2 evidence complements rather than displaces direct Hop-1 Top-10 anchors within limit_tokens.

3. Corpus-Scale-Aware Attribute-Level Abstention (src/retrieval-epistemic.js)

  • Compositional USER: Window Grounding (evaluateAttributeGrounding): Evaluates stem co-occurrence across ±1 same-session turn windows (W_k = U_{k-1} ∪ U_k ∪ U_{k+1}) with zero hardcoded word lists, distinguishing grounded multi-turn facts (GROUNDED), attested entities with unattested attributes (UNATTESTED_ATTRIBUTE), and unsupported queries (UNSUPPORTED).
  • Extreme-Value Isotropic Cosine Scaling: Applies log10(N) corpus-size calibration (scaleShift = min(0.11, max(0, (log10(N) - 2) * 0.055))) so 10,000+-turn dense neighbor distractors (0.48–0.59 raw cosine) cannot mask NO_KNOWN_SUPPORT or > ⚠️ **Attribute Grounding Note** signals.

🧱 Included Since v1.8.0 (v1.8.1 + v1.9.0 Foundations)

  • 8K snowflake-arctic-embed2 Default Local Embedder + Isotropic Calibration (v1.9.0): Upgrades default local Ollama embeddings from 512-token bge-large to 8,192-token snowflake-arctic-embed2 (1024d, ~664 MB VRAM, modern /api/embed with /api/embeddings fallback, asymmetric query: prefixing, MRL 1024d truncation for qwen3-embedding / nomic-embed-text, and model-aware calibrateQuerySimilarity). Zero-GPU offline mode remains the out-of-the-box default when Ollama is not running.
  • Compact Binary Float32Array Blobs & External-Content Porter FTS5 (v1.9.0): Stores 1024-dim vectors as 4,096-byte IEEE-754 embedding_blob columns (~4.5x smaller than JSON text, zero JSON.parse scan overhead) alongside ide_agent_memory_fts (porter unicode61) for sub-millisecond deep lexical recall.
  • Passive State Evolution & Speech-Act Timelines (v1.9.0): Surfaces ### 🔄 Chronological State Evolution (Earlier → Latest) (detectTemporalUpdates), ### 📅 Chronological Event Sequence (Oldest → Newest) on ordering queries, and corpus-wide Lexical Grounding Note diagnostics (computeCorpusUnattestedTerms).
  • Span-Grounded Provenance & Drift Self-Healing (INV-12, v1.8.1): Passing provenance: { file, locator, quoted_sentence } computes a SHA-256 anchor_hash (start_line..end_line). During retrieve({ include_state: true }), shifted line numbers self-heal automatically while mutated spans surface under ## ⚠️ Span-Grounded Drift Review.
  • Pre-Write Secret Refusal & Multi-Project Isolation (v1.8.1–v1.9.0): Blocks API keys, PATs, and private keys before SQLite persistence (scanForSecretLeak), scopes steering nuggets by composite (key, project), and scopes nudge feedback weights by project.

📊 Empirical Verification: Official ICLR 2026 BEAM (100K – 10M)

Evaluated hermetically (127.0.0.1:11434 loopback, zero cloud APIs, 124/124 automated tests, 8/8 verify_zero_leakage cryptographic provenance & anti-gaming gates) across all 100 conversations (163,559 turns / ~178.34M tokens) and all 2,000 / 2,000 questions of Mohammadta/BEAM (100K, 500K, 1M) and Mohammadta/BEAM-10M (10M) under a constant 8K-token retrieval budget (snowflake-arctic-embed2:latest + unmodified openai/gpt-oss-120b reader graded by cross-family qwen2.5:32b & hermes3:latest referees):

Official BEAM Split Corpus Scale v2.0.0 Hybrid 8K Recall (vs. Pure Dense 8K) Key v2.0.0 & Multi-Split Highlights
100K (n = 400) 20 convs · ~3.06M tok 92.58% (vs. 92.50% Dense; R@10 = 57.79%) 85.94% (qwen2.5:32b) / 92.50% (hermes3) Contradiction Resolution (98.13% w/ SRE LoRA vs. past.dev 78.13%).
500K (n = 700) 35 convs · ~19.40M tok 81.51% (+2.70 pp over Dense) Zero E2E QA degradation from 100K → 500K (71.82% hermes3, 89.64% Contradiction Resolution).
1M (n = 700) 35 convs · ~40.02M tok 74.15% (+1.91 pp over Dense; R@10 = 42.23%) Zero E2E QA degradation at 10x context growth (72.86% hermes3, 64.64% qwen2.5:32b; IF R@10 +5.71 pp).
10M (n = 200) 10 convs · ~115.87M tok 55.54% (+2.03 pp over Dense; 93.00% IE) Multi-Session Reasoning jumps +11.79 pp (29.54% Qwen) & +8.75 pp (38.33% Hermes)—surpassing cloud past.dev (28.17%) on both referees; Abstention +10.00 pp (55.00%).

⚡ Quickstart & Upgrade Notes (Zero Breaking Changes)

Requires Node.js >= 22.0.0:

npx krusch-context-mcp@latest init
  • Drop-In v1.8 / v1.9 → v2.0.0 Upgrade: ide_episode_rollups and ide_episode_rollups_fts initialize lazily on startup with zero manual DDL steps.
  • Upgrading Pre-v1.9.0 Local Dense Vectors: Both bge-large and snowflake-arctic-embed2 use 1024 dimensions (EMBED_DIMS=1024), but occupy distinct vector spaces. If upgrading a pre-v1.9.0 store that used bge-large, either pull ollama pull snowflake-arctic-embed2 and re-embed (or clear embedding / embedding_blob for lazy backfill), or set EMBED_MODEL="bge-large" in .env to keep your existing vector space. See docs/MIGRATION.md.

Full Changelog:

Assets 2
Loading