Skip to content

v0.6.0 — IR audit full implementation

Latest
Latest

Choose a tag to compare

View all tags
@ssmurfgg04-gif ssmurfgg04-gif released this 29 Aug 13:59
· 50 commits to main since this release
v0.6.0
0e0c492

What shipped in 0.6.0 — full IR audit implementation

Implements EVERY item from the user's Aug-2026 IR audit (12 Lucene/Solr primitives + 5 Google-style query-time fixes + SQLite PRAGMA tuning + parallel benchmark runner).

New modules (cortexm/bridge/)

  • synonyms.py — Lucene synonym_graph filter ported to μ=0. 8 curated concept clusters (employment, residence, pet_name, education, vehicle, preference, negation, family). Phrase-level substitution via master regex (longest-first sort). Default max_expansions=16. Runtime-extensible via register_cluster().
  • recognizers.py — Microsoft Recognizers-Text style deterministic entity resolution. 40+ holidays with algorithmic resolution (Thanksgiving = 4th Thursday of November via _nth_weekday; Easter = Gauss's computus; MLK = 3rd Monday of January; Memorial Day = last Monday of May). Currency extraction (USD/EUR/GBP/JPY/CNY/INR with both $/€/£ symbols and "dollars"/"euros"/"pounds" word forms).
  • fst.py — Lightweight FST for query normalization. 60+ abbreviation expansions (UCLA → University of California Los Angeles, MIT → Massachusetts Institute of Technology, NYC → New York City) + 25+ common misspellings (recieve → receive, teh → the). Idempotent. Case-preserving.
  • slang.py — Curated slang normalization dictionary (80+ entries). bruh → "", deadass → seriously, no cap → truthfully, finna → going to, gonna → going to, y'all → you all. Multi-word-phrase aware ("no cap" is one semantic unit).
  • negation.py — Negation detection + indexing as metadata (not as positive fact). 30+ negation markers (don't, do not, never, no longer, stopped, quit). Sentence-level detection. extract_with_negation() splits text into positive_text + negations. SQL schema for negation_records table.
  • multilingual.py — Language detection + per-language routing via Unicode script analysis (no model needed). Detects en/zh/ja/ko/ar/hi/ru/th. Conservative 30% threshold. segment_by_language() splits code-switched text into language-homogeneous segments.
  • query_rewrite.py — Orchestrator running 4 stages in order: slang → FST → synonyms → recognizers. Returns list of expanded queries (original always first). Holiday partial-match bug fixed via negative-lookahead regex + consumed_spans tracking.
  • ir_pro.py — All 12 IR primitives the audit flagged, consolidated in one module:
    • 6-stage analyze() pipeline: NFKC → lowercase → tokenize → stopword removal → stem
    • 165-word English stopword set + 6-language stopword lists (en/es/fr/de/it/pt)
    • nfkc_normalize() (fi → fi, ² → 2)
    • strip_accents() (Café → Cafe)
    • stem() lightweight Porter-like stemmer (running → runn, dogs → dog, cities → city)
    • build_phrase_query() (slop=0 → "phrase"; slop=2 → NEAR(a, b, 2))
    • phrase_search() (FTS5 NEAR())
    • highlight() (FTS5 snippet() with custom before/after markers)
    • facet_counts() (GROUP BY on facts table — relation/subject/value)
    • more_like_this() (term-vector similarity via FTS5 fts5vocab)
    • range_search() (numeric range via CAST(value AS REAL) B-tree scan)
    • suggest() (fts5vocab prefix search)
    • correct_spelling() (full-string Levenshtein DP — NOT Bitap which does substring matching)
    • correct_query() (per-token spell correction)
    • LRUCache with per-user invalidation
    • optimize_index() (VACUUM + FTS5 'optimize' + wal_checkpoint)

VerbatimPlugin extensions

  • search() now wraps _search_single() (the original v0.5.3 path) with the QueryRewriter — ADDITIVE (runs original + expansions through BM25, unions by rowid, can only surface MORE hits never fewer — protects existing 0.948 canonical score)
  • LRU query cache (capacity 1024) with per-user invalidation on add()
  • 11 new public methods: phrase_search, highlight, facet_counts, more_like_this, range_search, suggest, correct_spelling, correct_query, optimize_index, tune_bm25, invalidate_cache

TraceStore PRAGMA tuning

TraceStore.__init__ now takes 5 new kwargs: pragma_cache_mb (64MB default), pragma_mmap_mb (256MB), pragma_threads (4), pragma_temp_in_memory (True), pragma_locking_exclusive (False, opt-in). Memory class passes Config values through. Google-style read-heavy optimization.

Config

16 new Config fields, all default ON (additive — can only help, never hurt): query_rewrite_enabled, slang_normalization_enabled, abbreviation_expansion_enabled, spelling_correction_enabled, synonym_expansion_enabled, holiday_resolution_enabled, query_max_expansions (8), negation_indexing_enabled, multilingual_routing_enabled, query_cache_enabled, query_cache_capacity (1024), bm25_k1 (1.5), bm25_b (0.75), index_optimize_on_consolidate, highlight_tokens (10), suggest_min_count (2), pragma_cache_mb (64), pragma_mmap_mb (256), pragma_threads (4), pragma_temp_in_memory (True).

Parallel benchmark runner

scripts/longmemeval_canonical_parallel.py — multiprocessing.Pool (spawn context to avoid fork-safety issues). Each worker opens its own READ-ONLY SQLite connection (WAL allows concurrent readers). Expected 4× speedup on 4-core machine. Mirrors the sequential runner's arg parser so callers can swap transparently.

Tests

tests/test_v060_ir_pro.py — 109 new tests covering every new module + every new VerbatimPlugin method + Config new flags + PRAGMA tuning + μ=0 invariant upheld.

Full regression: 626 passed (was 517; +109), 24 skipped, 0 failures in 21s.

μ=0 invariant upheld

Verified via test_memory_add_does_not_increment_llm_calls: Memory.add + Memory.search does NOT bump the LLM_CALLS counter. All new modules are pure Python (regex + dict + DP), no LLM, no API, no statistics beyond IDF/BM25.

Bugs found + fixed during testing

  • recognizers.py had invalid Python syntax (walrus operator inside dict literal) → fixed
  • query_rewrite.py had a partial-match bug ("valentine" inside "valentine's day" produced "2026-02-14's Day") → fixed via negative-lookahead regex
  • ir_pro.stem() had regex rules that captured only ONE letter (so "running" → "n") → rewrote with re.subn using (.+) captures
  • ir_pro.correct_spelling() was using Bitap (which does SUBSTRING matching — "dogg" vs "dog" returned 0) → switched to full-string Levenshtein DP
  • multilingual.py segment_by_language() mislabeled "私は" (Hiragana) as "zh" because the first char "私" is CJK → rewrote to scan ALL chars in a token
  • multilingual.py detect_language() 20% threshold was too aggressive ("I love 東京" mislabeled as "zh") → bumped to 30%

Live on PyPI

pip install cortexm → https://pypi.org/project/cortexm/0.6.0/

Trusted-publish via .github/workflows/release.yml (OIDC, no API token). Workflow run: https://github.com/ssmurfgg04-gif/context-m/actions/runs/33256284241

Promises intact

✅ Always remembers · ✅ Flat cost μ=0 · ✅ Own your data · ✅ Doesn't lie · ✅ Same every time

No LLM embedder swap (HashingEmbedder stays per user instruction).

Assets 2
Loading