v2.5.0 - Domain Taxonomy Expansion, Expressive Graph Relations & Batched Consolidation

What's Changed

• Domain Taxonomy Expansion & De-concentration: Expanded CANONICAL_LEARNING_CATEGORIES with dedicated domains (ux, dev, infra, and media) to relieve the catch-all architecture bucket. Aligned extraction prompt and runtime alias normalization (ui_ux -> ux).
• High-Precision Knowledge Graph Relations: Broadened CANONICAL_RELATIONS from 26 to 32 expressive link types (documents, configures, applies_to, resolves, targets, supports) along with bidirectional inverse mappings (documented_by, configured_by, resolved_by, targeted_by, supported_by, used_by, stored_at, stored_in). Generic related_to link density reduced by >43%.
• Adaptive Batched Semantic Memory Consolidation: Refactored consolidate_memories from monolithic single-prompt serialization into category-chunked batches (≤ 25 facts per inference call), preventing CLI subprocess payload and timeout failures on large memory stores.
• Continuous FTS & Trigram Synchronization: Automated SQLite triggers (trg_memories_trigram_ai/au/ad) to keep the substring trigram virtual table synchronized with memories, plus full rebuild integration in rebuild_fts.
• Retrieval & Ingestion Hardening: Decoupled static persistent preferences and safety guardrails from graph expansion to guarantee deterministic zero-hop system prompt injection. Hardened automated calendar fact pruning and point-in-time episode lifecycle resolution.

Full Changelog: v2.4.0...v2.5.0

v2.4.0 - Self-Healing Vector Index Sync & Queue Partitioning

What's Changed

• Self-Healing Vector Index Synchronization: Implemented outbox pattern with fencing token to ensure 100% data consistency between SQLite FTS5 facts and sqlite-vec vector chunks during concurrent mutations.
• Resilient Queue Splitting: Over-attempted batches on debounce retry are now automatically partitioned instead of replaying the entire block, preventing queue starvation.
• Turn Enqueue Capping: Added safety bounds to turn payload sizing at ingestion to prevent toxic turn payloads from poisoning worker claim batches.
• Taxonomy Normalization: Automatic canonical mapping of relates_to variants to standard related_to relations.
• PR #2 Contribution: Full verification and merge of PR #2 by @tuanhung303.

Full Changelog: v2.3.0...v2.4.0

v2.3.0 - Hardening, WAL-Safe Queue Reliability & Native CLI Fallback

Highlights of v2.3.0

• WAL-Safe Storage & Queue Reliability: Atomic batch claims with durable batch receipts and recoverable expiring leases in turn_queue.db.
• Concurrency Hardening: Serialized schema bootstrap, entity revision tracking, and generation fencing preventing stale overwrites across restores.
• Native CLI Fallback: Bounded agy --print fallback for background extraction when AGY_MEMORY_INFERENCE_URL is unset, with automated JSON markdown block extraction and slash-command retry.
• Tolerant Graph Linking: Non-fatal relationship validation skips unknown or hallucinated endpoints with a warning instead of failing the extraction transaction (AGY_MEMORY_STRICT_GRAPH=false).
• Multi-User Dashboard Resilience: Shared maintenance locking (LOCK_SH) permitting seamless cross-user profile inspection (e.g. ubuntu / guest) without permission errors.
• Telegram First-Turn Routing: Session resolution in scripts/auto_sync_hook.py properly detects in-flight chats so notifications route reliably from the very first turn.
• Comprehensive Test Coverage: Expanded test suite to 193 unit and concurrency tests passing with full parity.

v2.2.0 - Semantic Recall & In-Process Hybrid Search

Highlights of v2.2.0

• In-Process Vector Extension via sqlite-vec: Native C-extension loaded directly into SQLite with zero daemon overhead or RAM bloat.
• Dense Multilingual Embeddings via FastEmbed: ONNX-based inference using sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 (384 dimensions, ~9ms latency).
• Reciprocal Rank Fusion (RRF): Combines BM25 lexical precision with dense semantic cosine similarity in search_memory and the real-time debug web dashboard.
• Zero Latency Impact on CLI Prompts: Pre-invocation prefetch remains strictly under 2ms using FTS5 + Trigram subtoken matching.
• Background Vector Lifecycle: Auto-indexes new facts, episodes, and learnings via Calm Memory Queue Worker; automatic SQLite triggers handle cascade deletion.
• Reindexing Utility: Included standalone scripts/reindex_vectors.py for database vector re-generation.
• Testing: 49/49 unit and integration tests passing.

v2.1.0 - Quality-First Extraction, Taxonomy Normalization & Graph Pruning

What's Changed in v2.1.0

• 🎯 Quality-First Extraction & Strict Learnings Filtering:

• Overhauled the memory extraction prompt (sync_turn): strictly excludes one-time bugfixes, debugging sessions, codebase internals, temporary configs, and UI layout tweaks.
• Implemented the Litmus test rule: "Would this insight help handle a SIMILAR situation in the future?"
• Re-focused experiential learnings exclusively on reusable heuristics, communication styles, and personal behavioral rules.

• 📐 Canonical Taxonomies & Runtime Normalization:

• Established a closed taxonomy for Entity Links with exactly 26 canonical relation types (hosted_on, runs_on, depends_on, monitors, part_of, etc.).
• Enforced strict category taxonomies across all 4 layers (20 Fact categories, 12 Learning categories, 13 Episode topics).
• Integrated runtime normalization (_normalize_category) and relation fallback mapping directly into the extraction pipeline.

• 🕸️ Full Entity Graph Pruning:

• Enhanced orphan link pruning (prune_orphan_links) to remove links whenever either endpoint (source or target) is missing or deleted.
• Cleaned up dangling graph pointers resulting from fact consolidation and entity merges.

• 🧹 Maintenance, Web Dashboard & DevSecOps:

• Integrated full taxonomy normalization and graph pruning into optimize_db and the dashboard endpoint POST /api/optimize.
• Rebuilt SQLite FTS5 indices and ran database VACUUM.
• Comprehensive DevSecOps pipeline validation: 0 secrets detected via Gitleaks, 45 of 45 unit tests passing.

v2.0.0 - Calm Memory, Queue Worker & Semantic Consolidation

What's Changed in v2.0.0

• 🧠 Calm Memory & Asynchronous Turn Queue Worker:
• Decoupled LLM memory extraction from the active conversation path: stop-hook writes turns to local SQLite queue in < 1ms with zero conversation latency.
• Added background worker daemon (memory_worker.py) with robust session-debouncing (5 min idle / 15 min max-timeout).
• Consolidated multi-turn sessions into a single concise batch LLM extraction pass.

• 🔍 German Compound Sub-Token Expansion (Hybrid FTS5):
• Integrated morphological sub-word decomposition for German composite nouns (e.g. Hundeversicherung ➔ hunde + versicherung).
• Boosts SQLite FTS5 BM25 recall without heavy vector database overhead or embedding models.

• 🔄 Semantic Consolidation & Audit Engine:
• Added automated duplicate fact merging and outdated knowledge pruning with rollback-safe JSON audit logging.
• Automated snapshot retention pruning.

• 📱 Dynamic Telegram Notification & Topic Routing:
• Ultra-compact visual change alerts with emoji indicators.
• Dynamic routing to active chat/topic or report channel (--chat-id / --reports).
• Built-in loop prevention (AGY_INTERNAL_INVOCATION guard).

• 🧪 360° Test & Benchmark Suite:
• Expanded unit test suite to 34 tests covering queue operations, debounce timeouts, notification formatting, and consolidation (100% passing).

v1.2.0 - Episode Aging & Entity Graph Linking

What's Changed in v1.2.0

• ⏳ Automatic Episode Aging & State Decay:
  • Automatically transitions episodes active ➔ cooling (after 30 days) ➔ historic (after 90 days).
  • Prefetch relevance ordering prioritizes active over cooling and historic.
  • Added CLI command agy_memory.py age-episodes.
  • optimize now automatically runs lifecycle decay.
• 🔗 Entity Graph & Relationship Linking:
  • Added entity_links table with bidirectional indexing & FTS5 search.
  • Automated 1-hop multi-entity expansion during prefetch (e.g. querying a service automatically pulls its linked host facts).
  • Added CLI commands link, unlink, and FastMCP tool link_entities_mcp.
  • sync_turn now extracts and stores entity relationships automatically.
• 🧪 Test Suite Expansion: Added 4 new integration tests (28 tests total, 100% passing).

v1.1.0 - Multi-Layer Memory & SQLite Robustness

What's Changed

• Multi-Layer Cognitive Architecture: Added Layer 2 (Narrative/Episodic dossiers) and Layer 3 (Experiential/Learnings store).
• FastMCP Server: Migrated MCP tool definitions to FastMCP standard for seamless Antigravity integration.
• SQLite Robustness: Context manager for database sessions, automated FTS5 synchronization triggers, and optimize/compact subcommands.
• Race Condition Protection: Added fcntl.flock file lock to sync_turn to avoid concurrent execution spikes.
• Test Suite & CI: Added 24 automated unit tests and GitHub Actions CI workflow across Python 3.10-3.12.
• Version Flag: Added --version / -v CLI support.

v1.0.0 - Initial Release

Full Changelog: https://github.com/sbolten/agy-memory-engine/commits/v1.0.0