Maintenance and Optimization

Database Maintenance & Optimization

Tip

Value Proposition: Memory Journal is built for the long haul. Its autonomous pruning and heuristic scoring keep your project's context window lean, performant, and relevant over months of development, preventing token bloat without losing critical history.

The memory journal is designed to operate continuously for months or years without failure. To prevent unbounded disk growth and context window saturation, the journal features a sophisticated maintenance architecture powered by heuristic scoring, an autonomous pruning engine, and interactive optimization workflows.

---

1. The Heuristic Scoring System

Entries in the journal are assigned heuristic scores by agents to determine their value, confidence, and blast radius. These scores dictate how the entries are filtered, retrieved, and eventually pruned.

Metric	Range	Description
impact_score	0.0 - 1.0	Blast Radius. Measures the systemic impact of an entry. For example, 0.1 represents a minor typo fix or simple documentation update, while 0.9 represents a core architectural shift or critical security patch.
trust_score	0.0 - 1.0	Confidence. Measures how rigorously an entry has been verified. Unverified drafts or assumed implementations sit at a baseline of 0.3 - 0.5. Fully verified code (e.g. passing test suites, peer review) scores 0.8 - 1.0. Failed validations or hallucinations drop to 0.0 - 0.2.

Status Markers

In addition to decimal scores, entries carry explicit categorical statuses:

• validation_status: Marks the state of the entry's claims (passed, none, or failed).
• significance_type: Distinct structural markers used to categorize highly critical entries (milestone, breakthrough, decision, architecture, lesson_learned, blocker_resolved, release, security).

---

2. Autonomous Pruning Engine

To maintain high performance, the journal uses an automated garbage collection system.

Importance Calculation

The SQLite adapter (src/database/sqlite-adapter/entries/importance.ts) dynamically computes a live importance score for each entry using four weighted base components:

1. Significance / Impact (30%): Math.max(impact_score, significance_type ? 0.5 : 0.0) * 0.30.
2. Relationships (35%): Math.min(rel_count / 5, 1.0) * 0.35, reaching its maximum at 5 linked entries.
3. Causal Relationships (20%): Math.min(causal_count / 3, 1.0) * 0.20 for blocked_by, resolved, and caused links, reaching its maximum at 3 causal links.
4. Recency (15%): Math.max(0, 1 - daysSince / 90) * 0.15, which provides up to a +0.15 boost for newly created entries and decays linearly over a 90-day window.

The sum of these four weighted components (baseScore) is then multiplied by anchorMultiplier * trustMultiplier:

• anchorMultiplier: 0.1 if any linked code anchor is invalidated, 0.8 if stale, or 1.0 otherwise.
• trustMultiplier: 0.5 + trust_score + validationMultiplier (where validation_status: "passed" adds +0.5 and "failed" subtracts -0.4).

Automated Garbage Collection

The run_auto_prune administrative tool scans the database for entries that have decayed beneath acceptable importance thresholds.

• MCP Tool Parameters & Return Fields (run_auto_prune): Accepts dry_run (boolean), archive_older_than_days (number), prune_older_than_days (number), hard_delete_older_than_days (number), and importance_threshold (number). When omitted, thresholds fall back to the server's configured lifecycle options (archiveOlderThanDays, pruneOlderThanDays, hardDeleteOlderThanDays, pruneImportanceThreshold), or default to 0 (disabled) for age thresholds and 0.15 for importance_threshold. Returns { success: boolean, archivedCount: number, prunedCount: number, hardDeletedCount: number, backupFile?: string, dryRun: boolean } (when dry_run is false, runAutoPrune automatically creates a pre-prune-backup-<timestamp>.db snapshot and retains the 5 most recent backups via deleteOldBackups(5)).
• Scheduling & CLI Flags: Startup auto-pruning runs automatically on server startup in both stdio and http modes whenever any *-older-than-days > 0 (--archive-older-than-days, --prune-older-than-days, or --hard-delete-older-than-days) is configured, whereas recurring scheduled pruning via --auto-prune-interval (AUTO_PRUNE_INTERVAL_MINUTES) runs in HTTP transport mode only. Pruning can also be invoked on-demand via memory-journal-mcp admin prune [--dry-run] [--exclude-tags <tags>]. Additional CLI/startup pruning flags include --prune-importance-threshold, --prune-dry-run, and --prune-exclude-tags (PRUNE_EXCLUDE_TAGS / CLI --exclude-tags).
• Safety Locks: Startup/scheduled pruning and the CLI admin prune command support tag-based exclusion (--prune-exclude-tags / PRUNE_EXCLUDE_TAGS / --exclude-tags), whereas the run_auto_prune MCP tool filters by age and importance_threshold (protecting entries scoring at or above importance_threshold, including entries with a defined significance_type or sufficient impact_score under default trust/threshold settings, as well as exclude_tags when configured via startup/CLI pruning). In the interactive optimize-journal workflow, db.getOrphanCount('*') counts unlinked entries whose computed importance score is strictly below 0.2, naturally excluding entries with a defined significance_type or high impact_score/trust_score from low-importance orphan candidates.

---

3. Session Summary Filtering

Agents are instructed to generate Session Summaries (entry_type: "retrospective") at the conclusion of work bursts using the session-summary prompt. The prompt structures summaries around ## Accomplished, ## Unfinished / Blocked, and ## Context for Next Session headers (plus a ## Commits section when validated via the direct MCP tools CreateEntrySchema / TeamCreateEntrySchema), alongside the auto_context.commits payload to securely bind session activity to the knowledge graph.

When retrieving historical context to construct these retrospectives, agents execute get_recent_entries utilizing the sort_by: "importance" parameter. Because of the heuristic scoring rules, this guarantees that critical architectural context (impact_score: 0.9) and successfully verified implementations (validation_status: "passed") surface to the absolute top of the context window, bypassing trivial or unverified scratchpad notes.

---

4. The Optimize Journal & CMS Workflows

optimize-journal (CMS-02 NREM Consolidation)

For exhaustive database maintenance, the server provides the /mcp:memory-journal-mcp:optimize-journal workflow prompt (src/handlers/prompts/optimizer.ts). It accepts six optional arguments:

• mode: 'quick' | 'targeted' | 'full' (default: 'quick')
• targetDimension: 'unassigned' | 'relationships' | 'scores' | 'tags' | 'all' (default: 'all')
• project: Target project name or number (omit to optimize all projects globally)
• limit: Entry processing cap (0 = unlimited across all history, or 1–50000, default: 500 or 0 in full/autoRun mode)
• lookback: Number of recent unverified entries to inspect (1–2000, default: 10)
• autoRun: "true" | "false" (default: "false") — execute all phases without interactive stopping

The workflow is structured into three progressive options:

1. Option A (Quick Health & Priority Workspace Unassigned Audit — Executed Automatically): Creates a pre-flight safety backup (backup_journal + cleanup_backups({ keep_count: 15 })) and scans all workspaces (workspace_id: '*') to immediately migrate any unassigned or 'default' entries to their registered project, accompanied by a root-cause investigation.
2. Option B (Ground-Truth Verification, Staleness Sweep & Targeted Quality Audit — Executed Automatically): Cross-checks recent entries against Git/GitHub ground truth, runs mj.git.checkEntryStaleness({ entry_id: id, workspace_id: '*' }) across sampled entries, updates stale code anchors via mj.core.markAnchorsStale, and audits the selected targetDimension (unassigned, relationships, scores, tags, or all).
3. Option C (Comprehensive Multi-Agent Domain Audit, Episodic Compaction & Noise Pruning — Executed Automatically in full / autoRun Mode): Partitions entries by project domain and performs 5-dimension single-pass optimization alongside CMS-02 NREM Episodic-to-Semantic Compaction (when >10 granular test_entry or incremental development_note entries older than 14 days exist for the same feature/session, synthesizing a semantic retrospective or project_decision entry linked via evolves_from and archiving the raw fragments with is_archived: true). Operations are logged with entry_type: 'maintenance' and tags ['cms', 'cms-02-nrem-consolidation', 'database-optimizer', 'maintenance'].

cms-rem-supervisor (CMS-04 REM Meta-Cognitive Supervisor)

Complementing optimize-journal, the /mcp:memory-journal-mcp:cms-rem-supervisor prompt (src/handlers/prompts/cms-supervisor.ts) runs the REM Meta-Cognitive Supervisor & Self-Adaptation workflow (CMS-04) with optional arguments lookbackHours (1–168, default 24) and autoRestore ("true" | "false", default "true"):

• Phase A (Post-NREM Supervisor Audit & Safety Rollback): Audits recent CMS-02 maintenance runs and soft-deleted entries, automatically restoring (mj.admin.restoreEntry) any high-value entry (significanceType !== null, impactScore >= 0.5, or active relationships) that was erroneously soft-deleted, and verifies zero unassigned entries and vector index parity.
• Phase B (Cross-Run Experience & Telemetry Mining): Mines logs/agent-issues.jsonl, logs/mcp-audit.jsonl, and recent ## Unfinished / Blocked sections to formulate targeted schema/healer/prompt adaptations.
• Phase C (Batched Adaptation & Dirty-Tree Conditional Single-Pass Validation Gate): Applies approved fixes and runs the 5-stage validation gate (lint, typecheck, build, vitest, playwright) strictly once only if working tree files were modified.
• Phase D (Telemetry Cleanup & Supervisor Journal Entry): Logs the completed supervisor run with tags ['cms', 'cms-04-rem-supervisor', 'meta-learning', 'maintenance'].

---

5. SQLite Engine & Statement Cache Optimizations

• Prepared Statement LRU Cache: NativeConnectionManager maintains a 256-entry FIFO/LRU prepared statement cache (prepare()), avoiding repeated SQL compilation on hot paths (createEntry, searchEntries, getRecentEntries, calculateImportance).
• PRAGMA Tuning: Concurrency, referential integrity, and read/write throughput are optimized via journal_mode = WAL, synchronous = NORMAL, foreign_keys = ON, cache_size = -64000 (64MB), temp_store = MEMORY, and busy_timeout = 5000.
• Virtual Table Query Planner Pinning: FTS5 (fts_content) queries use CROSS JOIN memory_journal to pin virtual table index driving, while sqlite-vec (vec_embeddings) queries drive from the vec0 table first using INNER JOIN memory_journal with k = limit * 4 over-fetching to filter out soft-deleted entries.
• Single-Transaction Mutations & Batched Importance: Write operations execute inside a single better-sqlite3 synchronous transaction, and search/list hydration computes importance scores in a single batched pass (attachImportanceScoresToEntries / getEntriesByIdsWithImportance).

Code Mode Batching

Instead of repeatedly calling search_entries over the top-level MCP tool transport, the workflow instructs the AI to utilize Code Mode (mj_execute_code). The JavaScript snippet executes inside an isolated worker_threads + vm.createContext sandbox and communicates with the host via an allowlisted MessageChannel RPC bridge (mj.* API). This keeps all intermediate loop data and filtering logic inside the sandbox so only the final compacted summary is returned to the LLM context window.

Orphan Detection & Graph Hygiene

The optimizer tracks global orphan count and density via db.getOrphanCount('*') (entries with 0 active relationships and computed importance < 0.2). In Option B and Option C, orphan entries are linked to their parent specifications, commits, or session summaries via mj.relationships.linkEntries, while zero-value transient noise (significanceType === null and unanchored) is safely soft-deleted via mj.admin.deleteEntry or mj.admin.batchDeleteEntries.