Capacity Planning

Capacity Planning

Tip

Value Proposition: Memory Journal MCP Server leverages local file-based storage for unmatched speed, privacy, and simplicity. It scales easily for the entire lifecycle of your projects without the overhead of cloud databases.

When integrating Memory Journal MCP Server into your enterprise development workflows, mastering its database growth mechanics allows you to effectively plan for long-term scalability and maintain optimal performance.

Memory Journal is built on top of better-sqlite3 and sqlite-vec. Because it uses a file-based database on your local disk, it easily scales to handle the lifespans of most software projects without the overhead of a dedicated database server.

---

📈 Database Size Growth

The size of your journal database depends primarily on the length of your entries, the number of tags, and whether you have Semantic Search enabled (which maintains a vector index).

Average Payload Assumptions

• Base Entry: ~1.5KB to 2.5KB of text (for verbose agent-authored entries, plus FTS5 and B-tree index overhead; see Schema → Database Size Estimates for raw row payload sizes on ~500B notes).
• Relational Metadata: Tags, revisions, and relationships add roughly 0.5KB per entry.
• Vector Embeddings (if enabled): The default Xenova/all-MiniLM-L6-v2 model produces 384-dimensional vectors. In sqlite-vec, each embedding takes about ~1.5KB of space plus chunk metadata overhead (chunk_size=256).

Expected Disk Footprint

Entry Count	Text + Metadata Size	Vector Index Size	Total Expected Disk Size
1,000	~3 MB	~2 MB	~5 MB
10,000	~30 MB	~20 MB	~50 MB
50,000	~150 MB	~100 MB	~250 MB
100,000	~300 MB	~200 MB	~500 MB

Note

Database sizes are rough estimates. Your mileage will vary depending on how verbose your AI agents are when generating journal entries. You can run a VACUUM command to reclaim unused space.

---

🧠 Memory (RAM) Requirements

Memory Journal is extremely lightweight, designed to run constantly in the background via the MCP Server protocol.

Baseline Memory Footprint

• Node.js Process: ~20 MB to 30 MB baseline.
• SQLite Page & Statement Cache: Configured with PRAGMA cache_size = -64000 (~64 MB page cache), PRAGMA temp_store = MEMORY, and a bounded 256-entry LRU prepared-statement cache.

Semantic Search Overhead

When semantic search is enabled, the server warms up the HuggingFace Transformers embedding model asynchronously in the background at startup (<1s server readiness, with an on-demand initialize() guard if queried before warmup completes).

• Model Size: Xenova/all-MiniLM-L6-v2 consumes roughly 80 MB to 100 MB of RAM when active.
• Vector Search Cache: sqlite-vec maintains small search caches.

Expected Peak RAM Usage

Feature Set	Entry Count	Peak RAM Usage
Basic FTS5	10,000	~60–100 MB (as ~64 MB SQLite page cache fills)
Basic FTS5	100,000	~90–100 MB
Semantic	10,000	~150–180 MB
Semantic	100,000	~180–200 MB

Memory limits are safely below standard IDE limits. AI context windows will bottleneck long before the Memory Journal server runs into resource starvation.

---

🗄️ When to Consider Archiving or Sharding

Memory Journal scales comfortably up to 100,000+ entries on modern SSDs. However, as the database approaches this size, you may start noticing slight latency increases during Full-Text Search (FTS5) or Vector queries.

Performance Degradation Thresholds

• < 10,000 entries: Sub-50ms searches across the board. No action needed.
• 50,000 entries: You may notice vector searches taking ~200ms-300ms. Still highly performant for AI agents.
• > 100,000 entries: FTS5 and vector searches may exceed 500ms. Depending on your context timeout settings, this could marginally slow down AI session initialization. Note that --auto-rebuild-index runs in the background on startup and consumes heavy CPU/disk I/O for several minutes on databases with >5,000 entries, so running rebuild_vector_index on-demand is preferred for large journals.

Strategy 1: Archiving (Recommended)

You can soft-delete or archive old or low-importance entries to exclude them from default search results, while keeping them recoverable.

• Set is_archived: true via update_entry or batch_update_entries (or use run_auto_prune) to hide historical entries from default queries.
• Utilize the Auto-Pruning feature (see Configuration) to automatically prune entries older than X days that fall below a certain significance threshold. Setting --archive-older-than-days, --prune-older-than-days, or --hard-delete-older-than-days (> 0) triggers auto-pruning on server startup in both stdio and http modes, whereas --auto-prune-interval schedules recurring background pruning jobs in HTTP mode.
• Use --vacuum-interval to schedule recurring PRAGMA optimize jobs in HTTP mode, memory-journal-mcp admin vacuum to run PRAGMA optimize + PRAGMA wal_checkpoint(TRUNCATE) on connection close (while clean server shutdown runs PRAGMA wal_checkpoint(TRUNCATE) on connection close), or execute a manual SQLite VACUUM command to physically reclaim disk space from permanently deleted entries.

Strategy 2: Sharding (Project Splitting)

If a journal gets excessively large, it might be holding data for too many distinct projects.

• Utilize Project Registries: Leverage PROJECT_REGISTRY to partition different projects.
• Multiple Databases: You can spin up separate instances of memory-journal-mcp pointing to different SQLite files (--db ./project-a.db, --db ./project-b.db). This guarantees zero cross-contamination and optimal performance for massive scale.

---

Next: Check Architecture or Performance.