Multiple Memory Database Instances with Runtime Switching

[doobidoo]

[Feature Request] Multiple Memory Database Instances with Runtime Switching

Problem Statement

Currently, MCP Memory Service uses a single memory database for all contexts. This creates challenges for users who need:

1. Project Isolation - Keeping memories separate per codebase to avoid cross-contamination
2. Privacy/Security - Separating work vs personal vs sensitive data
3. Subject Organization - Organizing memories by topic (coding, research, documentation) rather than chronologically
4. Safe Testing - Experimenting without polluting production data

Real-world scenario: You're working on three projects simultaneously - a work project, a personal open-source project, and documentation writing. Currently, all memories are mixed together, making it hard to:

• Find project-specific context
• Maintain privacy boundaries
• Test new features without affecting production memories
• Share project-specific knowledge without exposing other work

Proposed Solution

Add support for multiple named memory database instances with runtime switching via MCP tools.

Core Features

1. Dynamic Database Creation

# Create new instances on demand
> create_database(name="work", backend="hybrid", description="Work memories")
Created database 'work' at ~/.local/share/mcp-memory/work.db

> create_database(name="personal", backend="sqlite_vec")
Created database 'personal' at ~/.local/share/mcp-memory/personal.db

2. Runtime Switching (No Restart Required)

# Switch between databases instantly
> switch_database("work")
Switched to database 'work' (hybrid, 230 memories)

> store_memory("Important work decision", {"tags": ["architecture"]})
Stored in database 'work'

> switch_database("personal")
Switched to database 'personal' (sqlite_vec, 150 memories)

> retrieve_memory("work decision")
No results (correct - we're in personal database)

3. Instance Management

# List all instances
> list_databases()
✓ work (hybrid) - 230 memories - Active
  personal (sqlite_vec) - 150 memories
  test (sqlite_vec) - 45 memories

# Get current active database
> get_active_database()
{
  "name": "work",
  "backend": "hybrid",
  "memory_count": 230,
  "path": "~/.local/share/mcp-memory/work.db"
}

# Export memories between instances
> export_database(source="work", target="personal", tag_filter=["reference"])
Exported 15 memories from 'work' to 'personal'

Architecture Design

Database Registry

• Central registry file: ~/.local/share/mcp-memory/database_registry.json
• Tracks all instances, active instance, metadata
• Persistent across sessions

Storage Backend Support

Hybrid Approach (Recommended):

• Multiple SQLite instances (one per named database)
• Optional shared Cloudflare backend for cloud sync
• Tag-based isolation: Each memory tagged with instance_name
• Filtered retrieval ensures data isolation

Zero Breaking Changes

• Existing single database becomes "default" instance
• All current functionality works unchanged
• Opt-in feature - users can ignore it entirely

Use Cases

1. Multi-Project Developer

# Work on different projects with isolated contexts
switch_database("project-x")  # Focus on project X
store_memory("API endpoint design decision")

switch_database("project-y")  # Switch to project Y
retrieve_memory("API design")  # Won't find project X decision

2. Privacy-Conscious User

# Keep work and personal completely separate
create_database("work", backend="hybrid", cloudflare_sync=true)   # Synced
create_database("personal", backend="sqlite_vec")  # Local only, never synced

3. Safe Experimentation

# Test consolidation features without risk
create_database("test", backend="sqlite_vec")
switch_database("test")
# ... experiment safely ...
delete_database("test", confirm=true)  # Clean up

4. Subject-Based Organization

create_database("coding", description="Programming knowledge")
create_database("research", description="Research notes")
create_database("documentation", description="Writing references")

# Organize memories by type rather than time

Implementation Roadmap

Phase 1: Core Infrastructure (v8.51.0)

• Database registry system
• Instance creation/deletion
• Basic MCP tools (list_databases, switch_database, create_database)
• Backward-compatible migration

Phase 2: Advanced Features (v8.52.0)

• Export/import between instances
• Instance statistics
• Cloudflare tag-based isolation

Phase 3: Web UI Support (v9.0.0)

• Instance selector dropdown
• Instance management page
• Per-instance search/CRUD

Phase 4: Context-Aware Switching (v9.1.0)

• Auto-detect instance based on git repo
• Directory-based switching
• Time-based switching (work hours vs personal)

Technical Considerations

Performance

• Lazy loading of instances (only initialize on first use)
• Cache management per instance
• Max active instances limit to prevent memory bloat

Security

• File system isolation via SQLite files
• Tag-based soft isolation for Cloudflare
• Optional: Separate Vectorize indexes for sensitive instances

Cloudflare Integration

• Shared Cloudflare backend across instances (Phase 1)
• Each memory tagged with instance_name for filtering
• Alternative: Separate Vectorize indexes (Phase 2+)

Questions for Community

1. Default Instance Naming: Should the default instance be called "default", "main", or allow user customization?

2. Cross-Instance Search: Should we support searching across all databases simultaneously? Or keep instances completely isolated?

3. Instance Limits: Soft limit with warnings (e.g., 10 instances) or hard limit (e.g., 50)?

4. Auto-Switching: Would automatic instance switching based on context (git repo, directory) be useful? Or too "magical"?

5. Web UI Priority: Should we prioritize web UI support (Phase 3) or keep it CLI-first?

6. Cloudflare Strategy: Tag-based filtering (simpler, shared resources) vs separate Vectorize indexes (better isolation, more resources)?

Feedback Requested

• Use Cases: What scenarios would benefit most from this feature?
• UX Concerns: Any potential confusion points for users?
• Priority: How important is this compared to other features?
• Alternatives: Are there simpler approaches to achieve similar goals?

Related Issues

• #XXX - User request for project isolation
• #XXX - Privacy concerns with mixed work/personal data
• #XXX - Testing consolidation features safely

---

Technical Implementation Plan: See detailed implementation plan for complete architecture and code changes.

Estimated Timeline: 4-6 releases over 2-3 months
Breaking Changes: None (fully backward compatible)
Migration Required: Automatic (existing database becomes "default" instance)

[doobidoo]

The use cases here are valid and the isolation problem is real. Current state and near-term path:

What works today — the MCP_MEMORY_PATH environment variable already points to a specific database file. You can run separate server instances on different ports, each with its own MCP_MEMORY_PATH, and configure whichever client you want to use per-project context. It is not runtime-switchable from within a session, but project-level isolation works.

Runtime switching — not currently supported, and adding it cleanly requires some design work (what happens to in-flight requests when you switch? Does the embedding model need to match across databases?). It is on the long-term roadmap but not scheduled.

Tag-based separation — for many workflows, tagging memories with a project or context name (e.g. work, personal, project:acme) is a lighter-weight alternative that avoids running multiple instances. The search_by_tags tool with AND semantics gives you effective per-project recall without isolation at the storage level.

If you need hard isolation (security / privacy boundary), separate server instances per MCP_MEMORY_PATH is the right approach today.

[doobidoo]

Closing as resolved. Summary of the current stance (see the comment above for detail):

• Hard isolation today: run separate server instances, each with its own MCP_MEMORY_PATH — project/privacy boundaries work now.
• Soft separation: tag memories per context (project:acme, work, personal) and filter with tag search — no extra instances needed.
• Runtime switching within a session: not supported and not scheduled; a clean design has open questions (in-flight requests, embedding-model parity across DBs). It stays on the long-term roadmap.

The workarounds cover the concrete use cases raised here, so I'm marking this resolved. If demand for in-session runtime switching grows, please open a focused issue referencing this discussion and we can revisit the design.