Skip to content

Backend Synchronization Guide

Heinrich Krupp edited this page Nov 5, 2026 · 4 revisions

Backend Synchronization Guide

Overview

This guide covers the bidirectional synchronization capabilities between Cloudflare and SQLite-vec backends in MCP Memory Service. These tools enable hybrid deployment strategies combining the speed of local storage with the global availability of cloud storage.

Architecture

graph TD
    subgraph MCP_Memory_Service [MCP Memory Service]
        direction TB
        
        subgraph Local [Local Storage]
            SQLite[SQLite-vec]
        end
        
        subgraph Logic [Core]
            Sync[Sync Engine]
        end
        
        subgraph Cloud [Cloudflare Backend]
            CF[D1 + Vectorize + R2<br/>Global Distribution]
        end

        %% Connections
        SQLite <--> Sync
        Sync -->|Bidirectional Sync| CF
        CF -->|Fast Access / 5ms reads| SQLite
    end

    %% Styling
    style MCP_Memory_Service fill:none,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5
    style SQLite fill:#e1f5fe,stroke:#01579b
    style Sync fill:#fff,stroke:#333
    style CF fill:#fff4dd,stroke:#d4a017
Loading

Use Cases

1. Hybrid Cloud/Local Deployment

  • Primary: Cloudflare for global access
  • Backup: Local SQLite-vec for offline capability
  • Benefit: Resilient, always-available memory service

2. Development/Production Sync

  • Development: Local SQLite-vec for fast iteration
  • Production: Cloudflare for scalability
  • Benefit: Seamless dev→prod workflow

3. Multi-Machine Memory Sharing

  • Machine A: Local work with periodic sync
  • Machine B: Pull shared memories from cloud
  • Benefit: Consistent memory across devices

4. Disaster Recovery

  • Regular backups: Automated Cloudflare→SQLite sync
  • Recovery: Quick restore from local backup
  • Benefit: Zero data loss, minimal downtime

Installation

Prerequisites

  1. Both backends configured:

    • SQLite-vec: Default local storage
    • Cloudflare: Requires API token and resource IDs
  2. Environment files:

    # .env (Cloudflare configuration)
    CLOUDFLARE_API_TOKEN=your-token
    CLOUDFLARE_ACCOUNT_ID=your-account
    CLOUDFLARE_D1_DATABASE_ID=your-d1-id
    CLOUDFLARE_VECTORIZE_INDEX=your-index
    MCP_MEMORY_STORAGE_BACKEND=cloudflare
    
    # .env.sqlite (SQLite configuration)
    MCP_MEMORY_STORAGE_BACKEND=sqlite_vec
    MCP_MEMORY_SQLITE_PATH=/path/to/sqlite_vec.db

Sync Tools Setup

The sync utilities are located in the scripts/ directory:

# Navigate to project root
cd mcp-memory-service

# Verify sync tools are present
ls scripts/sync_memory_backends.py
ls scripts/claude_sync_commands.py
ls scripts/memory_service_manager.sh

Basic Usage

Check Sync Status

# Using main sync script
python scripts/sync_memory_backends.py --status

# Using convenience wrapper
python scripts/claude_sync_commands.py status

# Example output:
# === Memory Sync Status ===
# Cloudflare memories: 750
# SQLite-vec memories: 745
# Cloudflare configured: True
# SQLite-vec file exists: True
# Last check: 2024-01-15T10:30:00

Preview Changes (Dry Run)

Always preview before syncing:

# See what would be synced
python scripts/sync_memory_backends.py --dry-run

# Output shows:
# - Memories to add
# - Memories to skip (duplicates)
# - No actual changes made

Sync Operations

Cloudflare → SQLite (Backup)

# Backup cloud memories to local
python scripts/sync_memory_backends.py --direction cf-to-sqlite

# Or using wrapper
python scripts/claude_sync_commands.py backup

SQLite → Cloudflare (Restore)

# Restore local memories to cloud
python scripts/sync_memory_backends.py --direction sqlite-to-cf

# Or using wrapper
python scripts/claude_sync_commands.py restore

Bidirectional Sync

# Sync both directions (merge)
python scripts/sync_memory_backends.py --direction bidirectional

# Or using wrapper
python scripts/claude_sync_commands.py sync

Advanced Usage

Service Management (Linux)

The memory_service_manager.sh script provides comprehensive service management:

# Start with specific backend
./scripts/memory_service_manager.sh start-cloudflare
./scripts/memory_service_manager.sh start-sqlite

# Check status
./scripts/memory_service_manager.sh status

# Integrated sync operations
./scripts/memory_service_manager.sh sync-backup
./scripts/memory_service_manager.sh sync-restore
./scripts/memory_service_manager.sh sync-both

# Stop service
./scripts/memory_service_manager.sh stop

Automated Sync with Cron

Set up automated daily backups:

# Edit crontab
crontab -e

# Add daily backup at 2 AM
0 2 * * * cd /path/to/mcp-memory-service && python scripts/sync_memory_backends.py --direction cf-to-sqlite >> /var/log/memory-sync.log 2>&1

# Add hourly bidirectional sync
0 * * * * cd /path/to/mcp-memory-service && python scripts/sync_memory_backends.py --direction bidirectional >> /var/log/memory-sync.log 2>&1

Custom Database Paths

# Specify custom SQLite path
python scripts/sync_memory_backends.py --sqlite-path /custom/path/backup.db --status

# Verbose logging
python scripts/sync_memory_backends.py --verbose --direction bidirectional

Sync Algorithm

Deduplication Strategy

The sync engine uses content-based hashing to prevent duplicates:

  1. Content Hash: SHA256 hash of content + metadata
  2. Comparison: Check both backends for existing hashes
  3. Skip Logic: Skip if hash exists in target backend
  4. Metadata Preservation: All tags, timestamps preserved

Conflict Resolution

  • No conflicts: Different memories are merged
  • Duplicates: Skipped based on content hash
  • Timestamps: Preserved from original creation
  • Tags: All tags maintained

Performance Considerations

  • Batch Processing: Memories synced in batches
  • Caching: Embedding cache for efficiency
  • Network Optimization: Minimal API calls
  • Large Datasets: Handles 1000+ memories efficiently

Configuration Validation

Before syncing, validate your configuration:

# Run configuration validator
python scripts/validate_config.py

# Example output:
# 🔍 MCP Memory Service Configuration Validation
# ==================================================
#
# 1. Environment Configuration Check:
#    ✅ .env file has Cloudflare backend configured
#
# 2. Claude Code Global Configuration Check:
#    ✅ Found 1 Cloudflare memory configurations
#
# 3. Project-Level Configuration Check:
#    ✅ No local .mcp.json found (good - using global configuration)
#
# 4. Cloudflare Credentials Check:
#    ✅ All required Cloudflare environment variables found in .env
#
# 🎉 Configuration validation PASSED!

Troubleshooting

Common Issues

1. Sync Shows 0 Memories

Problem: One backend not initialized Solution:

# Initialize both backends
MCP_MEMORY_STORAGE_BACKEND=cloudflare uv run memory server &
sleep 5 && kill %1

MCP_MEMORY_STORAGE_BACKEND=sqlite_vec uv run memory server &
sleep 5 && kill %1

2. Authentication Errors

Problem: Invalid Cloudflare credentials Solution:

# Verify credentials in .env
cat .env | grep CLOUDFLARE

# Test with curl
curl -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
  https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/d1/database

3. Duplicate Memories After Sync

Problem: Content hash mismatch Solution:

# Run deduplication on SQLite
python scripts/find_duplicates.py --execute

# Verify sync status
python scripts/sync_memory_backends.py --status

4. Slow Sync Performance

Problem: Large dataset without optimization Solution:

# Use verbose mode to identify bottlenecks
python scripts/sync_memory_backends.py --verbose --dry-run

# Consider splitting large syncs
# First sync recent memories only

Best Practices

1. Regular Backups

  • Daily Cloudflare → SQLite backup
  • Weekly full bidirectional sync
  • Monthly verification of sync integrity

2. Pre-Sync Validation

  • Always run --dry-run first
  • Check sync status before major operations
  • Validate configuration regularly

3. Monitoring

  • Log sync operations
  • Monitor memory counts
  • Track sync duration trends

4. Testing

  • Test sync on development data first
  • Verify metadata preservation
  • Confirm search functionality post-sync

Integration with CI/CD

GitHub Actions Example

name: Memory Sync

on:
  schedule:
    - cron: '0 */6 * * *'  # Every 6 hours
  workflow_dispatch:

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          pip install uv
          uv pip install -e .

      - name: Sync memories
        env:
          CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
          CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
          CLOUDFLARE_D1_DATABASE_ID: ${{ secrets.CLOUDFLARE_D1_DATABASE_ID }}
          CLOUDFLARE_VECTORIZE_INDEX: ${{ secrets.CLOUDFLARE_VECTORIZE_INDEX }}
        run: |
          python scripts/sync_memory_backends.py --direction bidirectional

Future Enhancements

Planned improvements for sync functionality:

  1. Selective Sync: Sync by tags or time ranges
  2. Incremental Sync: Only sync changes since last run (available today via delta-sync / self-hosted hub)
  3. Conflict Resolution UI: Interactive merge tool
  4. Multi-Backend Support: Sync with ChromaDB, PostgreSQL
  5. Compression: Reduce bandwidth for large syncs
  6. Encryption: End-to-end encryption for sensitive memories

Delta-Sync (Self-Hosted Sync Hub)

For a self-hosted, event-log based alternative to the Cloudflare hybrid sync, MCP Memory Service supports delta-sync with a passive hub (ADR-0027 / ADR-0028). Spokes (clients) run periodic pull+push cycles against the hub; the hub never initiates connections. Transport is plain HTTPS, so a private network such as Tailscale works well. This mechanism is newer than the Cloudflare backup/restore scripts above and is opt-in.

Architecture

  • Hub (e.g. a small VM): one HTTP(S) node with MCP_SYNC_EVENTLOG=on. Configured with no MCP_SYNC_PEERS, so it schedules no outbound sync — a passive pivot.
  • Spoke (client): MCP_SYNC_EVENTLOG=on plus peer config. Each tick it pulls the hub's event log (the delta) and pushes its own queued events.

Configuration

Hub (.env):

MCP_MEMORY_STORAGE_BACKEND=sqlite_vec
MCP_HTTPS_ENABLED=true
MCP_HTTP_HOST=0.0.0.0
MCP_HTTP_PORT=8020
MCP_SSL_CERT_FILE=/path/to/certs/cert.pem
MCP_SSL_KEY_FILE=/path/to/certs/key.pem
MCP_SYNC_EVENTLOG=on

Spoke (.env):

MCP_SYNC_EVENTLOG=on
MCP_SYNC_SCHEDULE=15m              # pull+push every 15 minutes (also 6h, 90s …); unset/disabled = off
MCP_SYNC_PEERS=hub
MCP_SYNC_PEER_URL=https://hub.example.com:8020
MCP_SYNC_PEER_API_KEY="<the HUB's MCP_API_KEY>"
# MCP_SYNC_BASIC_USER=… ; MCP_SYNC_BASIC_PASS=…          # alternative auth
# MCP_SYNC_PUSH_ALLOWED_AGENTS=laptop,desktop            # hub: restrict which agents may push

MCP_SYNC_PEER_API_KEY is not MCP_API_KEY. On a spoke, MCP_API_KEY also enables auth enforcement for the spoke's own /mcp endpoint — setting it to the hub's value would break every local MCP client. Keep the hub credential in the dedicated variable.

Hub service

Run the modern, package-managed entrypoint under systemd (EnvironmentFile=.env supplies the variables):

[Service]
WorkingDirectory=/opt/mcp-memory-service
EnvironmentFile=/opt/mcp-memory-service/.env
ExecStart=/opt/mcp-memory-service/.venv/bin/memory launch --foreground

Verification

# Hub reachable + healthy (from the spoke or the hub itself)
curl -sk -H "X-API-Key: $KEY" https://hub.example.com:8020/api/health/detailed

# Spoke sync cycle — look for "Delta-sync <peer>: pulled N applied, pushed M"
journalctl -u mcp-memory-http.service | grep -i "delta-sync"

A cycle logs a line such as Delta-sync hub: pulled 16 applied, pushed 5.

Notes & known limitations

  • memory launch reads MCP_HTTP_HOST/MCP_HTTP_PORT from the process environment only (defaults 127.0.0.1:8000); under systemd the EnvironmentFile supplies them. It honours MCP_SSL_CERT_FILE/MCP_SSL_KEY_FILE and only generates a development certificate when neither is set.
  • systemd precedence: an EnvironmentFile overrides a drop-in Environment= directive.
  • The wheel ships only src/mcp_memory_service; scripts/ is intentionally not packaged. scripts/server/run_http_server.py is the legacy entrypoint — memory launch is the package-managed equivalent.
  • Observability gap: unlike the hybrid/Cloudflare sync, delta-sync currently has no Web GUI status panel, no MCP_SYNC_* entry in Settings, and does not feed the dashboard's live refresh. Tracked upstream (see Related Documentation).

Related Documentation

Support

For issues with sync functionality:

  1. Check configuration with python scripts/validate_config.py
  2. Review logs in /tmp/memory-sync.log
  3. Open an issue with sync debug output
  4. Join discussions for community support

Last updated: January 2025 Sync tools version: 1.0.0

Clone this wiki locally