Repository navigation
Backend Synchronization Guide
This guide covers the bidirectional synchronization capabilities between Cloudflare and SQLite-vec backends in MCP Memory Service. These tools enable hybrid deployment strategies combining the speed of local storage with the global availability of cloud storage.
graph TD
subgraph MCP_Memory_Service [MCP Memory Service]
direction TB
subgraph Local [Local Storage]
SQLite[SQLite-vec]
end
subgraph Logic [Core]
Sync[Sync Engine]
end
subgraph Cloud [Cloudflare Backend]
CF[D1 + Vectorize + R2<br/>Global Distribution]
end
%% Connections
SQLite <--> Sync
Sync -->|Bidirectional Sync| CF
CF -->|Fast Access / 5ms reads| SQLite
end
%% Styling
style MCP_Memory_Service fill:none,stroke:#333,stroke-width:2px,stroke-dasharray: 5 5
style SQLite fill:#e1f5fe,stroke:#01579b
style Sync fill:#fff,stroke:#333
style CF fill:#fff4dd,stroke:#d4a017
- Primary: Cloudflare for global access
- Backup: Local SQLite-vec for offline capability
- Benefit: Resilient, always-available memory service
- Development: Local SQLite-vec for fast iteration
- Production: Cloudflare for scalability
- Benefit: Seamless dev→prod workflow
- Machine A: Local work with periodic sync
- Machine B: Pull shared memories from cloud
- Benefit: Consistent memory across devices
- Regular backups: Automated Cloudflare→SQLite sync
- Recovery: Quick restore from local backup
- Benefit: Zero data loss, minimal downtime
-
Both backends configured:
- SQLite-vec: Default local storage
- Cloudflare: Requires API token and resource IDs
-
Environment files:
# .env (Cloudflare configuration) CLOUDFLARE_API_TOKEN=your-token CLOUDFLARE_ACCOUNT_ID=your-account CLOUDFLARE_D1_DATABASE_ID=your-d1-id CLOUDFLARE_VECTORIZE_INDEX=your-index MCP_MEMORY_STORAGE_BACKEND=cloudflare # .env.sqlite (SQLite configuration) MCP_MEMORY_STORAGE_BACKEND=sqlite_vec MCP_MEMORY_SQLITE_PATH=/path/to/sqlite_vec.db
The sync utilities are located in the scripts/ directory:
# Navigate to project root
cd mcp-memory-service
# Verify sync tools are present
ls scripts/sync_memory_backends.py
ls scripts/claude_sync_commands.py
ls scripts/memory_service_manager.sh# Using main sync script
python scripts/sync_memory_backends.py --status
# Using convenience wrapper
python scripts/claude_sync_commands.py status
# Example output:
# === Memory Sync Status ===
# Cloudflare memories: 750
# SQLite-vec memories: 745
# Cloudflare configured: True
# SQLite-vec file exists: True
# Last check: 2024-01-15T10:30:00Always preview before syncing:
# See what would be synced
python scripts/sync_memory_backends.py --dry-run
# Output shows:
# - Memories to add
# - Memories to skip (duplicates)
# - No actual changes made# Backup cloud memories to local
python scripts/sync_memory_backends.py --direction cf-to-sqlite
# Or using wrapper
python scripts/claude_sync_commands.py backup# Restore local memories to cloud
python scripts/sync_memory_backends.py --direction sqlite-to-cf
# Or using wrapper
python scripts/claude_sync_commands.py restore# Sync both directions (merge)
python scripts/sync_memory_backends.py --direction bidirectional
# Or using wrapper
python scripts/claude_sync_commands.py syncThe memory_service_manager.sh script provides comprehensive service management:
# Start with specific backend
./scripts/memory_service_manager.sh start-cloudflare
./scripts/memory_service_manager.sh start-sqlite
# Check status
./scripts/memory_service_manager.sh status
# Integrated sync operations
./scripts/memory_service_manager.sh sync-backup
./scripts/memory_service_manager.sh sync-restore
./scripts/memory_service_manager.sh sync-both
# Stop service
./scripts/memory_service_manager.sh stopSet up automated daily backups:
# Edit crontab
crontab -e
# Add daily backup at 2 AM
0 2 * * * cd /path/to/mcp-memory-service && python scripts/sync_memory_backends.py --direction cf-to-sqlite >> /var/log/memory-sync.log 2>&1
# Add hourly bidirectional sync
0 * * * * cd /path/to/mcp-memory-service && python scripts/sync_memory_backends.py --direction bidirectional >> /var/log/memory-sync.log 2>&1# Specify custom SQLite path
python scripts/sync_memory_backends.py --sqlite-path /custom/path/backup.db --status
# Verbose logging
python scripts/sync_memory_backends.py --verbose --direction bidirectionalThe sync engine uses content-based hashing to prevent duplicates:
- Content Hash: SHA256 hash of content + metadata
- Comparison: Check both backends for existing hashes
- Skip Logic: Skip if hash exists in target backend
- Metadata Preservation: All tags, timestamps preserved
- No conflicts: Different memories are merged
- Duplicates: Skipped based on content hash
- Timestamps: Preserved from original creation
- Tags: All tags maintained
- Batch Processing: Memories synced in batches
- Caching: Embedding cache for efficiency
- Network Optimization: Minimal API calls
- Large Datasets: Handles 1000+ memories efficiently
Before syncing, validate your configuration:
# Run configuration validator
python scripts/validate_config.py
# Example output:
# 🔍 MCP Memory Service Configuration Validation
# ==================================================
#
# 1. Environment Configuration Check:
# ✅ .env file has Cloudflare backend configured
#
# 2. Claude Code Global Configuration Check:
# ✅ Found 1 Cloudflare memory configurations
#
# 3. Project-Level Configuration Check:
# ✅ No local .mcp.json found (good - using global configuration)
#
# 4. Cloudflare Credentials Check:
# ✅ All required Cloudflare environment variables found in .env
#
# 🎉 Configuration validation PASSED!Problem: One backend not initialized Solution:
# Initialize both backends
MCP_MEMORY_STORAGE_BACKEND=cloudflare uv run memory server &
sleep 5 && kill %1
MCP_MEMORY_STORAGE_BACKEND=sqlite_vec uv run memory server &
sleep 5 && kill %1Problem: Invalid Cloudflare credentials Solution:
# Verify credentials in .env
cat .env | grep CLOUDFLARE
# Test with curl
curl -H "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \
https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/d1/databaseProblem: Content hash mismatch Solution:
# Run deduplication on SQLite
python scripts/find_duplicates.py --execute
# Verify sync status
python scripts/sync_memory_backends.py --statusProblem: Large dataset without optimization Solution:
# Use verbose mode to identify bottlenecks
python scripts/sync_memory_backends.py --verbose --dry-run
# Consider splitting large syncs
# First sync recent memories only- Daily Cloudflare → SQLite backup
- Weekly full bidirectional sync
- Monthly verification of sync integrity
- Always run
--dry-runfirst - Check sync status before major operations
- Validate configuration regularly
- Log sync operations
- Monitor memory counts
- Track sync duration trends
- Test sync on development data first
- Verify metadata preservation
- Confirm search functionality post-sync
name: Memory Sync
on:
schedule:
- cron: '0 */6 * * *' # Every 6 hours
workflow_dispatch:
jobs:
sync:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: '3.11'
- name: Install dependencies
run: |
pip install uv
uv pip install -e .
- name: Sync memories
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CLOUDFLARE_API_TOKEN }}
CLOUDFLARE_ACCOUNT_ID: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
CLOUDFLARE_D1_DATABASE_ID: ${{ secrets.CLOUDFLARE_D1_DATABASE_ID }}
CLOUDFLARE_VECTORIZE_INDEX: ${{ secrets.CLOUDFLARE_VECTORIZE_INDEX }}
run: |
python scripts/sync_memory_backends.py --direction bidirectionalPlanned improvements for sync functionality:
- Selective Sync: Sync by tags or time ranges
- Incremental Sync: Only sync changes since last run (available today via delta-sync / self-hosted hub)
- Conflict Resolution UI: Interactive merge tool
- Multi-Backend Support: Sync with ChromaDB, PostgreSQL
- Compression: Reduce bandwidth for large syncs
- Encryption: End-to-end encryption for sensitive memories
For a self-hosted, event-log based alternative to the Cloudflare hybrid sync, MCP Memory Service supports delta-sync with a passive hub (ADR-0027 / ADR-0028). Spokes (clients) run periodic pull+push cycles against the hub; the hub never initiates connections. Transport is plain HTTPS, so a private network such as Tailscale works well. This mechanism is newer than the Cloudflare backup/restore scripts above and is opt-in.
-
Hub (e.g. a small VM): one HTTP(S) node with
MCP_SYNC_EVENTLOG=on. Configured with noMCP_SYNC_PEERS, so it schedules no outbound sync — a passive pivot. -
Spoke (client):
MCP_SYNC_EVENTLOG=onplus peer config. Each tick it pulls the hub's event log (the delta) and pushes its own queued events.
Hub (.env):
MCP_MEMORY_STORAGE_BACKEND=sqlite_vec
MCP_HTTPS_ENABLED=true
MCP_HTTP_HOST=0.0.0.0
MCP_HTTP_PORT=8020
MCP_SSL_CERT_FILE=/path/to/certs/cert.pem
MCP_SSL_KEY_FILE=/path/to/certs/key.pem
MCP_SYNC_EVENTLOG=onSpoke (.env):
MCP_SYNC_EVENTLOG=on
MCP_SYNC_SCHEDULE=15m # pull+push every 15 minutes (also 6h, 90s …); unset/disabled = off
MCP_SYNC_PEERS=hub
MCP_SYNC_PEER_URL=https://hub.example.com:8020
MCP_SYNC_PEER_API_KEY="<the HUB's MCP_API_KEY>"
# MCP_SYNC_BASIC_USER=… ; MCP_SYNC_BASIC_PASS=… # alternative auth
# MCP_SYNC_PUSH_ALLOWED_AGENTS=laptop,desktop # hub: restrict which agents may push
MCP_SYNC_PEER_API_KEYis notMCP_API_KEY. On a spoke,MCP_API_KEYalso enables auth enforcement for the spoke's own/mcpendpoint — setting it to the hub's value would break every local MCP client. Keep the hub credential in the dedicated variable.
Run the modern, package-managed entrypoint under systemd (EnvironmentFile=.env supplies the
variables):
[Service]
WorkingDirectory=/opt/mcp-memory-service
EnvironmentFile=/opt/mcp-memory-service/.env
ExecStart=/opt/mcp-memory-service/.venv/bin/memory launch --foreground# Hub reachable + healthy (from the spoke or the hub itself)
curl -sk -H "X-API-Key: $KEY" https://hub.example.com:8020/api/health/detailed
# Spoke sync cycle — look for "Delta-sync <peer>: pulled N applied, pushed M"
journalctl -u mcp-memory-http.service | grep -i "delta-sync"A cycle logs a line such as Delta-sync hub: pulled 16 applied, pushed 5.
-
memory launchreadsMCP_HTTP_HOST/MCP_HTTP_PORTfrom the process environment only (defaults127.0.0.1:8000); under systemd theEnvironmentFilesupplies them. It honoursMCP_SSL_CERT_FILE/MCP_SSL_KEY_FILEand only generates a development certificate when neither is set. - systemd precedence: an
EnvironmentFileoverrides a drop-inEnvironment=directive. - The wheel ships only
src/mcp_memory_service;scripts/is intentionally not packaged.scripts/server/run_http_server.pyis the legacy entrypoint —memory launchis the package-managed equivalent. -
Observability gap: unlike the hybrid/Cloudflare sync, delta-sync currently has no Web GUI
status panel, no
MCP_SYNC_*entry in Settings, and does not feed the dashboard's live refresh. Tracked upstream (see Related Documentation).
- Self-Hosted Sync Hub (full guide in the repository)
- Delta-sync GUI gaps (tracked): status display #1555 · settings #1556 · auto-refresh #1557
- Cloudflare Backend Setup
- Scripts README
- Configuration Guide
- Troubleshooting Guide
For issues with sync functionality:
- Check configuration with
python scripts/validate_config.py - Review logs in
/tmp/memory-sync.log - Open an issue with sync debug output
- Join discussions for community support
Last updated: January 2025 Sync tools version: 1.0.0