Distributed deployment: mcp/server.py get_engine() deadlocks daemon with sqlite3.OperationalError

[unfall103-debug]

Problem Description
In a distributed/containerized deployment where a master daemon (unified_daemon.py) handles background tasks 24/7 while remote or separate client worker processes invoke MCP tools (e.g., list_recent), the client process frequently crashes with:

sqlite3.OperationalError: database is locked

This occurs because every cold MCP sub‑process aggressively re‑initializes the database, colliding with the write‑locks already held by the master daemon.

Root Cause Analysis
The flaw is inside superlocalmemory/mcp/server.py in the get_engine() singleton factory:

python
def get_engine():
global _engine
if _engine is not None:
return _engine
with _engine_lock:
if _engine is not None:
return _engine
...
config = SLMConfig.load()
new_engine = MemoryEngine(config, capabilities=Capabilities.LIGHT)
new_engine.initialize() # <--- ARCHITECTURAL BUG
_engine = new_engine
return _engine
_engine_lock is an in‑process lock – it cannot coordinate between the persistent daemon and separate MCP worker processes.

Every cold worker sees _engine as None and unconditionally calls new_engine.initialize(), which executes heavy DDL (create_all_tables) on an already‑locked SQLite database.

This directly collides with the daemon’s write operations, causing immediate deadlocks.

Error Traceback
text
2026-06-21 04:07:52,075 Processing request of type CallToolRequest
2026-06-21 04:07:57,083 list_recent failed
Traceback (most recent call last):
File "…/mcp/tools_core.py", line 337, in list_recent
engine = get_engine()
File "…/mcp/server.py", line 64, in get_engine
new_engine.initialize()
File "…/core/engine.py", line 125, in initialize
self._init_db_layer()
File "…/core/engine.py", line 150, in _init_db_layer
self._db.initialize(schema)
File "…/storage/database.py", line 123, in initialize
schema_module.create_all_tables(conn)
File "…/storage/schema.py", line 713, in create_all_tables
conn.executescript(ddl)
sqlite3.OperationalError: database is locked
Proposed Fix
Intercept the lock contention gracefully, skip redundant DDL, and allow the worker to reuse the existing database connection:

python
import sqlite3
import logging

logger = logging.getLogger("superlocalmemory")

def get_engine():
global _engine
if _engine is not None:
return _engine
with _engine_lock:
if _engine is not None:
return _engine
…
new_engine = MemoryEngine(config, capabilities=Capabilities.LIGHT)

    # --- SAFE DISTRIBUTED PATCH ---
    try:
        new_engine.initialize()
    except sqlite3.OperationalError as e:
        if "locked" in str(e).lower():
            logger.warning(
                "Database locked by master daemon; "
                "skipping redundant schema init and reusing existing connection."
            )
            pass
        else:
            raise
    # --------------------------------

    _engine = new_engine
return _engine

Environment
SLM Version: superlocalmemory 3.6.16 (verified via slm -v)

Python: 3.13+

Host: Linux SuperLocalMemory 7.0.2-2-pve, Proxmox VE 9.2.3

Topology: Multi‑LXC distributed cluster

CT 111 (192.168.50.140) – llama.cpp

CT 111 (192.168.50.143) – MCP Hub proxy

CT 106 (192.168.50.142) – OpenClaw frontend

CT 113 (192.168.50.144) – SuperLocalMemory server (24/7 daemon)

Database: SQLite, ~41 MB, under sustained concurrent read/write load

Additional Context
This issue is distinct from the database is locked errors that occur under heavy background task concurrency (which I’ve reported separately). Here, the crash is architectural – the get_engine() design assumes a single‑process environment and cannot safely coexist with a persistent daemon in a distributed deployment.

Thank you for your work on SLM – it has become a core part of my daily workflow. I’m happy to provide additional logs or testing feedback if needed.

Best regards,
sinking

[unfall103-debug]

📝 Follow-up: Root cause confirmed — internal thread contention and permanent lock leak, not external load

After applying the previous fixes and running the system with zero external traffic (no HTTP MCP calls, no agent requests, and proxy completely disconnected due to the indentation error), I observed the exact same database is locked error appearing within 60 seconds of a clean startup.

The structural log evidence is now undeniable:

1. Materializer begins processing 551 pending items and claims a long-lived exclusive write lock.
2. outcome_queue worker polls aggressively every 0.25 seconds (interval=0.25s), repeatedly colliding with the Materializer's active transaction.
3. EventBus also attempts a simultaneous write during this short window.
4. After 5 retries, the core layer panics: DB operation failed after 5 retries: database is locked.

No external load or network congestion was involved. This confirms that the severe lock contention is explicitly caused by SLM's own internal background threads racing against each other on the identical SQLite connection file pool.

Additionally, two other critical bugs were isolated during this specific diagnostic session:

• AttributeError causing Permanent Lock Leak: There is a syntax/naming regression where FeedbackCollector attempts to call record_dashboard_feedback, which does not exist (likely should be record_explicit or similar). This causes the background maintenance thread to crash hard. Because the transaction lacks an explicit ROLLBACK safety-net inside that specific exception block, the write lock remains permanently held in memory until a hard process kill.
• Redundant MCP Init Loop: The distributed-deployment patch suggested earlier (the try/except around new_engine.initialize()) was a necessary local bypass, though manually applying it produced an indentation error (MCP HTTP mount failed). This further proves that initialize() should never be invoked automatically by ephemeral client worker forks in the first place.

💡 Refined Recommendations for the Architecture:

To make SLM viable for anything beyond a single-threaded toy project, the core sync architecture requires:

1. A Consolidated Write Broker: Abstract and serialize all mutating DB operations from EventBus, Materializer, and outcome_queue through a thread-safe, unified single-writer queue/worker, instead of allowing concurrent multi-threaded transaction spamming.
2. Back-off / Relax Polling: Relax the aggressive 0.25s interval on outcome_queue or implement an exponential back-off strategy during heavy Materializer drain loops.
3. Expose Connection Parameters: Expose underlying parameters like _MAX_RETRIES, _BUSY_TIMEOUT_MS, and _RETRY_BASE_DELAY as Environment Variables so users can tune the SQLite endurance threshold to match their specific host I/O capabilities.

Thank you again for your dedicated work on SLM, and I hope these production-first metrics help scale the framework!

[varun369]

Worked through every claim here against the actual source. Mixed picture, so being precise:

Confirmed and fixed (your secondary finding was spot-on): server/routes/learning.py called feedback.record_dashboard_feedback(...), but FeedbackCollector only had record_implicit / record_explicit — the method did not exist. Every dashboard thumbs/pin/dwell write raised AttributeError. Implemented it in v3.6.17 (maps the dashboard vocabulary to stored (signal_type, value) pairs; raw query is hashed, never stored) with full regression tests. One nuance: the route wraps the call in except Exception, so it was a dead feature, not the "permanent lock leak" — no lock was actually held.

Could not reproduce (the primary title claim): I read mcp/server.py:get_engine() closely. It uses a correct double-checked lock around a single initialize(); there's no code-level deadlock and no redundant init loop in current main. The sqlite3.OperationalError: database is locked you saw is lock contention, not a deadlock — and the core DB already runs WAL + busy_timeout + retry.

Mitigations shipped for the contention (also in v3.6.17, see #53): the busy_timeout / max-retries / retry-delay are now env-tunable (SLM_DB_*), and the outcome_queue 0.25s poll now backs off when idle.

pip install -U superlocalmemory==3.6.17 && slm restart

Closing as addressed. If you still see a genuine deadlock (not just transient BUSY under heavy write load) on 3.6.17, please reopen with the daemon log around the stall and your deployment topology — I'll dig in.