fix: remove exclusive locks from read-only storage operations

[greysonlalonde]

Summary

• remove store_lock from read-only storage operations across LanceDB, ChromaDB, and RAG adapters that were causing exclusive lock contention under multi-process prefork workers
• wrap lock_store BaseLockException with an actionable error message pointing users to set REDIS_URL
• add exception handling in encoding_flow/recall_flow ThreadPoolExecutor calls so a failed storage search degrades gracefully instead of crashing the flow
• fix flow.py double-logging where ancestor listeners re-logged downstream errors

Context

under prefork workers with 8 concurrency, read operations like search, list_scopes, get_scope_info, count were all holding exclusive Redis locks via portalocker.RedisLock. with 8 workers x N threads all fighting for the same lock channel, contention triggered a portalocker bug where AlreadyLocked is raised with the exceptions module object as its argument, producing the uninformative <module 'portalocker.exceptions'> error.

LanceDB uses MVCC and ChromaDB is thread-safe since v0.4 — neither needs exclusive locks for reads.

Test plan

• 108 existing memory/chromadb/lock tests pass
• 8 concurrent crew kickoffs via celery prefork + redis locks: 0 errors
• 50 concurrent crew kickoffs via celery prefork + redis locks: 0 errors, 104k log lines, zero portalocker/lock/trace_id errors
• LanceDB concurrent read safety verified empirically: 1400 reads + 40 writes across 16 threads, 0 errors

---

Note

Medium Risk
Touches concurrency/locking around storage reads and modifies error handling in memory flows; behavior under multi-process load could change (e.g., races or previously-serialized reads now concurrent) and should be validated in real deployments.

Overview
Reduces lock contention by removing exclusive store_lock usage from read-only operations across LanceDB/ChromaDB integrations (e.g., LanceDB adapter query, RAG query, ChromaDB client search/asearch, and multiple LanceDBStorage read paths like search, get_record, list_scopes, get_scope_info, count, etc.).

Improves resilience and diagnostics under concurrency by (1) wrapping portalocker lock acquisition failures with a clearer LockException message in lock_store.py, (2) making EncodingFlow and RecallFlow degrade gracefully when parallel storage searches/listing fail (log warning + skip/fallback instead of crashing), and (3) preventing duplicate listener error logs in Flow._execute_single_listener by marking exceptions as already logged.

^{Written by Cursor Bugbot for commit 91bad3b. This will update automatically on new commits. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}