Follow-up to #206 / #207 (from Claude, reviewed by TW).
#207 serializes all Store operations with one internal lock. That is correct and sufficient for today's callers (borg's #9988 pack store-thread issues one op at a time), but it means the store can never execute two backend calls concurrently - even on backends where that would be safe and beneficial.
Idea
Let a backend declare itself safe for concurrent calls, and let Store exploit that:
- Declaration:
BackendBase.thread_safe: bool = False class attribute. posixfs can honestly set True (independent syscalls per call). sftp/rest/rclone/s3 stay False with their single session - or later become True via a connection pool.
- Store side: this is not just "skip the lock" - the Store's own bookkeeping (
_stats Counter updates, writethrough-cache accounting, _cache_disabled) needs protection regardless of the backend. The finer-grained design: keep a small lock for stats/cache mutations, but allow the backend calls themselves (backend.load/store/delete/...) to run outside the lock when backend.thread_safe is true.
- Cache backend: the cache backend is a second, independent backend with its own
thread_safe - concurrent primary + cache calls need both declarations (or per-backend locks).
Why bother (later)
None of these exist as callers today, which is why #207 deliberately kept the simple unconditional lock. This issue exists so the design above does not get lost, and to be picked up once a real parallel-I/O caller shows up.
Follow-up to #206 / #207 (from Claude, reviewed by TW).
#207 serializes all Store operations with one internal lock. That is correct and sufficient for today's callers (borg's #9988 pack store-thread issues one op at a time), but it means the store can never execute two backend calls concurrently - even on backends where that would be safe and beneficial.
Idea
Let a backend declare itself safe for concurrent calls, and let
Storeexploit that:BackendBase.thread_safe: bool = Falseclass attribute. posixfs can honestly setTrue(independent syscalls per call). sftp/rest/rclone/s3 stayFalsewith their single session - or later becomeTruevia a connection pool._statsCounter updates, writethrough-cache accounting,_cache_disabled) needs protection regardless of the backend. The finer-grained design: keep a small lock for stats/cache mutations, but allow the backend calls themselves (backend.load/store/delete/...) to run outside the lock whenbackend.thread_safeis true.thread_safe- concurrent primary + cache calls need both declarations (or per-backend locks).Why bother (later)
get_manyon packs living in different objects.None of these exist as callers today, which is why #207 deliberately kept the simple unconditional lock. This issue exists so the design above does not get lost, and to be picked up once a real parallel-I/O caller shows up.