Skip to content

Let backends declare thread_safe and allow concurrent backend calls in Store (follow-up to #206) #208

Description

@ThomasWaldmann

Follow-up to #206 / #207 (from Claude, reviewed by TW).

#207 serializes all Store operations with one internal lock. That is correct and sufficient for today's callers (borg's #9988 pack store-thread issues one op at a time), but it means the store can never execute two backend calls concurrently - even on backends where that would be safe and beneficial.

Idea

Let a backend declare itself safe for concurrent calls, and let Store exploit that:

  1. Declaration: BackendBase.thread_safe: bool = False class attribute. posixfs can honestly set True (independent syscalls per call). sftp/rest/rclone/s3 stay False with their single session - or later become True via a connection pool.
  2. Store side: this is not just "skip the lock" - the Store's own bookkeeping (_stats Counter updates, writethrough-cache accounting, _cache_disabled) needs protection regardless of the backend. The finer-grained design: keep a small lock for stats/cache mutations, but allow the backend calls themselves (backend.load/store/delete/...) to run outside the lock when backend.thread_safe is true.
  3. Cache backend: the cache backend is a second, independent backend with its own thread_safe - concurrent primary + cache calls need both declarations (or per-backend locks).

Why bother (later)

None of these exist as callers today, which is why #207 deliberately kept the simple unconditional lock. This issue exists so the design above does not get lost, and to be picked up once a real parallel-I/O caller shows up.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions