Repository.get_many: coalesce pack reads and prefetch — remote extract/mount/check are pure-latency-bound

[ThomasWaldmann]

From a global code review of the borg2 codebase (2026-08).

Problem

With the borg1 RPC gone, all object reads go through borgstore, and Repository.get_many() is strictly sequential: one store.load(offset, size) round trip per object (src/borg/repository.py, get_many → PackReader.read). PackReader.iter_headers likewise does one short range read per object header.

borg 1.x's remote protocol pipelined gets with readahead; borg2 currently has no equivalent, so on high-latency backends (rest over WAN, s3, sftp, rclone) restore/mount/check throughput is dominated by RTT, not bandwidth: restoring 100k objects at 30 ms RTT is ~50 minutes of pure waiting, regardless of object size.

Proposed direction

All transparent behind the existing get_many() generator contract:

1. Group by pack: the chunk index already knows (pack_id, offset, size) for every requested id. Sort requests per pack by offset and coalesce adjacent/nearby ranges into a single ranged store.load per cluster (objects written together during create tend to be read together during extract, so locality is usually excellent).
2. Prefetch across clusters: a small bounded pool (or async loop) keeping N loads in flight, yielding results in request order.
3. Reuse the same machinery for check's header walks (iter_headers currently pays one RTT per object).

Related: #1678 (repeated-chunk cache, merged), #5110 (FUSE fetch path), #37 (parallelism in general), #9988 (write-side async pack upload). This issue is the read-side counterpart and is probably the single biggest performance win available for remote repos.

🤖 Generated with Claude Code