Skip to content

PackReader.read(): return a memoryview of the in-memory pack instead of copying - #10037

Merged
ThomasWaldmann merged 1 commit into
borgbackup:masterfrom
mr-raj12:packreader-read-memoryview
Aug 4, 2026
Merged

PackReader.read(): return a memoryview of the in-memory pack instead of copying#10037
ThomasWaldmann merged 1 commit into
borgbackup:masterfrom
mr-raj12:packreader-read-memoryview

Conversation

@mr-raj12

@mr-raj12 mr-raj12 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Description

PackReader.read() read chunks from the in-memory pack by slicing pack_contents, a bytes object, so every read allocated and copied. It now returns a memoryview slice, which is a view into the pack with no copy. On a profiled 6 GiB extract that copy cost about 1.3% of extract CPU.

Supporting changes:

  • parse() and parse_meta() take bytes or memoryview for cdata (both wrap it in memoryview() anyway).
  • get(read_data=False) returns header plus meta as bytes.

The store range-read path is unchanged and still returns bytes. Re-put and key detection already accept the buffer protocol.

fixes #10030

Checklist

  • PR is against master
  • New code has tests and docs where appropriate
  • Tests pass (repository, repoobj, and extract/check/debug consumer subsets)
  • Commit messages are clean and reference related issues

…of copying

Avoids a per-chunk memcpy out of the cached pack; parse()/parse_meta() accept the memoryview.
@codecov

codecov Bot commented Aug 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.15%. Comparing base (2fe079e) to head (7f0f3cb).
⚠️ Report is 1 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #10037   +/-   ##
=======================================
  Coverage   86.15%   86.15%           
=======================================
  Files          96       96           
  Lines       17360    17360           
  Branches     2659     2659           
=======================================
  Hits        14956    14956           
  Misses       1664     1664           
  Partials      740      740           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@ThomasWaldmann
ThomasWaldmann merged commit cdb3a33 into borgbackup:master Aug 4, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PackReader.read(): return a memoryview instead of copying out of the in-memory pack

2 participants