Skip to content

unify the three archive-as-filesystem implementations - #10023

Open
ThomasWaldmann wants to merge 1 commit into
borgbackup:masterfrom
ThomasWaldmann:unify-vfs-10020
Open

unify the three archive-as-filesystem implementations#10023
ThomasWaldmann wants to merge 1 commit into
borgbackup:masterfrom
ThomasWaldmann:unify-vfs-10020

Conversation

@ThomasWaldmann

Copy link
Copy Markdown
Member

Fixes #10020.

borg materialized an archive as a browsable tree in three independent places: fuse.py
(llfuse/pyfuse3, low-level FUSE), hlfuse.py (mfusepy, high-level FUSE) and webdav.py
(its own ArchiveVFS + the WebDAV/HTTP server). All three re-implemented tree building,
hardlink handling, the versions view, uid/gid/mode/time mapping and reading file content
from chunk lists - so every behaviour fix had to be applied N times (e.g. the ACL/xattr
exposure fix #9954 had to touch both FUSE variants separately).

What this does

New module src/borg/vfs.py has that logic exactly once:

  • ArchiveVFS: archive selection and name deduplication, lazily built per-archive trees,
    the versions view, hardlinks (nodes sharing one inode), item storage (msgpacked and
    path-less, as hlfuse.py did it), attribute mapping, xattrs/ACLs.
  • DataReader: reads byte ranges out of chunk lists, with the decrypted-chunk cache
    (BORG_MOUNT_DATA_CACHE_ENTRIES) and the sequential-read position hint.
  • parse_mount_options(): the borg mount -o ... parsing that both mounts duplicated.

fuse.py (805 -> 251 lines), hlfuse.py (737 -> 184) and webdav.py (1062 -> 858) are now
thin protocol adapters over it - 2604 -> 2009 lines in total, and both FuseBackend classes
and ItemCache are gone.

Behaviour changes that fell out of the unification

  • The mounts read via DownloadPipeline.fetch_many() now, so the all-zero chunk shortcut
    and the parsed-chunk cache (better handling of repeated chunks to speed up extracting sparse files #1678) finally cover the FUSE path, too (FUSE micro-opt benchmarking #5110).
  • The mounts get webdav's Unicode NFC lookup fallback (macOS decomposes file names).
  • Directories report st_nlink >= 2 (the hlfuse.py behaviour) in both mounts.
  • A chunk that is read to its end is no longer put into the data cache, so a full download
    does not evict the chunks that partial (range) reads need. This was the FUSE behaviour,
    webdav shares it now.
  • Synthesized (never archived) directories keep showing the mtime of their archive in
    webdav, and now do so in the mounts as well.

Trade-off worth a look: dropping ItemCache means the llfuse/pyfuse3 mount now has the
same memory profile as the mfusepy mount (a msgpacked item per inode, kept in memory)
instead of the 9-bytes-per-item meta-array that re-fetched metadata chunks from the
repository on access. That is what the default implementation (mfusepy) already does, but
it is more memory than the low-level mount used for very large archives.

Tests

  • The ACL/xattr emulation and the NFC lookup are tested against the core now
    (testsuite/vfs_test.py, no FUSE dependency at all); testsuite/fuse_test.py keeps
    testing what is left in the adapters: the errno mapping.
  • The full test suite is green locally, and the mount tests were run with real mounts
    against both implementations (llfuse and mfusepy on macFUSE).

🤖 Generated with Claude Code

@codecov

codecov Bot commented Aug 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.49097% with 64 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.57%. Comparing base (d5eb3f7) to head (2573ddf).
⚠️ Report is 3 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/borg/vfs.py 89.43% 31 Missing and 10 partials ⚠️
src/borg/hlfuse.py 85.93% 6 Missing and 3 partials ⚠️
src/borg/fuse.py 91.17% 6 Missing ⚠️
src/borg/webdav.py 92.59% 4 Missing and 2 partials ⚠️
src/borg/archiver/mount_cmds.py 75.00% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #10023      +/-   ##
==========================================
+ Coverage   86.14%   86.57%   +0.43%     
==========================================
  Files          96       97       +1     
  Lines       17326    16788     -538     
  Branches     2649     2533     -116     
==========================================
- Hits        14925    14534     -391     
+ Misses       1663     1566      -97     
+ Partials      738      688      -50     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

@ThomasWaldmann

Copy link
Copy Markdown
Member Author

CI found one real failure (test_fuse_allow_damaged_files, all three FUSE legs): a chunk missing in the repository produced TypeError: object of type 'NoneType' has no len() instead of the intended EIO.

Root cause is a latent bug in DownloadPipeline.fetch_many(), not in the refactoring: with replacement_chunk=False it is documented (and used) to yield None for a missing chunk, but the size check right before the yield then did len(None). Nothing hit it so far, because borg webdav was the only caller passing replacement_chunk=False for file content - so webdav's "chunk missing" path (abort the connection instead of serving corrupted data) never actually ran, the TypeError ended up as a 500. Now that the mounts read via fetch_many(), too, the damaged-files test found it.

Fixed in addad81, with unit tests for both flavours of a missing chunk and an end-to-end webdav test for downloading a file with a chunk missing. Happy to split that fix into its own PR if you prefer.

@ThomasWaldmann

ThomasWaldmann commented Aug 3, 2026

Copy link
Copy Markdown
Member Author

Moved the fetch_many() fix into its own PR: #10024. This PR is back to the single refactoring commit and now depends on #10024 - without it, test_fuse_allow_damaged_files fails on all three FUSE legs (the mounts read via fetch_many() now, and a missing chunk raised TypeError instead of yielding None).

So expect this PR's CI to be red until #10024 is merged; I will rebase then. The last run that had both commits was fully green: all 3 FUSE legs, windows, the 4 VM legs, docs, mypy, lint, security, asan/ubsan, CodeQL and codecov (https://github.com/borgbackup/borg/actions/runs/30796758829).

…0020

borg materialized an archive as a browsable tree in three independent places:
fuse.py (llfuse/pyfuse3, low-level FUSE), hlfuse.py (mfusepy, high-level FUSE)
and webdav.py (ArchiveVFS + WebDAV/HTTP server). All three re-implemented tree
building, hardlink handling, the versions view, uid/gid/mode/time mapping and
reading file content from chunk lists - so every behaviour fix had to be applied
N times (e.g. the ACL/xattr exposure fix borgbackup#9954 touched both FUSE variants).

New module vfs.py has that logic exactly once:

- ArchiveVFS: archive selection and name deduplication, lazily built per-archive
  trees, the versions view, hardlinks (nodes sharing one inode), item storage
  (msgpacked, path-less, as hlfuse did it), attribute mapping, xattrs/ACLs.
- DataReader: reads byte ranges out of chunk lists, with the decrypted-chunk
  cache (BORG_MOUNT_DATA_CACHE_ENTRIES) and the sequential-read position hint.
- parse_mount_options(): the "borg mount -o ..." parsing both mounts duplicated.

fuse.py, hlfuse.py and webdav.py are now thin protocol adapters over it (2604 ->
2009 lines in total). Behaviour changes that fell out of the unification:

- webdav reads now go through DownloadPipeline.fetch_many(), so the all-zero
  chunk shortcut and the parsed-chunk cache (borgbackup#1678) apply to mounts as well.
- the mounts get webdav's Unicode NFC lookup fallback (macOS decomposes names).
- directories report st_nlink >= 2 (hlfuse behaviour) in both mounts.
- a chunk that is read to its end is no longer put into the data cache, so a
  full download does not evict the chunks partial (range) reads need - this was
  the FUSE behaviour, now webdav shares it.

The ACL emulation and the NFC lookup are now tested against the core
(testsuite/vfs_test.py, no FUSE dependency); fuse_test.py keeps testing what is
left in the adapters: the errno mapping.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@ThomasWaldmann

Copy link
Copy Markdown
Member Author

@PhrozenByte if you have time, give this some practical testing, please.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

three parallel archive-as-filesystem implementations: unify fuse.py, hlfuse.py and the webdav VFS

1 participant