unify the three archive-as-filesystem implementations - #10023
unify the three archive-as-filesystem implementations#10023ThomasWaldmann wants to merge 1 commit into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #10023 +/- ##
==========================================
+ Coverage 86.14% 86.57% +0.43%
==========================================
Files 96 97 +1
Lines 17326 16788 -538
Branches 2649 2533 -116
==========================================
- Hits 14925 14534 -391
+ Misses 1663 1566 -97
+ Partials 738 688 -50 ☔ View full report in Codecov by Harness. |
|
CI found one real failure ( Root cause is a latent bug in Fixed in addad81, with unit tests for both flavours of a missing chunk and an end-to-end webdav test for downloading a file with a chunk missing. Happy to split that fix into its own PR if you prefer. |
addad81 to
fb5da8f
Compare
|
Moved the So expect this PR's CI to be red until #10024 is merged; I will rebase then. The last run that had both commits was fully green: all 3 FUSE legs, windows, the 4 VM legs, docs, mypy, lint, security, asan/ubsan, CodeQL and codecov (https://github.com/borgbackup/borg/actions/runs/30796758829). |
…0020 borg materialized an archive as a browsable tree in three independent places: fuse.py (llfuse/pyfuse3, low-level FUSE), hlfuse.py (mfusepy, high-level FUSE) and webdav.py (ArchiveVFS + WebDAV/HTTP server). All three re-implemented tree building, hardlink handling, the versions view, uid/gid/mode/time mapping and reading file content from chunk lists - so every behaviour fix had to be applied N times (e.g. the ACL/xattr exposure fix borgbackup#9954 touched both FUSE variants). New module vfs.py has that logic exactly once: - ArchiveVFS: archive selection and name deduplication, lazily built per-archive trees, the versions view, hardlinks (nodes sharing one inode), item storage (msgpacked, path-less, as hlfuse did it), attribute mapping, xattrs/ACLs. - DataReader: reads byte ranges out of chunk lists, with the decrypted-chunk cache (BORG_MOUNT_DATA_CACHE_ENTRIES) and the sequential-read position hint. - parse_mount_options(): the "borg mount -o ..." parsing both mounts duplicated. fuse.py, hlfuse.py and webdav.py are now thin protocol adapters over it (2604 -> 2009 lines in total). Behaviour changes that fell out of the unification: - webdav reads now go through DownloadPipeline.fetch_many(), so the all-zero chunk shortcut and the parsed-chunk cache (borgbackup#1678) apply to mounts as well. - the mounts get webdav's Unicode NFC lookup fallback (macOS decomposes names). - directories report st_nlink >= 2 (hlfuse behaviour) in both mounts. - a chunk that is read to its end is no longer put into the data cache, so a full download does not evict the chunks partial (range) reads need - this was the FUSE behaviour, now webdav shares it. The ACL emulation and the NFC lookup are now tested against the core (testsuite/vfs_test.py, no FUSE dependency); fuse_test.py keeps testing what is left in the adapters: the errno mapping. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fb5da8f to
2573ddf
Compare
|
@PhrozenByte if you have time, give this some practical testing, please. |
Fixes #10020.
borg materialized an archive as a browsable tree in three independent places:
fuse.py(llfuse/pyfuse3, low-level FUSE),
hlfuse.py(mfusepy, high-level FUSE) andwebdav.py(its own
ArchiveVFS+ the WebDAV/HTTP server). All three re-implemented tree building,hardlink handling, the versions view, uid/gid/mode/time mapping and reading file content
from chunk lists - so every behaviour fix had to be applied N times (e.g. the ACL/xattr
exposure fix #9954 had to touch both FUSE variants separately).
What this does
New module
src/borg/vfs.pyhas that logic exactly once:ArchiveVFS: archive selection and name deduplication, lazily built per-archive trees,the versions view, hardlinks (nodes sharing one inode), item storage (msgpacked and
path-less, as
hlfuse.pydid it), attribute mapping, xattrs/ACLs.DataReader: reads byte ranges out of chunk lists, with the decrypted-chunk cache(
BORG_MOUNT_DATA_CACHE_ENTRIES) and the sequential-read position hint.parse_mount_options(): theborg mount -o ...parsing that both mounts duplicated.fuse.py(805 -> 251 lines),hlfuse.py(737 -> 184) andwebdav.py(1062 -> 858) are nowthin protocol adapters over it - 2604 -> 2009 lines in total, and both
FuseBackendclassesand
ItemCacheare gone.Behaviour changes that fell out of the unification
DownloadPipeline.fetch_many()now, so the all-zero chunk shortcutand the parsed-chunk cache (better handling of repeated chunks to speed up extracting sparse files #1678) finally cover the FUSE path, too (FUSE micro-opt benchmarking #5110).
st_nlink >= 2(thehlfuse.pybehaviour) in both mounts.does not evict the chunks that partial (range) reads need. This was the FUSE behaviour,
webdav shares it now.
webdav, and now do so in the mounts as well.
Trade-off worth a look: dropping
ItemCachemeans the llfuse/pyfuse3 mount now has thesame memory profile as the mfusepy mount (a msgpacked item per inode, kept in memory)
instead of the 9-bytes-per-item meta-array that re-fetched metadata chunks from the
repository on access. That is what the default implementation (mfusepy) already does, but
it is more memory than the low-level mount used for very large archives.
Tests
(
testsuite/vfs_test.py, no FUSE dependency at all);testsuite/fuse_test.pykeepstesting what is left in the adapters: the errno mapping.
against both implementations (llfuse and mfusepy on macFUSE).
🤖 Generated with Claude Code