Repository navigation
v0.34.0
DittoFS v0.34.0
A modular virtual filesystem that decouples file interfaces from storage backends.
Upgrade notes
The metadata store's on-disk format moves to version 2, and the move is one-way.
Once a v0.34.0 server opens a Badger metadata store, v0.33.0 and earlier will refuse to open it and report a future-format error. Nothing is lost and nothing is corrupted, but there is no downgrade path short of restoring a backup taken before the upgrade. Take one first if you need the option.
Existing stores migrate themselves. A file's chunk list is now held in segments rather than one value, reads accept both shapes, and the old whole-list key is retired the first time the file is written. There is no migration step to run.
Why it changed: a chunk list past a few thousand refs crossed Badger's large-value threshold, and every later commit appended a fresh copy to a log that its own reclamation pass never learned to collect. One field store reached 245 GiB of log against 0.15 GiB of live data.
Two operations that used to succeed now fail
Renaming a directory into its own subtree returns EINVAL. It used to be accepted, and it built a cycle of directories that no path reached, that the link count reported as alive, and that nothing ever collected. Any client with write permission on two directories could create one. Cycles created by an earlier build are still there and still uncollected.
Setting extended attributes past 256 KiB per file returns NFS4ERR_XATTR2BIG, or STATUS_EA_TOO_LARGE over SMB. The limit is on the encoded total for one file, not on each value, and it is all-or-nothing: nothing is stored when it trips. Per-value limits were already enforced at 64 KiB. For scale, ext4 bounds a file's whole xattr set to a single filesystem block, typically 4 KiB.
Installation
Quick install (macOS / Linux):
curl -fsSL https://github.com/marmos91/dittofs/releases/download/v0.34.0/install.sh | shHomebrew (macOS/Linux):
brew tap marmos91/tap
brew install marmos91/tap/dfs # Server daemon
brew install marmos91/tap/dfsctl # Client CLIDebian/Ubuntu (APT):
curl -fsSL https://s3.cubbit.eu/dittofs-binaries/apt/dittofs.gpg.key | gpg --dearmor --yes | sudo tee /usr/share/keyrings/dittofs.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/dittofs.gpg] https://s3.cubbit.eu/dittofs-binaries/apt stable main" | sudo tee /etc/apt/sources.list.d/dfs.list
sudo apt update && sudo apt install dfs
sudo systemctl enable --now dfsRHEL/Fedora (YUM):
sudo curl -fsSLo /etc/yum.repos.d/dfs.repo https://s3.cubbit.eu/dittofs-binaries/rpm/dfs.repo
sudo yum install dfs
sudo systemctl enable --now dfsArch Linux:
curl -fsSLO https://github.com/marmos91/dittofs/releases/download/v0.34.0/dfs_0.34.0_amd64.pkg.tar.zst
sudo pacman -U dfs_0.34.0_amd64.pkg.tar.zst
sudo systemctl enable --now dfsScoop (Windows):
scoop bucket add marmos91 https://github.com/marmos91/scoop-bucket
scoop install dfs # Server daemon
scoop install dfsctl # Client CLIDocker:
docker run -d \
-p 12049:12049 \
-p 12445:12445 \
-p 8080:8080 \
-v /path/to/config.yaml:/config/config.yaml:ro \
-v dittofs-data:/data \
marmos91c/dittofs:v0.34.0Binary Download:
# dfs (server daemon)
curl -sfL https://github.com/marmos91/dittofs/releases/download/v0.34.0/dfs_0.34.0_Linux_x86_64.tar.gz | tar xz
# dfsctl (client CLI)
curl -sfL https://github.com/marmos91/dittofs/releases/download/v0.34.0/dfsctl_0.34.0_Linux_x86_64.tar.gz | tar xzChangelog
Features
Bug Fixes
- 927bef7: fix(bench): collect parseable repeated blockstore samples (#2862) (@badrkamel)
- 60c9a6e: fix(engine): read the syncer's wiring under the lock that publishes it (#2856) (@marmos91)
- b8d0934: fix(hooks): check module tidiness without rewriting files (#2847) (@badrkamel)
- 79561cd: fix(hooks): test operator packages own module (#2848) (@badrkamel)
- 142c9f8: fix(journal): carry a retired segment's markers forward before unlinking it (#2854) (@marmos91)
- a116793: fix(journal): wire the local-tier eviction and backpressure metrics (#2872) (@marmos91)
- d122bc4: fix(metadata): bound badger conflict retries by deadline and randomise the backoff (#2889) (@marmos91)
- 654fdbd: fix(metadata): bound the total extended-attribute bytes a file may carry (#2888) (@marmos91)
- 226532e: fix(metadata): refuse a rename that moves a directory under itself (#2887) (@marmos91)
- d8591b2: fix(metadata): segment the badger chunk manifest so it stays out of the value log (#2882) (@marmos91)
- 1cfaa29: fix(smb): enumerate every metadata directory page (#2879) (@badrkamel)
- 995589b: fix(test): cap conformance grader exit statuses (#2876) (@badrkamel)
- 0830990: fix(test): reject incomplete POSIX TAP results (#2874) (@badrkamel)
- 526dba4: fix(test): size the hot-file retry test inside the budget it asserts (#2890) (@marmos91)
- 9f00199: fix: two flakes at the cause, two Wave 1 deletion gaps (#2859) (@marmos91)
Performance Improvements
- 81a5716: perf(badger): avoid repeated sparse chunk index scans (#2875) (@badrkamel)
Others
- 3e47bf6: build: compile complete CLI entrypoint packages (#2868) (@badrkamel)
- 4f878b2: chore(planning): drop spent audit and perf docs (@marmos91)
- 7cb442e: chore(planning): drop spent wave and step docs, make MEMORY.md an index (#2830) (@marmos91)
- 71cf4e1: chore(release): v0.34.0 (@marmos91)
- 13cde69: ci: build conformance binaries in-job instead of via artifacts (#2844) (@badrkamel)
- 9281dcd: ci: deduplicate Nix cache publication across test matrices (#2861) (@badrkamel)
- 9ad1d8a: ci: key the operator module cache on its own go.mod (#2835) (@badrkamel)
- 8eae797: ci: prevent vendor helper from seeding an empty Go cache (#2864) (@badrkamel)
- b1ec969: ci: run tests when SQL migrations change (#2836) (@badrkamel)
- 8da8fd3: ci: run tests when bench workloads change (#2845) (@badrkamel)
- d650b7d: ci: stagger periodic conformance suite starts (#2866) (@badrkamel)
- f72278b: ci: start conformance services only when required (#2834) (@badrkamel)
- faca5a1: ci: use the codecov files input (#2837) (@badrkamel)
- 0769dfe: docs(changelog): resume the changelog and describe 0.34.0 (#2896) (@marmos91)
- c66a3f2: docs(planning): fold the LocalStore deletion and the metrics rethink into Wave 4 (#2858) (@marmos91)
- 95206bf: docs(planning): fold the open issues that bear on the block plan into it (#2831) (@marmos91)
- 824a88b: docs(planning): plan the pkg/block simplification, and record why the checkpoint died (#2828) (@marmos91)
- 5f5024c: refactor(block): Wave 1 — pure deletion across journal, engine, stores and root (#2842) (@marmos91)
- 5750fe6: refactor(block): Wave 2 — extract pkg/block/gc, group the decorators under middleware (#2846) (@marmos91)
- 71ffb2e: refactor(block): Wave 3 — split syncer.go and reclaim.go, withdraw 3A (#2855) (@marmos91)
- 1de0fc3: refactor(block): cut the Store back-reference and the BlockSize alias (#2839) (@marmos91)
- b167091: refactor(config): use maintained mapstructure decode hooks (#2867) (@badrkamel)
- be85d27: refactor(middleware): compose block transforms as a pipeline of stages (#2850) (@marmos91)
- 38fa6d5: test(badger): gate the sparse-lookup cost, and repair the marker it moved (#2892) (@marmos91)
- 52f7cc6: test(bench): cover embedded Samba re-export configuration (#2865) (@badrkamel)
- ee95cc3: test(rpc): fuzz AUTH_UNIX decoding against bounded XDR oracle (#2863) (@badrkamel)
Full Changelog: v0.33.0...v0.34.0
Verifying Checksums
sha256sum -c checksums.txtVerifying Signatures
Release checksums are signed with Sigstore cosign using keyless OIDC:
cosign verify-blob \
--bundle checksums.txt.sigstore.json \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-identity-regexp '^https://github\.com/marmos91/dittofs/\.github/workflows/release\.yml@refs/tags/' \
checksums.txtDocker Images
Multi-architecture images are available for linux/amd64 and linux/arm64:
marmos91c/dittofs:v0.34.0- This specific versionmarmos91c/dittofs:latest- Latest stable releasemarmos91c/dittofs:v0- Latest v0.x.xmarmos91c/dittofs:v0.34- Latest v0.34.x