vdo: transparent compression under the dm-thin pool - #22
Merged
Conversation
Creating, removing, suspending, and messaging a dm device works the same whatever target is behind it, but all of that lived in dm_thin and dm_thin::pool. Anything else we want to stack under the pool would have to reach into the thin-pool module for suspend, which is exactly the cross-boundary reach our conventions warn about. Move the target-agnostic half to crate::dm, which only ever handles opaque table strings, and leave dm_thin owning what is actually specific to thin-pool and thin: the table layouts, PoolStatus, MetadataSnap, thin ids, and name derivation. Two small improvements come along with the move. dmsetup error labels now name the device rather than the target type, which is what you want when reading a failure. And the target-availability probe matches the first field of each dmsetup targets line instead of substring matching the whole listing, so a kernel offering thin can no longer be read as offering thin-pool.
dm-thin stores blocks verbatim, while ZFS compresses by default and delivers a little over 2x on real ember pools. `ember init --storage dm-thin --vdo` closes that gap by putting a dm-vdo target between the pool's data device and its backing store. The layer is opt-in, off by default, permanent at pool creation, and confined to dm-thin. Sizing is the part worth being careful about, because VDO adds a third level of over-provisioning. --size keeps meaning the real disk budget, and --vdo-logical-size defaults to it, so the default configuration does not deliberately over-promise: compression shows up as a smaller footprint on the host filesystem rather than as capacity that may not materialise. It is still a small bet even at 1:1, because VDO's own metadata reserve comes out of the physical side, which is what sets the 32 GiB floor: there the reserve is about a tenth of the volume and break-even is 1.1x, where at 8 GiB it would be nearer 1.6x. Running out of physical space under a thin pool is not graceful. The pool sees I/O errors from its data device and drops to read-only, taking every running VM with it, and its own accounting gives no warning because it is counting logical space. So VDO is checked before the pool on every allocating operation, refused above 95% physical, warned about above 85%, and reported by storage usage. A read-only volume is refused on every activation rather than only when allocating, since disk_device_path is what vm start uses and a guest booted onto one just gets EIO. None of this stops a running VM from filling the pool between commands, and the spec says so. Reporting needed one new field. PoolUsage::addressable carries the space a pool exposes when a compressing layer lets it exceed capacity, which keeps free() meaning real physical headroom and compression_ratio() meaning bytes stored over bytes on disk. VDO's own metadata goes to PoolUsage::reserved, read from the volume's `stats` message rather than from `dmsetup status`, which reports data and metadata blocks pre-summed. That distinction turned out to matter more than expected. The reserve is several gigabytes on any volume worth creating and barely moves with the data stored, so charging it to `allocated` alone divides the ratio by a wide constant: measured on a 60 GiB pool holding a 6.7 GiB Ubuntu image, 2.64 GiB of data alongside 4.05 GiB of metadata reported 1.00x where the real figure is 2.5x. Per-volume figures stay pre-compression, since VDO has no idea which thin volume a physical block belongs to, and the CLI says so with a footnote derived from the data rather than from a flag. storage grow gained a second dimension. --size preserves the logical-to-physical ratio so doubling the disk doubles the pool, and --logical-size overrides it. The whole request is planned and validated before anything is touched, and the resolved sizes are recorded between the VDO reload and the pool reload: the kernel durably adopts VDO's new sizes on resume, so a config still describing the old ones from that moment on is a pool that cannot be activated. Three pre-existing problems fell out of the work and are fixed here. A --size smaller than the current one silently truncated the backing file and destroyed data. The raw-device branch of grow told the operator to grow the device and re-run, which would have hit the same error again. And ember init only refused when the *backend* differed, so re-running it against a live pool zeroed the thin metadata superblock and destroyed every VM and image before the config write got around to reporting the conflict. Metadata is now sized from the space the pool can address rather than from the disk under it. Sized from the physical figure, a pool over-provisioned 4:1 would exhaust thin metadata at a quarter of its capacity and drop to read-only, which is the failure over-provisioning is a bet against and would have happened even when compression delivered. Deduplication is off by default. It costs RAM proportional to pool size on a host that is also running VMs, and it would fold two effects into the single savings figure VDO reports, making the compression number mean something else. --vdo-dedup turns it on, and measurement says leaving it off is right for this workload: on the same pool it took 2.64 GiB of data down to 2.58 GiB for 256 MiB more RAM, because dm-thin already shares everything between a VM and its fork and there was no duplication left below the pool to find. Also tidies the extraction this builds on: dm gains a load-and-resume table swap, because vdo documents that its volumes should not be suspended and suspending one under a live thin pool risks blocking on in-flight I/O; an unused helper and two pure passthrough wrappers are dropped; and the comments no longer describe dm-thin from inside the target-agnostic module. See docs/VDO-SPEC.md.
aljoscha
force-pushed
the
vdo-compression
branch
from
August 18, 2026 14:25
df6aa30 to
9d06e2e
Compare
Two user-facing messages in the dm-thin backend sent the operator the wrong way. The size-mismatch check at init is reachable two ways, and they have different fixes. A raw block device is whatever size it is, so --size has to match it. A data.img left behind by a `deinit` without --purge is a leftover pool, and telling that case to match --size only trades this error for `vdoformat` refusing to overwrite a volume it already holds. The message now names the backing kind and the fix that works for it. The "cannot address less than it did" message in `plan_grow` was missing a line continuation, so it printed an eighteen-space gap mid-sentence. Also drops a dangling doc line above `resolve_pool_size`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`check_physical_size` rejected a physical size past the kernel's 8192 slabs at init, but `check_growth` did not, so a grow past 16 TiB reached `dmsetup` and came back as EINVAL. `explain_einval` reads that as the recorded sizes disagreeing with what the volume was formatted with, which sends the operator to inspect a config.json that is fine. The ceiling is out of reach for any pool ember is used for, but `grow` discovers a raw device's size rather than being told it, so a larger device gets there without anyone naming a number. `check_physical_ceiling` gives the ceiling one owner so both paths report it the same way. The spec's hazard said the ceiling was left to the kernel, so it is updated to match. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`ember storage usage` and `ember info` printed the addressable line whenever it differed from the physical capacity, but the two figures are rounded differently. Capacity is a whole 4 KiB VDO block, while the addressable figure is the pool's data capacity rounded down to a whole pool block. On a raw device whose size is not a multiple of the pool block, a 1:1 pool therefore announced an over-provision that sat below its own capacity. Testing for strictly greater is simpler and is exactly the condition both comments already claimed to express. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds an opt-in
dm-vdocompression layer beneath the dm-thin pool's datadevice, closing the gap against ZFS, which compresses by default while
dm-thin stores blocks verbatim.
Off by default, permanent at pool creation, confined to dm-thin. ZFS and
APFS are untouched.
Measured
Same ubuntu-dev image, VM, and fork built in three independent
installations on a 60 GiB file-backed pool:
Deduplication earned 60 MiB for 256 MiB of extra RAM, because dm-thin
already shares everything between a VM and its fork through
copy-on-write and there is no duplication left below the pool to find.
That is why it is off by default.
Write throughput is not established. The build step took 62s, 79s, and
67s across the three cases, but dedup came out faster than
compression-only, which cannot be true, so that is noise on a workload
dominated by docker and
mkfs.ext4.Sizing
VDO adds a third level of over-provisioning, so
--sizekeeps meaningthe real disk budget and
--vdo-logical-sizedefaults to it. The defaultconfiguration does not deliberately over-promise: compression shows up as
a smaller footprint rather than as capacity that may not materialise.
It is still a small bet even at 1:1, because VDO's metadata reserve comes
off the physical side. That sets the 32 GiB floor, where the reserve is
about a tenth of the volume and break-even is ~1.1x; at 8 GiB it would be
nearer 1.6x.
Running out of space
Exhausting VDO's physical space under a thin pool is not graceful: the
pool sees I/O errors from its data device and drops to read-only, taking
every running VM with it, and its own accounting gives no warning because
it counts logical space. So VDO is checked before the pool on every
allocating operation, refused above 95% physical, warned above 85%, and
reported by
storage usage. A read-only volume is refused on everyactivation, not only when allocating, since
disk_device_pathis whatvm startuses. None of this stops a running VM filling the pool betweencommands, and the spec says so rather than implying otherwise.
Reporting
PoolUsagegainsaddressable, the space a pool exposes when acompressing layer lets it exceed capacity, which keeps
free()meaningreal physical headroom. VDO's own metadata goes to
reserved, read fromthe volume's
statsmessage becausedmsetup statusreports data andmetadata blocks pre-summed. That distinction is load-bearing: charged
together the ratio reads 1.00x, charged apart it reads 2.5x.
Per-volume figures stay pre-compression, since VDO cannot attribute a
physical block to a thin volume. The CLI prints a footnote when it
detects that shape, derived from the data rather than from a flag.
Also fixed
Three pre-existing problems surfaced while rewriting
storage grow:--sizesmaller than the current one silently truncated the backingfile and destroyed data.
which hit the same error again.
ember initonly refused when the backend differed, so re-running itagainst a live pool zeroed the thin metadata superblock and destroyed
every VM and image before the config write reported the conflict.
Metadata is now sized from what the pool can address rather than from the
disk under it. Sized from the physical figure, a pool over-provisioned
4:1 would exhaust thin metadata at a quarter of its capacity, which is
the failure over-provisioning is a bet against.
Review
Two adversarial review passes plus a verification pass over the fixes.
The verification round found two bugs in the fix pass itself:
--sizewasvalidated then discarded on a raw device without VDO, and the re-init
guard was bypassable by an unparseable
config.json. Both haveregression tests.
385 unit tests. Eight integration tests are
#[ignore]d and need root,dm-vdo, andvdoformat:See
docs/VDO-SPEC.md.