A volume nobody wrote to is not snapshotted again - #112
Merged
Conversation
ccomb
force-pushed
the
send-refuses-what-the-remote-already-has
branch
from
August 27, 2026 19:46
a848877 to
4364c5c
Compare
Base automatically changed from
send-refuses-what-the-remote-already-has
to
master
August 27, 2026 19:46
A snapshot scheduled every minute on a volume at rest wrote one identical subvolume per minute, and a replication to two hosts wrote two. BTRFS imposes the order: a send needs a readonly subvolume, so a live volume cannot be compared to a snapshot. The copy is taken first, compared against the previous snapshot, and deleted when it carries nothing new. That is a deletion, and it is deliberate: the object deleted is the one this very call created, and it holds nothing that is not already in the snapshot whose name is answered instead. The comparison pipes a send into a receive without a shell, with the four precautions run_btrfs_send_receive documents, and a failure to compare keeps the copy rather than losing it. The Created flag is part of the same subject, not an extra. A replication that fails deletes the snapshot it took for the occasion, and the name answered can now come from an earlier round: without the flag, a network outage would delete a snapshot this round never created. api.snapshot answers the pair, the scheduler reads it, and the command line still prints one name. Two tests took two snapshots of a volume without writing in between and were counting on both existing. They write a byte now, which leaves what they were checking intact.
btrfs receive --dump writes the value of an extended attribute exactly as it is, without escaping it, in the version of btrfs-progs the plugin image ships. A volume holding a binary attribute, which is what a file server writes beside the files it stores, made the decoding raise a UnicodeDecodeError. That is not a BtrfsError, so it went past the guard that keeps a comparison we could not make from losing a snapshot: the endpoint answered an error while the copy it had just taken stayed on the disk, and the replication of that volume skipped the round. Counting the lines on the bytes needs no decoding at all. The two error messages decode with replacement, since a second failure while saying why the first one happened would hide it.
The deletion sits inside the same try as the comparison, so a filesystem that turns read-only between the two makes the log say the comparison could not be told, when it was told and the deletion is what failed. The answer stays right either way, since the copy really is still there.
The endpoint takes a copy, compares it with the previous snapshot and deletes it when the two hold the same thing. Two calls on the same volume, interleaved, each compare against the copy the other one just took, find it identical, and delete their own. The volume comes out of the two calls with neither, while both answers name a snapshot that is no longer there. A deploy script that snapshots a volume while the scheduled replication of the same volume is running is enough to meet it, and until now two calls at once simply left two snapshots. The daemon serves several requests at a time, so one lock is what answers. One for every volume rather than one per volume: the scheduler runs its jobs one after the other, and the only other snapshot is the one somebody asks for.
ccomb
force-pushed
the
an-unchanged-volume-is-not-snapshotted-again
branch
from
August 27, 2026 19:48
083653d to
4f1b168
Compare
ccomb
added a commit
that referenced
this pull request
Sep 2, 2026
Moving an application between two hosts was done by hand: stop it, send a last snapshot, restore that snapshot on the other host, start it there. Docker Swarm moves a service without asking, and its volume stayed behind. This is the proposal of #68 and the subject of #69, built on the `replicate:<host>` line rather than on a new verb, since what it does is replication with the handover automated, and "sync" is the name the README gives to the other thing. **What changes for a volume with a `replicate:<host>` line scheduled** - The first container to use the volume on a host asks that host for the last snapshot of the volume to appear there, receives it, and restores it when it is the last to have appeared here and came from another host. What the volume held is kept as a snapshot first. A host that does not answer refuses the mount, and the error says how to mount without asking: pause the line. - The last container to stop snapshots the volume and sends that snapshot, waiting for a scheduled replication under way so that what leaves is the final state. - While no container uses the volume here, each scheduled round fetches what appeared on the other host, without restoring it, so that a mount receives a difference and not a whole volume within the thirty seconds Docker gives it. - A send refuses to bury a history this host never saw: a host that crashed with unsent writes and comes back after the application ran elsewhere keeps those writes aside, and the work done elsewhere wins. **Every `replicate:<host>` line already in `schedule.csv` changes meaning with this version**, and the changelog says so. **What had to move underneath** - The order of snapshots is read from the order BTRFS created them in, and whether one came from another host from its `Received UUID`, never from the date in the name: a host whose clock runs ahead can no longer pass its copy off as the most recent one. `buttervolume receive` follows the same rule. - A restore keeps what the volume held the way a snapshot is kept: nothing new when the volume is unchanged since its last snapshot, nothing at all when it is empty, and nothing done when the volume already holds the snapshot asked for. `VolumeBackup` is always in the answer, and a `Restored` field says whether the volume was replaced. - A snapshot is compared with the last snapshot taken of the volume here, not with a received one. - `buttervolume replicate <host> <volume>` snapshots and sends in one step, through a new `/VolumeDriver.Replicate` endpoint the scheduler now calls; the replication lock lives in the plugin, where the unmount needs it. - A volume can ask for its scheduled jobs as it is created, `-o replicate:node2=1`, which is how a Swarm service says once what happens to its volume on whatever host it lands on. An option nobody reads is now refused. The README has a new section, "Move an application between hosts", and AGENTS.md says that a mount the user scheduled may replace a volume, keeping what it held. **Verified** - `./test_local.sh`: 129 passed, 11 skipped (the ones over ssh). - The whole suite in the plugin image, ssh included: 140 passed, nothing skipped, nothing failed. - Two BTRFS facts the design rests on were checked on a real filesystem before writing the code: a writable snapshot of a received subvolume clears its `Received UUID`, and the size of a BTRFS directory is zero exactly when it is empty. **Not verified here**, and worth doing on a machine where the plugin can be replaced: two containers on one volume as seen by Docker, a mount refused as seen by Docker and Swarm, and the duration of a first mount against `docker plugin enable --timeout`. The plugin enabled on the machine this was written on holds volumes, and a second one would share its directories and its ssh port. The design was reviewed before implementation by a separate session, which found the four problems the plan now records: dates versus creation order, "never written" versus "empty", one Mount per container, and the option read after the "volume already exists" return. One more surfaced while writing the tests: fetching ahead of the mount made the send barrier pass by mere presence of the name, so the barrier reads the BTRFS lineage instead. Supersedes #69, whose four side fixes were merged earlier as #110, #111, #112 and #113.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This work needed #111, which is now in
master. Without it, the secondreplication of a volume at rest names a snapshot the remote host already holds,
and the fallback of
snapshot_senddeletes the good remote copy.volume_snapshotalways createdvolume@now. Asnapshotscheduled everyminute on a volume at rest wrote one identical subvolume per minute, and
replicate:hostAplusreplicate:hostBwrote two, which is the case ticket #68reports.
BTRFS imposes the order.
btrfs sendneeds a readonly subvolume, so a livevolume cannot be compared to a snapshot: the copy is taken first, compared to
the previous snapshot with
send --no-data -ppiped intoreceive --dump, anddeleted when the stream carries nothing but its header. It is a deletion, which
rubs against "Destruction stays deliberate" in AGENTS.md, and the answer is that
the thing deleted is the one this very call created and holds nothing that is
not already in the snapshot named back. A comparison that fails keeps the copy.
The comparison lives in
btrfs.pyand pipes two processes without a shell, withthe four precautions
run_btrfs_send_receivedocuments: the sync first, thesend stderr in a temporary file rather than a pipe nobody drains, the
stdout.close()so the send gets a SIGPIPE, and one deadline for both waitswith the two processes killed and reaped on expiry. The dump is read whole
rather than two lines at a time;
--no-dataleaves the contents out, andCOMPARE_TIMEOUTbounds the volume where millions of files changed.Createdis part of the subject, not an extra.run_replicatedeletes thesnapshot it took when the replication fails. The name it now holds can come from
an earlier round, so without the flag a network outage would delete a snapshot
this round never created: a data loss introduced by this change.
/VolumeDriver.Snapshotanswers the flag,api.snapshotreturns the pair, thescheduler reads it, and
buttervolume snapshotstill prints one name (returningthe pair would break
main(), which readsis Falseas an exit code of 1).Two existing tests took two snapshots of a volume without writing in between and
counted on both existing. They write a byte now, which leaves what they were
checking intact.
A review of this branch found three things, fixed in the three commits that
follow the first one. The comparison decoded the dump of
btrfs receive, whichwrites the value of an extended attribute exactly as it is: a volume holding a
binary one raised a
UnicodeDecodeError, which is not aBtrfsError, so itwent past the guard that keeps a comparison we could not make from losing a
snapshot. The lines are counted on the bytes now. Two calls on the same volume
at once each compared their copy with the copy of the other, found it
identical, and deleted their own, leaving the volume with neither while both
answers named one: one lock answers that. And the deletion sat inside the same
tryas the comparison, so a failed deletion was reported as a comparisonnobody could make.
Verified:
88 passed, 6 skippedundertest_local.sh,94 passedunder./test.sh, including the two new tests that restore the remote copy. Andthe scenario the work is for: ten scheduled rounds on a volume at rest give one
local snapshot, one trace, one remote copy, nine "has not changed since" and
nine "nothing to send".