Skip to content

A volume nobody wrote to is not snapshotted again - #112

Merged
ccomb merged 4 commits into
masterfrom
an-unchanged-volume-is-not-snapshotted-again
Aug 27, 2026
Merged

A volume nobody wrote to is not snapshotted again#112
ccomb merged 4 commits into
masterfrom
an-unchanged-volume-is-not-snapshotted-again

Conversation

@ccomb

@ccomb ccomb commented Aug 27, 2026

Copy link
Copy Markdown
Owner

This work needed #111, which is now in master. Without it, the second
replication of a volume at rest names a snapshot the remote host already holds,
and the fallback of snapshot_send deletes the good remote copy.

volume_snapshot always created volume@now. A snapshot scheduled every
minute on a volume at rest wrote one identical subvolume per minute, and
replicate:hostA plus replicate:hostB wrote two, which is the case ticket #68
reports.

BTRFS imposes the order. btrfs send needs a readonly subvolume, so a live
volume cannot be compared to a snapshot: the copy is taken first, compared to
the previous snapshot with send --no-data -p piped into receive --dump, and
deleted when the stream carries nothing but its header. It is a deletion, which
rubs against "Destruction stays deliberate" in AGENTS.md, and the answer is that
the thing deleted is the one this very call created and holds nothing that is
not already in the snapshot named back. A comparison that fails keeps the copy.

The comparison lives in btrfs.py and pipes two processes without a shell, with
the four precautions run_btrfs_send_receive documents: the sync first, the
send stderr in a temporary file rather than a pipe nobody drains, the
stdout.close() so the send gets a SIGPIPE, and one deadline for both waits
with the two processes killed and reaped on expiry. The dump is read whole
rather than two lines at a time; --no-data leaves the contents out, and
COMPARE_TIMEOUT bounds the volume where millions of files changed.

Created is part of the subject, not an extra. run_replicate deletes the
snapshot it took when the replication fails. The name it now holds can come from
an earlier round, so without the flag a network outage would delete a snapshot
this round never created: a data loss introduced by this change.
/VolumeDriver.Snapshot answers the flag, api.snapshot returns the pair, the
scheduler reads it, and buttervolume snapshot still prints one name (returning
the pair would break main(), which reads is False as an exit code of 1).

Two existing tests took two snapshots of a volume without writing in between and
counted on both existing. They write a byte now, which leaves what they were
checking intact.

A review of this branch found three things, fixed in the three commits that
follow the first one. The comparison decoded the dump of btrfs receive, which
writes the value of an extended attribute exactly as it is: a volume holding a
binary one raised a UnicodeDecodeError, which is not a BtrfsError, so it
went past the guard that keeps a comparison we could not make from losing a
snapshot. The lines are counted on the bytes now. Two calls on the same volume
at once each compared their copy with the copy of the other, found it
identical, and deleted their own, leaving the volume with neither while both
answers named one: one lock answers that. And the deletion sat inside the same
try as the comparison, so a failed deletion was reported as a comparison
nobody could make.

Verified: 88 passed, 6 skipped under test_local.sh, 94 passed under
./test.sh, including the two new tests that restore the remote copy. And
the scenario the work is for: ten scheduled rounds on a volume at rest give one
local snapshot, one trace, one remote copy, nine "has not changed since" and
nine "nothing to send".

@ccomb ccomb added the agent-reviewed pr-cycle ran label Aug 27, 2026
@ccomb
ccomb force-pushed the send-refuses-what-the-remote-already-has branch from a848877 to 4364c5c Compare August 27, 2026 19:46
Base automatically changed from send-refuses-what-the-remote-already-has to master August 27, 2026 19:46
ccomb added 4 commits August 27, 2026 21:47
A snapshot scheduled every minute on a volume at rest wrote one
identical subvolume per minute, and a replication to two hosts wrote
two. BTRFS imposes the order: a send needs a readonly subvolume, so a
live volume cannot be compared to a snapshot. The copy is taken first,
compared against the previous snapshot, and deleted when it carries
nothing new. That is a deletion, and it is deliberate: the object
deleted is the one this very call created, and it holds nothing that is
not already in the snapshot whose name is answered instead.

The comparison pipes a send into a receive without a shell, with the
four precautions run_btrfs_send_receive documents, and a failure to
compare keeps the copy rather than losing it.

The Created flag is part of the same subject, not an extra. A
replication that fails deletes the snapshot it took for the occasion,
and the name answered can now come from an earlier round: without the
flag, a network outage would delete a snapshot this round never
created. api.snapshot answers the pair, the scheduler reads it, and the
command line still prints one name.

Two tests took two snapshots of a volume without writing in between and
were counting on both existing. They write a byte now, which leaves what
they were checking intact.
btrfs receive --dump writes the value of an extended attribute exactly as
it is, without escaping it, in the version of btrfs-progs the plugin
image ships. A volume holding a binary attribute, which is what a file
server writes beside the files it stores, made the decoding raise a
UnicodeDecodeError. That is not a BtrfsError, so it went past the guard
that keeps a comparison we could not make from losing a snapshot: the
endpoint answered an error while the copy it had just taken stayed on
the disk, and the replication of that volume skipped the round.

Counting the lines on the bytes needs no decoding at all. The two error
messages decode with replacement, since a second failure while saying
why the first one happened would hide it.
The deletion sits inside the same try as the comparison, so a filesystem
that turns read-only between the two makes the log say the comparison
could not be told, when it was told and the deletion is what failed. The
answer stays right either way, since the copy really is still there.
The endpoint takes a copy, compares it with the previous snapshot and
deletes it when the two hold the same thing. Two calls on the same
volume, interleaved, each compare against the copy the other one just
took, find it identical, and delete their own. The volume comes out of
the two calls with neither, while both answers name a snapshot that is
no longer there. A deploy script that snapshots a volume while the
scheduled replication of the same volume is running is enough to meet
it, and until now two calls at once simply left two snapshots.

The daemon serves several requests at a time, so one lock is what
answers. One for every volume rather than one per volume: the scheduler
runs its jobs one after the other, and the only other snapshot is the
one somebody asks for.
@ccomb
ccomb force-pushed the an-unchanged-volume-is-not-snapshotted-again branch from 083653d to 4f1b168 Compare August 27, 2026 19:48
@ccomb
ccomb merged commit 14ed36a into master Aug 27, 2026
2 checks passed
@ccomb
ccomb deleted the an-unchanged-volume-is-not-snapshotted-again branch August 27, 2026 19:49
ccomb added a commit that referenced this pull request Sep 2, 2026
Moving an application between two hosts was done by hand: stop it, send
a last snapshot, restore that snapshot on the other host, start it
there. Docker Swarm moves a service without asking, and its volume
stayed behind. This is the proposal of #68 and the subject of #69, built
on the `replicate:<host>` line rather than on a new verb, since what it
does is replication with the handover automated, and "sync" is the name
the README gives to the other thing.

**What changes for a volume with a `replicate:<host>` line scheduled**

- The first container to use the volume on a host asks that host for the
last snapshot of the volume to appear there, receives it, and restores
it when it is the last to have appeared here and came from another host.
What the volume held is kept as a snapshot first. A host that does not
answer refuses the mount, and the error says how to mount without
asking: pause the line.
- The last container to stop snapshots the volume and sends that
snapshot, waiting for a scheduled replication under way so that what
leaves is the final state.
- While no container uses the volume here, each scheduled round fetches
what appeared on the other host, without restoring it, so that a mount
receives a difference and not a whole volume within the thirty seconds
Docker gives it.
- A send refuses to bury a history this host never saw: a host that
crashed with unsent writes and comes back after the application ran
elsewhere keeps those writes aside, and the work done elsewhere wins.

**Every `replicate:<host>` line already in `schedule.csv` changes
meaning with this version**, and the changelog says so.

**What had to move underneath**

- The order of snapshots is read from the order BTRFS created them in,
and whether one came from another host from its `Received UUID`, never
from the date in the name: a host whose clock runs ahead can no longer
pass its copy off as the most recent one. `buttervolume receive` follows
the same rule.
- A restore keeps what the volume held the way a snapshot is kept:
nothing new when the volume is unchanged since its last snapshot,
nothing at all when it is empty, and nothing done when the volume
already holds the snapshot asked for. `VolumeBackup` is always in the
answer, and a `Restored` field says whether the volume was replaced.
- A snapshot is compared with the last snapshot taken of the volume
here, not with a received one.
- `buttervolume replicate <host> <volume>` snapshots and sends in one
step, through a new `/VolumeDriver.Replicate` endpoint the scheduler now
calls; the replication lock lives in the plugin, where the unmount needs
it.
- A volume can ask for its scheduled jobs as it is created, `-o
replicate:node2=1`, which is how a Swarm service says once what happens
to its volume on whatever host it lands on. An option nobody reads is
now refused.

The README has a new section, "Move an application between hosts", and
AGENTS.md says that a mount the user scheduled may replace a volume,
keeping what it held.

**Verified**

- `./test_local.sh`: 129 passed, 11 skipped (the ones over ssh).
- The whole suite in the plugin image, ssh included: 140 passed, nothing
skipped, nothing failed.
- Two BTRFS facts the design rests on were checked on a real filesystem
before writing the code: a writable snapshot of a received subvolume
clears its `Received UUID`, and the size of a BTRFS directory is zero
exactly when it is empty.

**Not verified here**, and worth doing on a machine where the plugin can
be replaced: two containers on one volume as seen by Docker, a mount
refused as seen by Docker and Swarm, and the duration of a first mount
against `docker plugin enable --timeout`. The plugin enabled on the
machine this was written on holds volumes, and a second one would share
its directories and its ssh port.

The design was reviewed before implementation by a separate session,
which found the four problems the plan now records: dates versus
creation order, "never written" versus "empty", one Mount per container,
and the option read after the "volume already exists" return. One more
surfaced while writing the tests: fetching ahead of the mount made the
send barrier pass by mere presence of the name, so the barrier reads the
BTRFS lineage instead.

Supersedes #69, whose four side fixes were merged earlier as #110, #111,
#112 and #113.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-reviewed pr-cycle ran

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant