Repository navigation
Asynchronous mirror: a tested implementation of "deferred redundancy" for a two-way mirror, looking for review #19890
johnpwhite
started this conversation in
Ideas
Replies: 1 comment 11 replies
|
While interesting one question: will the slow mirror side still queue reads? Bonus question: what is the reasoning behind the thumbs down on this? |
11 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
This idea has come up several times: #15886 (deferred redundancy), #11982 and #4879 (a write-back cache), and #1742 and #11234 (read from the fast side of an uneven mirror). As far as I know, none of them has a patch. I built one to see whether the idea can be done safely inside the ZFS on-disk model, and I would like to know whether the maintainers would consider it (and would likely want to rework it, given it is AI-written code; see below), and whether there is demand from the user base.
I regularly experiment with slightly older tech. Recently I wanted to set up a farm of old micro desktops as representative servers for the SDLC (build, dev, test, demo etc.), but I only had enough SSDs for a single disk in each, and lots of HDDs. I wanted some disk protection, but setting up snapshot scripts felt archaic, so I went looking for near-real-time replication, to no avail.
I had no idea how to code ZFS (I was a coder for a bank a long time ago, but that was front-end and back-end app development for online banking), so I gave the challenge to Claude Opus 5.5 and Fable 5.1 and 5.2, with some Codex thrown in for token management and diversity of "thought".
How this was built
I designed this and made the design decisions; the code, tests and man pages were written with AI coding agents, under my direction and review:
What it does
One child of a two-way mirror is the leader, normally an SSD. The other child is the lagging child, normally a hard disk. A write completes when the leader has it. A worker copies each transaction group (txg) to the lagging child, in txg order. So the lagging child always holds a complete, older, importable state of the pool.
The use case: an SSD for speed plus a large hard disk for protection, without the hard disk's latency on every synchronous write. The hard disk is at most a few seconds behind, and the lag has a hard limit.
Numbers
Test VM (8 CPUs, 8 GB, Linux 6.18); the "SSD" and "hard disk" are file-backed virtio disks on a real SSD and a real hard-disk pool; fio, 45 s per run, 3 rounds, mean shown. Full report and raw data on request.
fsynceach (writes/s)Limits: virtual disks, one host, short runs. The ratios are the useful part.
Architecture
ms_holdingtree, notms_freeing. The range stays allocated in the space maps, as a checkpointed block does, so load, unload, condense and verify need no change.ms_freeing, oldest first. The release joins a txg that the pool syncs anyway; it never forces one, so the uberblock ring and the rewind history are not used up.zfs_vdev_async_mirror_max_pending_bytesor..._max_pending_txgs, writers are delayed indmu_tx_delay(), as for dirty data.spa_sync()never waits for the copy.async_mirror=off|laggingon a leaf turns it on and off live. The read-only propertyasync_mirror_statereportsoff,starting,behind,current,stopping,frozen,failed, orleader(on the other child).org.openzfs:async_mirror, depends onspacemap_v2. It is not read-only compatible: a read-only import does not load the DTLs, so older software could read stale blocks from the lagging child. The feature is active only while the async mirror holds state, and returns to enabled when the copies are equal.max_pending_bytes,max_pending_txgs,copy_limit,release_limit,read_retries, and two test-only switches,suspend_copyandsuspend_release.Limits and how it degrades
multihostor a checkpoint.reguid, vdev removal and child changes (attach, detach, replace, split, offline) are refused while it is on: turn it off first.zpool import --async-mirror-keep=<guid>. Nothing is lost silently.zpool import -Tbelow the lagging child's txg erases its uberblocks first and resilvers it. A rewind at or above it resumes the copy.Tests
functional/async_mirrorof 90 tests with descriptive names (78 positive, 12 negative), among them simulated crashes during a copy, split-mirror and rewind imports, a frozen pool, ENOSPC with held space, send/receive and a leader write error. It passes 90 of 90 on a real kernel in about 9 minutes. Several are negative controls that break one rule on purpose and show that the damage is detected.zinject_args,zinject_countsandgang_blocks_redundantfailed on the base only.zpool_initialize_import_exportandraidz_expand_scrub_repairfailed on the fork only in the full run, but in 20-run reruns they fail on the unpatched base too (3 of 20 each; the fork 1-4 of 20), with the same messages. The last one is a scrub racing a raidz expansion, reported as raidz_expand_scrub_repair fails intermittently (3 of 20): scrub during raidz expansion reports a permanent error #19882.ztest -Aruns the async mirror with ganging forced on half of the blocks, pauses and restarts the copy at random, and ends some passes with the copy stopped.zloop.shnow picks-Ain one of five passes.mancheckpass. FreeBSD 14.4 built clean (0 warnings) on an earlier revision of the code.One upstream fix in the series
A resilver could issue blocks of the txg it started in, while that txg still wrote them; with a failing write in the same txg, the resilver could miss a block. The series has a separate commit that caps a resilver at the txg it starts in and schedules a deferred resilver for the rest, with its own test (
zpool_resilver_failing_writes). It can be reviewed and taken on its own.Next steps (not in this series)
Where the code is
Branch
async-mirrorin my fork, 10 commits on current master, each one builds on its own: master...johnpwhite:zfs:async-mirrorzio_checksum: verify a block against its block pointer without a ziodmu_traverse: addtraverse_root()to walk the tree under a given rootzio: move the body ofzio_flush()intozio_flush_impl()vdev_label: move the uberblock write of one leaf out ofvdev_uberblock_sync()dsl_scan: do not let a resilver issue blocks of the txg it starts in (the separate fix above)zdb: account for and print the records of an asynchronous mirrorztest,zloop: exercise the asynchronous mirrorasync_mirrortest groupCommits 1-4 are small refactors with no change in behaviour; commit 5 stands alone. I have not opened a pull request: I would rather hear first whether this is wanted, and in what shape.
Questions for the maintainers:
vdev_mirrorthe right place, or should it be its own vdev type?async_mirror=laggingthe right interface?Thanks for reading. I am happy to split this, change the design, or answer questions.
All reactions