MeshXFS 0.90.130

⚠️ Advisory, updated 2026-10-20: three defects found while validating 0.90.131

Testing of the next release found three defects in configurations that this
release claims. Run MeshXFS on test data only, keep backups, and after a test
check the volume with chk_mxfs -v <device> while it is unmounted on every
node.

1. Network lock manager (net/mesh: the package default,
   force_transport=1, and the DRBD dual-primary pair): a node can shut its
   filesystem down after a peer is fenced and rejoins. On a two-node
   multipath cluster with about 40,000 files in use, a full page of the lock
   ledger reached the inode lock as an error instead of a wait. The surviving
   node, then the node that had rejoined, shut their filesystems down, and the
   cluster stopped serving until it was mounted again. The failed reads we
   inspected were I/O errors, not wrong data. 0.90.130 has the same code path.
   The cause is identified; the fix ships in 0.90.131 once it passes
   verification.
2. On-disk locks (disk/caw) on an eight-node multipath cluster: one file
   was left named by two directory entries while its link count is 1. The
   filesystem check after the test suite found it; the rest of the volume
   checked clean. During a test that repeatedly took one node's storage path
   away, another node wrote an out-of-date copy of a small directory over a
   newer one, which brought back an entry that had been removed; the inode
   number was later reused for a new file. 0.90.130 has the same code path.
   The cause is identified: a node that held the directory only for reading
   wrote it back. The fix has passed a directed test and ships in 0.90.131
   once it passes verification.
3. Network lock manager (net/mesh): a node can shut its filesystem down
   when its peer stops responding while it creates a file. When a create
   reuses an inode number at the moment the other node asks for that inode,
   the create can wait about 6 seconds behind this node's own release of it
   while holding the allocation group it is creating in. On a healthy pair
   that pause is all that happens. On a two-node multipath cluster whose other
   node stopped responding during the pause (the test froze it), the surviving
   node shut its filesystem down about 16 seconds later instead of waiting for
   its peer to be declared dead, and the cluster stopped serving until it was
   mounted again. 0.90.130 has the same code path. The cause is identified and
   the fix has passed its first directed tests; it ships in 0.90.131 once it
   passes verification.

If you hit any of these, or chk_mxfs reports anything, please open an issue at
https://github.com/sshoecraft/meshxfs/issues with every node's kernel log
(dmesg) and the chk_mxfs output. The README carries the current state of
this advisory.

---

2026-10-16 — 0.90.130 — Stronger Recovery, Broader Platform Support

What changed for someone upgrading from 0.90.129:

• The network lock manager over multipath, at 2, 4, 8 and 16 nodes, and
  over a single path at 8 and 16 (0.90.130). A node that loses every
  storage path is fenced and its journal replayed while the others keep
  writing; every other node now refuses the fenced node's late lock
  releases before recovery collects its locks, so none of them can land
  afterwards and leave the filesystem refusing every write.
• Ubuntu 26.04 and Debian 12 are supported (0.90.130): the .deb, on
  Ubuntu 26.04's kernel 7.0 and Debian 12's 6.1.
• Fixes (0.90.130): on kernels before 6.5 (Debian 12) every buffered
  write now advances the file position (it rewrote the same offset); a
  node frozen by a backup or snapshot tool (fsfreeze) answers another
  node's lock request without a kernel warning, which on a host that sets
  panic_on_warn was a crash; on kernels before 6.6 a userspace thaw no
  longer ends the freeze a node holds while another node joins.

Released configurations and what 0.90.130 was validated on:

configuration	nodes	storage	validated on 0.90.130
2/net/mesh/drbd	2	each host's own disk, DRBD 8.4 dual-primary	tests/drbd_release_verify.sh; two physical Proxmox VE 9 hosts, a network cable pulled under load and plugged back in
2/net/mesh/direct	2	one shared iSCSI LUN, network lock manager	the 31-row suite on the test rig; installed packages
4/net/mesh/direct	4	one shared iSCSI LUN, network lock manager	the 31-row suite on the test rig; installed packages
8/net/mesh/direct	8	one shared iSCSI LUN, network lock manager	the 31-row suite on the test rig
16/net/mesh/direct	16	one shared iSCSI LUN, network lock manager	the 31-row suite on the test rig
2/net/mesh/mpath	2	one shared iSCSI LUN over two paths, network lock manager	the 31-row suite plus eight path-fault rows on the test rig
4/net/mesh/mpath	4	one shared iSCSI LUN over two paths, network lock manager	the 31-row suite plus eight path-fault rows on the test rig
8/net/mesh/mpath	8	one shared iSCSI LUN over two paths, network lock manager	the 31-row suite plus eight path-fault rows on the test rig
16/net/mesh/mpath	16	one shared iSCSI LUN over two paths, network lock manager	the 31-row suite plus eight path-fault rows on the test rig
2/disk/caw/direct	2	one shared iSCSI LUN, on-disk locks	the 31-row suite on the test rig; installed packages
4/disk/caw/direct	4	one shared iSCSI LUN, on-disk locks	the 31-row suite on the test rig; installed packages
8/disk/caw/direct	8	one shared iSCSI LUN, on-disk locks	the 31-row suite on the test rig; installed packages
16/disk/caw/direct	16	one shared iSCSI LUN, on-disk locks	the 31-row suite on the test rig; installed packages
2/disk/caw/mpath	2	one shared iSCSI LUN over two paths, on-disk locks	the 31-row suite plus nine path-fault rows on the test rig
4/disk/caw/mpath	4	one shared iSCSI LUN over two paths, on-disk locks	the 31-row suite plus nine path-fault rows on the test rig
8/disk/caw/mpath	8	one shared iSCSI LUN over two paths, on-disk locks	the 31-row suite plus nine path-fault rows on the test rig
16/disk/caw/mpath	16	one shared iSCSI LUN over two paths, on-disk locks	the 31-row suite plus nine path-fault rows on the test rig; three Proxmox VE 9 hosts on two paths (path faults, live migration, datastore operations, HA power loss), kernels 6.17 and 7.0

The test rig is Ubuntu 24.04 (kernel 6.8.0-101-generic). "Installed
packages" means the release packages on two nodes of each supported
platform: Proxmox VE 9 (kernels 6.17 and 7.0), RHEL 9.8, Ubuntu 24.04,
Ubuntu 26.04, Debian 12 and Debian 13. Every host in a cluster needs the
same MeshXFS version; hosts on different kernels are fine.

0.90.130 in detail

The work this version is for: the cluster-wide seal barrier that keeps a node
which has lost every storage path from retiring a lock after recovery sealed
it (D-NETMESH-MPATH-ALL-LOST-SURVIVORS-STALL-131S, which holds
4/net/mesh/mpath, 8/net/mesh/mpath and 16/net/mesh/mpath out of the
release), and every platform check and test running on a lab VM built here
with mkosimage and held through the VM pool, never in a container or on a
downloaded image.

A platform's kernel build check runs on that platform's own lab VM.
scripts/vm_kbuild_check.sh <platform> takes the first node of the
platform's verification set, holds it in the VM pool for the length of the
build (tests/lib/runlock.sh vmhold_take), starts it if it is off, copies
the module source staged as the packages stage it (packaging/common.sh), and
compiles it against the headers of the kernel the node runs (and, on
Proxmox, of each claimed kernel the node carries). It replaces
scripts/debian_kbuild_check.sh, scripts/rhel_kbuild_check.sh and
scripts/pve_kbuild_check.sh, which compiled in docker containers, the first
two kept running between runs; five of those had sat on the build host for
days. Each released platform's build_check in data/platforms.json now
names the new script, Ubuntu 24.04's included (it compiled on the build host
itself before). Rocky 9 had only a container check and has no lab set, so it
has no check until one is built. On debian13-1 the check compiled the module
against 6.12.107+deb13-amd64 in 206 s with the node started by the pool.

Debian 12 is a released platform. Kernel 6.1.0-53-amd64, the .deb. Its
verification set (debian12-1, debian12-2) was built with mkosimage from
debian-12-latest-x86_64 and upgraded to the claimed kernel; its first
round found the buffered-write position fault below, and with that fixed the
whole platform chain passed on both nodes. tests/full_verify.sh now reads
the released platforms from data/platforms.json instead of a fixed list of
four, and holds and starts each platform set through the VM pool when it is
not powering the lab itself (POWER=0). It now holds the sets when it powers
them too (POWER=1): the first POWER=1 platform run of 0.90.130 lost the
first node of the pve9, rhel9 and debian12 sets part way through their rounds,
shut off by the release helpers that scripts/release.sh's package builds and
kernel build checks leave behind for those nodes, which shut off any VM nobody
holds once their grace has passed.

Ubuntu 26.04 is a released platform. Kernel 7.0.0-38-generic (26.04.1),
the .deb. Its verification set (ubuntu2604-1, ubuntu2604-2) was built with
mkosimage from ubuntu-26-latest-x86_64 and set up by the new
scripts/platform_node_setup.sh: 4 vCPUs and 4 GiB like the 24.04 set, the
upgrade, DKMS and headers, and three things the image itself gets wrong. The
live-server image ships open-iscsi with its initiator name already written, so
both nodes had the same one: the target saw one initiator, each login dropped
the other node's session, and SCSI PR refused the moun...

MeshXFS 0.90.129 — Shared Proxmox Storage, Faster DRBD Setup, Stronger Recovery

2026-10-13 — 0.90.129 — Shared Proxmox Storage, Faster DRBD Setup, Stronger Recovery

What changed for someone upgrading from 0.90.124, the last published release
(0.90.125 to 0.90.128 were never published; their work ships here):

• Proxmox VE 9 hosts share one SAN LUN as their datastore, over
  multipath (0.90.129). Three hosts on one iSCSI LUN over two storage
  networks run guests on it: live migration with the guest writing
  throughout, HA restart after a host loses power, snapshots and rollback,
  backup and restore, disk moves, templates and linked clones, containers,
  and a storage path lost and returned under load. Verified on both
  Proxmox VE 9 kernels: 6.17, which the 9.1 installer ships, and 7.0, which
  proxmox-default-kernel now installs.
• A DRBD pair is set up in minutes, on whole disks (0.90.129). The
  guide gives DRBD the disk that will hold the VMs and skips DRBD's first
  copy of it, which took about an hour per 250 GB: the device is formatted
  next, so nothing on it before is ever read. Two physical Proxmox VE 9
  hosts set up this way wrote at the replication link's full speed and
  passed live migration, datastore operations, a full disk, a host reset
  under load, a host leaving and returning cleanly, and the 31-row suite,
  on both kernels, and through a rolling kernel upgrade with one host on
  each.
• A DRBD pair whose first host dies recovers, given a QDevice (0.90.128):
  authority=quorum in /etc/mxfs/drbd-fence.conf lets corosync's vote
  decide which host carries on.
• Fixes to locking and recovery (0.90.128, 0.90.129): a host that dies no
  longer stalls the others for a minute; a new lock tenure is no longer
  taken for one already released; a waiter registration that fails no
  longer leaves its mark on the lock; a fenced host reports being fenced,
  not corruption; a new file on a reused inode number no longer crashes its
  creator; a shared mmap write no longer hangs on RHEL 9; a full DRBD
  filesystem refuses a write at once instead of hanging the writer.
• Installing on a small host no longer runs it out of memory (0.90.129):
  building the module checks which kernel features the host has, and those
  checks now run a few at a time instead of all at once. A check that cannot
  finish stops the build and names itself, where before it could quietly
  build the module wrong.
• vSphere shared disks (0.90.128): MeshXFS mounts a multi-writer VMDK
  or a physical-mode RDM with fence=vcenter, fence=manual or
  fence=quorum. Not yet a released configuration.
• The module's messages say MeshXFS (0.90.128), not XFS, and the
  product is named MeshXFS throughout (0.90.125).
• Proxmox VM disk settings are documented (0.90.129): the README and the
  DRBD guide say to give VM disks on MeshXFS cache=none, the Proxmox
  default and the mode every Proxmox validation ran with.

Released configurations and what 0.90.129 was validated on:

configuration	nodes	storage	validated on 0.90.129
2/net/mesh/drbd	2	each host's own disk, DRBD 8.4 dual-primary	tests/drbd_release_verify.sh; two physical Proxmox VE 9 hosts set up from docs/drbd-setup.md (the 31-row suite, live migration, datastore operations, a full disk, a host reset and a clean departure, kernels 6.17 and 7.0)
2/net/mesh/direct	2	one shared iSCSI LUN, network lock manager	the 31-row suite on the test rig; installed packages
4/net/mesh/direct	4	one shared iSCSI LUN, network lock manager	the 31-row suite on the test rig; installed packages
2/disk/caw/direct	2	one shared iSCSI LUN, on-disk locks	the 31-row suite on the test rig; installed packages
4/disk/caw/direct	4	one shared iSCSI LUN, on-disk locks	the 31-row suite on the test rig; installed packages
8/disk/caw/direct	8	one shared iSCSI LUN, on-disk locks	the 31-row suite on the test rig; installed packages
16/disk/caw/direct	16	one shared iSCSI LUN, on-disk locks	the 31-row suite on the test rig; installed packages
2/disk/caw/mpath	2	one shared iSCSI LUN over two paths, on-disk locks	the 31-row suite plus nine path-fault rows on the test rig
4/disk/caw/mpath	4	one shared iSCSI LUN over two paths, on-disk locks	the 31-row suite plus nine path-fault rows on the test rig
8/disk/caw/mpath	8	one shared iSCSI LUN over two paths, on-disk locks	the 31-row suite plus nine path-fault rows on the test rig
16/disk/caw/mpath	16	one shared iSCSI LUN over two paths, on-disk locks	the 31-row suite plus nine path-fault rows on the test rig; three Proxmox VE 9 hosts on two paths (path faults, live migration, datastore operations, HA power loss), kernels 6.17 and 7.0

The test rig is Ubuntu 24.04 (kernel 6.8.0-101-generic). "Installed
packages" means the release packages on two nodes of each released
platform: Proxmox VE 9 (kernels 6.17 and 7.0), RHEL 9.8, Ubuntu 24.04
and Debian 13. Every host in a cluster needs the same MeshXFS
version; hosts on different kernels are fine.

0.90.129 in detail

• Three Proxmox VE 9 hosts on one LUN over two storage networks.
  scripts/san_net.sh now configures hosts that use ifupdown (Proxmox): each
  gets a fixed address on each storage network (.2-.9, below the DHCP pool,
  so no lease can rewrite the host's DNS), and san_net.sh link <host> a|b down unplugs one path from the hypervisor side for path-loss tests.
• scripts/pve_caw_lun.sh presents a LUN over several portals
  (PORTALS="192.168.121.1 192.168.123.1"): it installs multipath-tools,
  writes the verified multipath and iSCSI settings
  (tools/mpath_settings.sh), logs in on every portal and reports the
  multipath map (dev=/dev/disk/by-id/dm-uuid-mpath-<wwid>), which must have
  every path active. A path whose disk appeared before multipath-tools was
  installed has no udev record of the properties its rules add, and
  multipathd blacklisted it ("udev property missing"), so the map came up with
  one path of two; the script now replays udev on every path of the target
  first. Each session is bound to the NIC that reaches its portal (an iSCSI
  iface mxfs-<nic>), as the rig's LUN pool does: unbound, a session whose
  cable was pulled reconnects through the default route, since clyde answers
  for the portal address on that route too, and a path taken away is not
  lost. An unbound session left by an earlier login is replaced only while
  no MXFS volume is mounted.
• The multipath fault rows (tests/mpath/) run on Proxmox hosts.
  tools/mxfs_sshpass.sh reaches a node that no resolver knows by the
  address tools/mxfs_lab.sh resolves for it (its lab-file addr line), so
  the rows can name the fixed-address hosts pve9-5/6/7 like rig nodes; the
  path probe is handed the row's device instead of assuming /mnt/shared;
  MXFS_MOUNT_OPTS gives remounts and rejoins the options the cluster was
  mounted with (transport=caw on Proxmox, whose packaged module forms TCP
  clusters); and a rebooted node with no rig module (/root/mxfs.ko.prep)
  loads the installed one. Every row's start gate mounts clyde's /src on
  a node that lacks it: a Proxmox host has no fstab entry for it, so one
  restarted between rows came back without the harness and the next row
  aborted at its first capture.
• Verified on pve9-5/6/7 (Proxmox VE 9, three nested hosts, the 30 GiB
  LUN pve-caw-02 on two paths, build srcversion 62EB76441F21B8AE0B6BD45):
  • live migration (tests/pve_live_migrate.sh, 6 laps round the three
    hosts, the guest writing throughout): 12-14 s a move, downtime 65-97 ms,
    every file verified after every move, no bad kernel line;
  • path loss and return under load on all three (tests/mpath/path_failover.sh,
    F1/F2, victim pve9-7): longest stall 15.3-15.8 s against a 30 s bound, no
    operation failed, no double grant, 6,143 acknowledged files read back
    from another host, reservations intact.
• scripts/pve_tree_install.sh replaces a loaded module. modprobe
  leaves an already loaded mxfs in place, so a reinstall kept running the
  old build; an unmounted host now unloads it first, and a mounted one
  refuses.
• A fenced node no longer reports corruption. A node cut off from the
  storage that took a reservation conflict (xfs/xfs_mxfs_ilock.c), or whose
  lock authority was withdrawn, shut its filesystem down as "Corruption of
  in-memory data"; it now shuts down with its own reason, "Fenced by the
  cluster (this node lost its access to the shared storage)". The two
  livelock shutdowns keep the corruption reason.
• A new lock tenure is no longer mistaken for one already released. A
  node that released a file's write lock records that tenure as cleanly
  released, and refuses to take the same tenure back as write authority. It
  matched the tenure on the file and its grant number only; but when the
  file's lock record is re-created in another slot it gets a new identity
  (its lineage) and its grant numbers start again at 1, so the first write
  lock on it looked like the released one. The node then wrote with no
  authority behind its changes, and a recovery replaying its log would have
  discarded them. It now matches lineage too, as recovery and the cluster
  lock already did (xfs/xfs_mxfs_authority.c). Seen 45 times on
  pve9-5/6/7 during the whole-fabric and path-flap rows, every one with a
  different lineage. With the fix, all six multipath fault rows pass on
  the three hosts (whole fabric, flap, a death on one path, a fenced node's
  return, every path lost, a peer withdrawn) and no host refused once.
  The fenced node in path_fenced_return now logs "Fenced by the cluster",
  and no row's evidence says "Corruption of in-memory data".
• A directory warning no longer cries clobber where nothing is
  rewritten. "P36-EVICT-LOCKED ... stale base KEPT -- clobber risk" f...

MeshXFS 0.90.124 — Multipath SAN Support, Stronger Recovery, Better Scalability

2026-10-12 — 0.90.124 — Multipath SAN Support, Stronger Recovery, Better Scalability

What changed for someone upgrading from 0.90.116, the last published
release (0.90.117 to 0.90.123 were never published; their work ships here):

• dm-multipath is released again for SAN clusters on the on-disk lock
  manager (0.90.124): 2/disk/caw/mpath, 4/disk/caw/mpath,
  8/disk/caw/mpath and 16/disk/caw/mpath, each node reaching the shared
  iSCSI LUN over two paths on separate networks. Last claimed in 0.90.51.
• Fixed on the way there (0.90.123, 0.90.124): a directory acquire that
  waited on a child's lock held by a peer waiting for that directory, which
  stalled all 16 nodes for eight minutes; a node that rejoined after being
  fenced and could not open a file, its open() retrying for ever and
  impossible to kill; an open that can never be granted now fails with EIO,
  and a killed open always returns.
• More than one MeshXFS filesystem on a host (0.90.120): the filesystems no
  longer cross each other's lock traffic, and the new mount option
  transport=tcp|caw picks each one's lock manager, so a host can keep a
  DRBD resource mounted beside a SAN LUN.
• A DRBD pair with several MeshXFS resources recovers every one (0.90.121,
  0.90.122): a dead host is fenced and recovered on each resource under its
  own evidence; one failed link cuts the peer off on every shared resource;
  the module is loaded before DRBD connects.
• Less wasted lock work under heavy VM load on a DRBD pair (0.90.117,
  0.90.118): abandoned lock answers 84 → 17, the longest queued request 39 s
  → 13 s, the longest allocation walk 46 s → 8.5 s.
• A misleading kernel warning is gone (0.90.117): P958-ACQ-IDLE-LOST.

Released configurations and what 0.90.124 was validated on:

configuration	nodes	storage	validated on 0.90.124
2/net/mesh/drbd	2	each host's own disk, DRBD 8.4 dual-primary	tests/drbd_release_verify.sh; scripts/drbd_rig.sh multi-death-test (3 resources, a host destroyed; 6 resources, one link cut); tests/pve_drbd_multi_mount_crash.sh on a nested Proxmox VE 9 pair
2/net/mesh/direct	2	one shared iSCSI LUN, network lock manager	the 31-row suite on the test rig; installed packages
4/net/mesh/direct	4	one shared iSCSI LUN, network lock manager	the 31-row suite on the test rig; installed packages
2/disk/caw/direct	2	one shared iSCSI LUN, on-disk locks	the 31-row suite on the test rig; installed packages
4/disk/caw/direct	4	one shared iSCSI LUN, on-disk locks	the 31-row suite on the test rig; installed packages
8/disk/caw/direct	8	one shared iSCSI LUN, on-disk locks	the 31-row suite on the test rig; installed packages
16/disk/caw/direct	16	one shared iSCSI LUN, on-disk locks	the 31-row suite on the test rig; installed packages
2/disk/caw/mpath	2	one shared iSCSI LUN over two paths, on-disk locks	the 31-row suite plus nine path-fault rows on the test rig
4/disk/caw/mpath	4	one shared iSCSI LUN over two paths, on-disk locks	the 31-row suite plus nine path-fault rows on the test rig
8/disk/caw/mpath	8	one shared iSCSI LUN over two paths, on-disk locks	the 31-row suite plus nine path-fault rows on the test rig
16/disk/caw/mpath	16	one shared iSCSI LUN over two paths, on-disk locks	the 31-row suite plus nine path-fault rows on the test rig

The test rig is Ubuntu 24.04 (kernel 6.8.0-101-generic, multipath-tools
0.9.4, tools/mpath_settings.sh). "Installed packages" means the release
packages on two nodes of each of Proxmox VE 9, RHEL 9.8, Ubuntu 24.04 and
Debian 13, on one path, so mpath is claimed on Ubuntu 24.04 only. Several
filesystems per host were verified with tests/multi_mount.sh on both lock
managers, with and without peers=, and with a node destroyed while it held
one of two filesystems. Every host in a cluster needs the same version.

0.90.124 in detail

One version for the whole of the mpath work. 0.90.123 was opened for it
and 0.90.124 opened in error for a defect that work found; both are folded
here, and nothing under either number was committed, pushed or installed.

A node that rejoins after being fenced cannot open a file its new mount has never touched (16/disk/caw/mpath path_all_lost)

A new in-core inode no longer inherits a terminal release state from the
memory it was allocated in (xfs/xfs_mxfs_dlm.c). xfs_inode_alloc takes
its inode from the slab without zeroing it, and mxfs_dlm_inode_init reset
the release-defer episode but never assigned i_mxfs_rel_state. WEDGED can
be entered and never left, so an inode whose memory last held a wedged inode
started out wedged: every lock acquire on it was refused, and open()
restarted forever without a grant. Measured on 16/disk/caw/mpath
path_all_lost (0.90.123,
tests/evidence/board_20261011T225324Z-g16_path_all_lost/): test16 lost both
paths, was fenced and withdrew; its unmount wedged 125837620 at teardown
(P-INODE-WEDGE ... teardown=1, 437 s). It remounted at 502 s, and at 506 s
the first acquire on 125829259, an inode the new mount had just read, was
refused as wedged (P-INODE-WEDGE-FENCE) with no wedge of it in that mount.
Its load spun in mxfs_dlm_open_protect (P95-OPEN-PROTECT-RESTART, state R)
until the row ended, and could not be killed, which aborted the next row
(path_peer_withdrawn: STRAY killed=1 left=1). The initializer now starts
every inode ACTIVE, and says when the memory it was given held a wedge
(P-RELSTATE-RECYCLED).

dbg_relstate_wedge_on_free (test only, xfs/xfs_icache.c) frees every
inode with its release state WEDGED, so the next inode allocated in that
memory meets what test16 met; path_all_lost reaches it only when a teardown
wedge happens and the slab then hands that object to an inode the remounted
node opens (lap 2 of the same 16-node run had no wedge at all).
tests/relstate_recycle_inject.sh sets it on one node, drops that
node's inode cache and reads and appends to files a peer created; it passes
only when the poisoned memory was reached (P-RELSTATE-RECYCLED ≥ 1) and no
acquire was refused as wedged. On 0.90.124 it passed: 149 inodes freed
poisoned, new inodes allocated in that memory, no wedge refusal, 200 of 200
reads and appends, the slowest pair 18 ms. On a control build without the
assignment the same test refused 210 acquires as wedged and its cat and
sh stayed in the kernel through timeout -s KILL until the node was
reset. D-RECYCLED-INODE-INHERITS-WEDGED-REL-STATE is removed from the
defect queue.

An open that cannot be granted fails, and a killed open leaves
(xfs/xfs_mxfs_open.c). mxfs_dlm_open_protect restarts an open that
lost the race for its grant, treating every missing grant as contention, and
its only way out was the mount shutting down. An acquire refused outright —
a wedged inode, refused for the life of the mount — therefore restarted
forever, and the loop never looked for a fatal signal, so the task could not
be killed: that is what aborted path_peer_withdrawn above. A wedged
inode's open now fails with EIO (P95-OPEN-REFUSED), as the admission gate
fails every other operation on it, and a fatal signal ends the wait
(P95-OPEN-KILLED), both at each restart and in the 30 s demote wait.
tests/open_wedged_inode.sh (test knobs dbg_openprotect_wedge_ino,
dbg_openprotect_refuse_ino): on a build with the knobs and without the
change, the open of a wedged inode never returned and survived kill -9,
and an open held in contention survived SIGKILL until the contention ended;
with the change the first fails in 55 ms and the second is gone within 2 s
of SIGKILL. D-OPEN-PROTECT-RESTART-LOOP-UNBOUNDED-AND-UNKILLABLE is removed
from the defect queue.

The 4-node mpath sequential-write slowdowns were clyde's disk, not
MeshXFS (D-4NODE-MPATH-SEQ-WRITE-COLLAPSE, disproved and removed). With
clyde's / back to 74% used, six host-sampled fio_perf laps on 4
disk/caw/mpath nodes passed at 1801-3503 MiB/s sequential write (the low
runs read 243-761, all taken with / 92-100% full), and in every lap the
guests' time per write followed the host disk's own (host p95 74 ms → guests
47-120 ms; 508 → 312-648). In the earlier low run every node had kept 32
writes in flight at 60-970 ms each. fio_perf_vs_xfs now reads 189% of
native XFS for sequential write. It is not over: in the release gates'
repeats on 8 nodes one run of ten read 665 MiB/s (62%) at a host load of
11, with / 74% used, which the full-disk explanation does not cover; that
is D-MPATH-SEQ-WRITE-LOW-RUN-RECURS-WITH-DISK-NOT-FULL, open. That run had
nothing sampling the host, so every throughput row now records clyde's
pool disk and memory while it runs (run.sh): host_disk[...] (the
device's time per write, p95 and worst, and its peak MiB/s) and
host_mem[...] (peak dirty and writeback, least available), the raw
capture kept as tests/evidence/hostdisk_<config>_<row>_<time>.txt. The
first version held each row still for the sampler's whole lifetime —
run_coord waits for its node launches with a bare wait, which waited
for the sampler too — so fio ran only after the sample had ended; the
sampler is now out of the shell's job table, and a 4-node fio_perf row
takes 82 s again with its stamp agreeing with an independent sample
(13,135 MiB against 13,137 in /proc/diskstats; peak 1883 MiB/s, p95 575
ms per write). tests/mpath/fio_host_timeline.sh runs rows with the
pool disk sampled from outside run.sh and prints the heavy-write seconds
beside run.sh's timestamped lines.

With that stamp the low run came back in the release gates' repeats on 8
nodes (seqW 348 MiB/s) and was clyde's disk: through the whole
sequential-write phase the pool NVMe held about 270 writes in flight and
completed 58-440 MiB/s at 300-1250 ms per write, where a pa...

MeshXFS 0.90.116 — two-host Proxmox VE on DRBD: VM disks no longer freeze on slow SSDs, and a host that leaves, fails or rejoins comes back cleanly

For a two-host Proxmox VE pair running MeshXFS on DRBD dual-primary
(2/net/mesh/drbd), upgrading from 0.90.107, the last release:

• VM disk writes no longer freeze under heavy load on slow disks. With
  seven AlmaLinux installs running at once on two physical Proxmox VE 9 hosts
  with 2011-era SATA SSDs under DRBD, a guest write that needed new space
  could hold its disk image locked for 30 s to a minute. The host reported
  hung tasks. On the same disks and load in 0.90.116, no task hung and all
  seven installs finished. No write ever failed and no data was lost
  (0.90.115, 0.90.116).
• A host that drops out and rejoins comes back cleanly. When one host's
  mount is declared dead and recovered, and then remounts without a restart,
  its mount no longer waits minutes to be admitted (0.90.113). Its first reads
  no longer take 69 to 95 s (0.90.114).
• Shutting down or unmounting one host is quick, however busy the
  filesystem was. A clean departure is bounded in time whatever it was
  serving (0.90.110). It no longer waits out the other host's rejoin
  (0.90.109).
• When one host leaves, the other takes over everything it served,
  whichever of the two it is (0.90.111), and leaves nothing stranded
  (0.90.113).
• Live migration between the two hosts is tested (0.90.112).
• Both hosts deleting VM disk images at once, still open, leave nothing
  behind (0.90.116).
• Fewer false alarms in the kernel log. Messages a healthy, loaded host
  printed as warnings or as recovery are now debug probes (0.90.108).

Validated on:

• the two physical Proxmox VE 9 hosts (both their Samsung 850 EVO and their
  older Kingston SSDs);
• a nested Proxmox VE 9 pair through every failover step: crashes, power
  cuts, restarts, promotions, a slow heartbeat and withdrawals;
• the DRBD test rig;
• the six shared-LUN configurations on the test rig;
• the installed packages on Proxmox VE 9, RHEL 9.8, Ubuntu 24.04 and
  Debian 13.

Every host in a cluster needs the same version. The README lists the
configurations this release claims.

What was measured and changed, release by release, follows here and in the
0.90.108 to 0.90.115 sections below.

0.90.116 in detail

An AG lock's no-queue request does no ledger I/O and never waits for a
ledger page's lock. 0.90.115's fix answered would-block only when the
ledger page's preparation returned -EAGAIN. Its final run on the physical
pair's slow spare disks still froze a guest's writes for over 30 s on pve1,
by another route. The allocator's trylock, holding the VM image's ILOCK, slept
in mxfs_tauth_ledger_activate on a page lock that a thread writing the
page to the slow disk held. QEMU's other writers and the direct-I/O
completion worker waited on the ILOCK behind it, and the kernel reported hung
tasks. Two narrower answers were measured and rejected:

• Answering would-block for any page not yet ready stopped the freeze.
  But only trylocks touched some pages, so nothing activated them: one
  AG's page was skipped 26 times, and pve1's installs missed packer's
  45 minutes.
• Answering would-block only for a page another thread held left the
  trylock doing its own page reads and commit: 2.7 s under the ILOCK in one
  leg.

Now such a request is decided on at once only when its page is this node's,
imported under the current generation, and its lock free this instant
(mxfs_tauth_ledger_page_ready_nowait, a trylock). Any other page answers
would-block and is marked. The hand-off worker, which holds nothing,
prepares the marked pages every 500 ms, ahead of its hand-off pass
(mxfs_dlm_prepare_wanted). So the page is ready for the next try.
dl_noqueue_ledger_nowait (1; 0 restores the request preparing the page
itself, for A/B only). New probes: P-ACQ-NOQUEUE-LEDGER-NOTREADY,
P-PREPARE-WANTED (debug).

Verified two ways:

• Userspace: new case 18 of tests/tauth/dlm_ledger_test. With the knob
  at 0, a no-queue request waits 1537 ms behind a held activation. With it
  at 1, the request answers in 0 ms. On an unready page nobody holds, it
  answers in 0 ms with no ledger commit; the worker's pass then prepares the
  page, and the next no-queue request is granted.
• Physical pair: MeshXFS on DRBD on the TRIMmed Kingston spare disks, 7
  concurrent AlmaLinux installs. The would-block path ran on both hosts
  (pve1 24, pve2 17), and the worker prepared those pages in 5 and 6 passes.
  • No slow preparation: the knob-0 legs on these disks had up to 4.3 s.
  • No hung task, I/O error, warning or withdrawal on either host. The
    knob-0 legs had hung tasks.
  • 7 of 7 installed: pve1 2311–2441 s, pve2 1856–2174 s.
  • The guests' flush means were 40 to 54 ms, against 47 to 105 ms in the
    earlier legs.

Only AG locks take either no-queue answer. The inode allocator's no-queue
tries probe one inode number after another. With 0.90.115's would-block
applied to them too, an early would-block skipped the number. On
4/net/mesh/direct the allocation witness then found no inode chunk filled
whole and could not grade the run (4 of 4). Restricted to AG locks, 3 of 3
graded and chk_clean read CLEAN.

tests/pve_image_delete_logs.sh passes. Both hosts delete three
fragmented images at once while still holding them open. The four runs on
0.90.116 (256 and 512 MiB) left every file gone on both hosts, with no
warning-level line, unrecovered-unlinked message or trace in either kernel
log. The three earlier runs failed: one on lines that 0.90.108 made probes,
and one, "still sees 2 files", on the test's own count. grep -c img matched
the two directory headers ls prints. It now matches ^img.

Defect queue:

• the allocator's trylock slept on a ledger page lock while holding a VM
  image's ILOCK (D-NOQUEUE-AG-TRYLOCK-BLOCKS-ON-LEDGER-ACTIVATE-MUTEX-HOLDING-ILOCK)
  — FIXED AND VERIFIED by the change above (the stacks named the cause; case
  18 proves the behaviour; the Kingston leg exercised it with no slow
  preparation and no hung task).
• 0.90.115's would-block skipped inode numbers, so the allocation witness
  could not grade (D-NOQUEUE-WOULDBLOCK-SKIPS-INODE-NUMBERS-WITNESS-CANNOT-GRADE)
  — FIXED AND VERIFIED. Every no-queue request: 4 of 4 runs aborted,
  tests/evidence/alloc_witness/20261010T040221Z-g4 and three more,
  against 0 of 21 on 0.90.114/0.90.107. AG locks only, build
  069D73E31601F207DF824C6: coverage PASS 3 of 3 (20261010T044929Z-g4,
  045143Z-g4, 045355Z-g4), chk_clean CLEAN.
• a survivor's takeover skipped, for good, the pages the view routed to a
  peer that had just joined (D-TAKEOVER-SKIPS-PAGES-ROUTED-TO-A-JUST-JOINED-PEER-FOR-GOOD)
  — FIXED AND VERIFIED (fixed in 0.90.113). Before: skipped=413 noinc=413,
  with 397 pages left. After, in both 16-step nested-pair suites
  (tests/evidence/pve_pair_failover/20261009T145710Z-192.168.120.211,
  20261010T031509Z-192.168.120.211): withdraw and power-cut steps logged
  skipped=0 with noinc 39, 91, 130 and 71, and every withdraw census
  found no page left under the dead incarnation.
• New: a no-queue AG request whose master is the peer waits a whole 1 s
  attempt when the master is slow to answer, AG after AG, under the ILOCK
  (D-REMOTE-NOQUEUE-AG-TRYLOCK-WAITS-A-FULL-ATTEMPT-PER-AG-UNDER-ILOCK).
  In the same leg, QEMU's writers on pve1 met 61 such waits, in runs of 5 or
  6 a second apart: 5 to 6 s per allocation pass. No hung task.
• New: P958-ACQ-IDLE-LOST reads as a wait of idle_ms, and is not one
  (D-IDLE-LOST-WARNS-OF-A-LONG-WAIT-FOR-A-NO-QUEUE-TRY-ABANDONED-MINUTES-AGO).
  All 27 in that leg are the leftover records of unanswered no-queue tries,
  each logged exactly idle_ms after its try. Read as AG waits of up to 20
  minutes, they had been taken as evidence against the first rejected
  answer above. That answer's install timeouts and flush times still rule it
  out.
• The pair's corosync QDevice is not voting (recorded in
  D-DRBD-PAIR-NO-EXTERNAL-QUORUM-SO-P1-CANNOT-TAKE-OVER-WHEN-P0-DIES). Its
  qnetd host, 192.168.1.108, has been unreachable since at least
  2026-10-05, so the pair has 2 of 3 votes. No takeover design can rest on
  that vote until it exists.

tests/pve_build_leg_kmsg.sh turns on and counts the new probes, and
counts P-ACQ-NOQUEUE-UNANSWERED and P958-ACQ-IDLE-LOST.

MeshXFS 0.90.107

2026-10-08 — 0.90.107 — MeshXFS on DRBD dual-primary is released again, verified on two physical Proxmox VE 9 hosts; seven configurations are claimed

What this release claims. 2/net/mesh/drbd again, withdrawn on
2026-10-06 and now verified three ways:

• on two physical Proxmox VE 9.1 hosts (kernel 6.17.2-1-pve, DRBD 8.4.11)
  under seven concurrent VM installs;
• on a nested Proxmox VE 9 pair through 15 crash, power-cut, restart,
  promotion and withdraw tests;
• on the test rig by tests/drbd_release_verify.sh.

It also claims six shared-LUN configurations on the direct attachment,
verified again on this build: 2/net/mesh/direct, 4/net/mesh/direct, and
2/disk/caw/direct through 16/disk/caw/direct. It does not claim
8/net/mesh/direct, 16/net/mesh/direct or any mpath configuration.
Defects found after 0.90.51 that can lose data or stop a node reach them
(tools/defects.py <configuration> --release); the README lists them under
"Released in an earlier version". The last published release was 0.90.42, and
every change since is in the entries below, 0.90.43 to 0.90.107. For DRBD the
substance is in 0.90.55 to 0.90.106: what eight concurrent VM installs on the
physical pair found, fixed one cause at a time.

The DRBD rig no longer releases a fenced node before the survivor's MeshXFS
has proved it excluded. The fence and split tests in scripts/drbd_rig.sh
run with MeshXFS mounted, and they released the loser's inhibit as soon as DRBD
on the survivor had settled. That was 1–2 s after the fence in
tests/drbd_release_verify.sh on 0.90.106 (clyde's authority log: fenced
10:03:06, released 10:03:08). The survivor's MeshXFS recovers a dead peer's
journal slice only after its witness sees the peer off and still inhibited
under the fence's episode. Every answer it got was running, inhibit none,
because the loser was already booting again. So it never certified the
death, kept the slot unreplayed with its locks frozen (fail-closed), and the
next step's unmount on it never finished: test1 would not release mxfs,
and the remaining steps of the run failed behind it. That refusal is the
designed behaviour. A product pair's guard releases a fenced peer only after
the survivor's recovery completes, as tests/pve_pair_failover.sh
(answering-restart) checks. Both tests now wait, when MeshXFS is mounted, for
the survivor's P163-RECOVERY-COMPLETE and its P238-DRBD-FENCE-WITNESSED
before releasing, as the death test and rejoin_node already did. With it,
each survivor recovered the fenced peer's slice under the witness 12 s into
its test, and the unmounts after both went through.

The release chain no longer starts boards that cannot run.
tests/release_verify_chain.sh on 0.90.107 ran three of its six boards.
It powered every board group up and started all the boards at once. The
guests' boot put about 45 kernel lines/s on clyde for a few seconds (SCST
session threads, bridge ports), above the host preflight's 40/s gate, so
2/disk/caw/direct's run.sh was refused. The groups also overlap: g16
holds every node of g8, g4, g2 and g2b. The 16-node board and
2/net/mesh/direct therefore found their nodes held by the other boards'
runs and refused too. The chain now waits, at most 120 s, for the host
preflight to pass before any board starts. It also runs the boards in
waves: a board joins the first wave whose groups share no node with its
own, and the waves run one after another. For this release that is the
16-node board with 4/disk/caw/direct on g4b, then the other four.
Verified by a full run of the fixed chain on 0.90.107 (CLAIM=16 FULL=0 POWER=1 LOWER="16 8 4 2"): every rig group was powered off and up again,
the settle wait passed before any board started, and both waves exited 0.
Each of the six boards' own logs (board_*_bL1_0.90.107.log,
board_*_bL2_0.90.107.log) holds 30 rows with no FAIL, no preflight refusal
and no node held by another run, and every board reads 31 of 31 PASS.
D-RELEASE-CHAIN-STARTS-BOARDS-INSIDE-ITS-OWN-POWER-UP-LOG-BURST is
removed from the defect queue.

2/disk/caw/direct's crash_audit row reads PASS again. It read FLAKY
because of a FAIL recorded on 2026-10-05 in which the filesystem was never
exercised: tests/tcp_death_replay.sh aborted at its first check because
test15 was running a module other than the tree's
(board_20261005T050017Z-g2b_crash_audit/oracle.log), and crash_audit.sh
scores any oracle exit other than 0 as a failed death oracle. Nine laps of
the row on this build plus the board's own run took that record out of the
11-run window; every one passed.

Verified on 0.90.107: tests/drbd_release_verify.sh passed every step
(DRBD_RELEASE_VERIFY PASS, module C930E3CE4539BDFFAAC8739): the suite,
remount, death, fence, split, resolve, pair-outage and both takeover tests.
That closes D-UNMOUNT-FREES-DLM-CONTEXT-UNDER-AN-IN-FLIGHT-RUNTIME-SB-COVER-OOPS
(the 0.90.106 fix below), whose deterministic reproduction had already been
clean 3 of 3. Also on the 0.90.106 code (36AE6F621EFE1608895C6C3 on the
Proxmox kernels):

• The nested Proxmox VE 9 pair passed all 15 steps of
  tests/pve_pair_failover.sh.
• The physical pair ran 7 concurrent VM installs for 2604 s on the shared
  filesystem. No guest I/O error, hung task, warning, withdraw or self-fence
  on either host. The slowest heartbeat swap was 8.6 s. No build finished
  within the budget: that is the pair's write capacity, an open,
  non-blocking record.

MeshXFS 0.90.42

2026-10-03 — 0.90.42 — sixteen nodes: 16/net/mesh/direct and 16/disk/caw/direct released; a give-up that reads a reused lock slot no longer leaves its waiter registration behind

The release matrix gains 16/net/mesh/direct and 16/disk/caw/direct.
Both pass the 31-row suite on this build, side by side on two 16-node rig
groups (g16 test1-16, g16b test17-32), each on an 80 GB pool LUN. The 2-,
4- and 8-node boards and the four platforms' packaged rounds were run again on
the same build. tests/release_verify_chain.sh takes its smaller node
counts from the release matrix instead of a fixed "4 2".

Three records stood between 8 nodes and 16.

A give-up reconcile handed a stale slot index (disk/caw). When a lock
wait is abandoned (grant-wait timeout, claim or convert exhaustion),
caw_drop_own_waiter clears the node's waiter and holder bits at the slot
index the acquire remembered. Slots are recycled through tombstones, so the
index is not an identity.

• The record (D-RECONCILE-SLOT-IDENTITY-UNCHECKED) was a code proof that the
  clean-up validated only the slot magic, so it could clear this node's bit in
  a slot now holding a different lock. The comparison of the slot's resource
  by content has been in the tree since 0.11.441 and had never been exercised.
• New exerciser arm, caw_samenode_selftest mode 6 (recycled): the node
  takes EX on a second reserved key, waits on the shared key behind a peer,
  and its forced give-up is pointed at the second key's slot by a one-shot
  knob (caw_inject_dow_slot).
• Control, comparison disabled: the give-up's clean-up CAS released the
  second key's EX grant (P6H-ABORT-RECONCILE at that slot).
• With the comparison, before this version: the second key's grant survived,
  but the reconcile treated "another resource here" as proof that nothing of
  ours was left anywhere, discharged the obligation, and the abandoned
  waiter bits stayed on the shared key's slot for the 8 s the arm watched
  (a_w=1 a_wex=1). A waiter nobody is behind makes every peer defer.
• Fix: a tombstone or another resource at a give-up's remembered index proves
  nothing and the obligation stands (P277-DOW-STALE-INDEX); the owed
  worker, which resolves the resource to its slot in the same pass, collects
  it. The same two findings stay terminal for the worker itself.
• Verified: all five arms PASS, 3 laps on test1/test2 and 2 laps on
  test16/test15, b_hex=1 a_w=0 a_wex=0 on every recycled run
  (tests/evidence/20261004T02*_samenode, 20261004T03*_samenode).

The same five laps are the clean exerciser run two sibling records were
waiting for, and both are removed fixed and verified:
D-SAMENODE-WAITER-CANCEL-COLLISION (one local attempt's give-up clearing a
bit another live local attempt depends on: collide, collide_late and
collide_owed keep the survivor's EX bit through the owed pass) and
D-RECONCILE-EXHAUSTION-SILENT (a clean-up that ends unconfirmed leaves its
obligation standing and the worker discharges it).

D-TAUTH-FORMATION-RAMP-LEDGER-DENY-EXHAUSTS-RETRIES-EAGAIN-0352 is
disproved. It read last_rc=-35 in the usermode ramp test as a retry
budget exhausted with -EAGAIN. -35 is EDEADLK, the master's blocked-upgrade
refusal, returned at once without spending the budget; 0.87.22 instrumented
it (every failure an EX denied at local mode NL, zero budget give-ups) and
taught the test the kernel's own handling, and removed its newer record of
the same finding. Re-measured here: 30 of 30 runs pass with no failed
request and no budget give-up, with 0 to 5 ledger denies per run
(tests/evidence/formation_ramp_30runs_0.90.42.txt).

The strand-repair row covers sixteen nodes. The first
16/net/mesh/direct board failed ag_strand_repair 11 of 16: five nodes
never released an AG during the 16-round load, so their injected strand never
fired (strands=0, writes fine, no faults) — the under-coverage the row
already documents at 32 nodes. Sixteen nodes now get 32 rounds; 16 of 16
nodes create and repair a strand in 76 s of the 240 s budget.

The hung-node test runs on a set of two. tests/tcp_peer_freeze_death.sh
checked that survivor and victim were both in the set with one pattern
wanting " S " then " V "; in a set of two the names share the single
space between them, so it refused every two-node platform set ("must both be
in the set", rc=2, before anything ran). The platform sets have been two
nodes since 0.90.41 and this is the first release verified on them. Each
name is now checked on its own.

D-MATRIX-UNMEASURED is narrowed to what its evidence still supports. It
recorded that every column but 32 nodes held cached cells from older builds
and that the network column could not be prepped. Every direct column it
named is now a measurement on this build (2, 4, 8 and 16 nodes, both
methods, each 31/31), and a release requires that. The disk/caw/pass
column it also named has no board on a current build; the record stays for
that.

MeshXFS 0.90.41

WITHDRAWN 2026-10-06: 2/net/mesh/drbd (MeshXFS on DRBD dual-primary) is not a released configuration. On two physical Proxmox VE 9 hosts, eight concurrent VM installs on the shared filesystem shut both hosts' MeshXFS mounts down and the guests whose disks were on it got I/O errors. Do not deploy MeshXFS on DRBD from this or any release until a release lists 2/net/mesh/drbd again; see the top of the README. The other configurations this release lists are not affected by this notice.

2026-10-02 — 0.90.41 — the DRBD release: a pair outage with a second failure during its recovery is finished by a resume or a takeover, never left for an operator; a pair outage on 2/net/mesh/direct no longer ends REFUSED while the other host boots

0.90.40 was committed but never published: its release validation found that
a second failure during pair-outage recovery left the volume unmountable
until an operator repaired it. 0.90.41 is the 2/net/mesh/drbd release with
those fixed.

A bootstrap whose completion failed partway is resumed and finished. Two
causes, each proven by instrument on 2/net/mesh/drbd (completion's tombstone
swap refused, then a same-boot remount):

• The orphan sweep queued at mount init retired 42 of the adopted victim K's
  ledger records half a second after K's pre-replay verification passed: K
  has no recovery descriptor (its claim consumed it) and its slot carries the
  owner's own record, so v5_recovery_judging_cb answered "not judging". The
  resume replays K again and its verification found 36 of 67 manifest records
  gone (P-RMAN-PREREPLAY-VERIFY mutated=36). The sweep was queued 36 ms
  before the first verification ran, so only its scan time kept it from
  refusing the first pass as well. K is now under judgement while this mount
  holds the open term (P-TAUTH-RETENTION-BOOT-K), and RECOVERY_COMPLETE
  queues the sweep again, which then takes K's pages.
• The resume then refused at finish (P-BOOT-FINISH-INCOMPLETE complete=0x1): the other victim's descriptor was at GRANTS_RELEASED with no
  completion bit, and the barrier's pending sweep reads GRANTS_RELEASED as
  complete because on the live path only the sector zero is left. Under a
  term the bit comes before that zero, so a sealed victim without its bit now
  stays in the recovery cut (P-BOOT-COMPLETION-OWED) until the ladder writes
  it.

A DRBD bootstrap whose owner died is taken over. The takeover authorised
itself by fencing the dead owner's PR key, which a DRBD owner never registers,
so it was refused at once (rc=-22). It now startup-fences the peer and
proves the old owner excluded by kind 25, with the contender's key derived as
the claim derives it; the bootstrap-owner record accepts kind 25.

A stale takeover attempt of the same boot is superseded. A contender that
failed after its election and could not clear its takeover-journal entry
blocked every later takeover on that host until it rebooted: the entry was
fenced like a foreign contender, and its key is this boot's own. An entry of
this host's this boot that stood still for the whole abandon window is an
attempt that has returned (one fill_super per device; its journal heartbeat
was joined in its unwind), so it is superseded without a fence
(P-BOOT-CONTENDER-OWN-BOOT).

Not DRBD-specific: a pair outage on a PR-fenced configuration no longer
ends REFUSED because the other host is booting. After both nodes of
2/net/mesh/direct lose power, the first node back proves its own previous
boot's victim key absent only by the witnessed LU reset, which is refused
while any other initiator is registered: the peer's previous boot's key (the
peer not back yet, or never coming back), or the peer's new key while its own
mount is in flight. Phase 3 tried that slot first, and one refused attempt,
which submitted no command, made the term terminally REFUSED until an
operator ran chk_mxfs --clear-bootstrap. Measured on test15/test16; the
original reading (the peer's REGISTER replacing the victim key) was wrong.
Now a miss that submitted no command is deferred while the other victims are
fenced (the peer's old key is removed by its own victim's PREEMPT AND ABORT);
the second pass waits for the certificate and drives the fence retry itself,
because the PR worker re-drives only on a live mount; a series that stays
BLOCKED (nothing consumed) leaves the term RECOVERING for a resume instead of
REFUSED; and a certificate for a sealed victim of the open term is not handed
to the late-death dispatch.

A TCP mount refuses at setup when another node's term is in progress, now
exercised. The fix shipped in 0.90.40 but its branch had never run: a
mount whose peek read the record IDLE and whose setup found another node's
term RECOVERING. mxfs.bootstrap_inject=17 (held after the peek) and =18
(held registered, before setup) put a mount exactly there: it logs
P-BOOT-SETUP-REFUSED, runs no recovery, claims no slot and unregisters, and
the term it met completes.

Measured on test15/test16 (2/net/mesh/direct), build
547CEA42C1BF30E540E7F3B, tests/boot_setup_refusal_2n.sh: race PASS (the
peer held before its REGISTER: slot 1 certified by PREEMPT AND ABORT, then
slot 0 by the driven LU-reset retry, RECOVERY_COMPLETE; the peer refused at
setup and joined afterwards), regwindow PASS (the peer held registered under
its new key: the owner waited, the peer refused and unregistered, both slots
certified, RECOVERY_COMPLETE); every fsynced file read back on both nodes and
cold chk_mxfs clean in each. The undelayed concurrent and staggered (45 s)
pair-outage laps also PASS.

Measured on test1/test2 (DRBD 8.4.11), build 0A68DE356318542DB7A34C3:

• outage-test OUTAGE_TOMB_ARM=1 PASS: with every bootstrap swap refused the
  mount was refused and sectors 0/31/32-39 were unchanged; with the tombstone
  swaps refused the completion failed with sectors 32-39 unchanged; the
  same-boot resume verified K (mutated=0), wrote the tombstone and reached
  RECOVERY_COMPLETE; the peer rejoined; every fsynced file of both nodes
  intact on both; cold chk_mxfs clean.
• takeover-test self and takeover-test foreign PASS, each with
  TAKEOVER_NOCAW_ARM=1 (takeover-journal swaps refused: mount refused,
  sector 31 unchanged) and TAKEOVER_CLEAR_ARM=1 (the contender's stage swap
  and journal clear refused, its entry superseded by the next attempt in the
  same boot): term 1→2 by DRBD_STONITH_WITNESSED_V1, the other node rejoined,
  every file intact on both, cold chk_mxfs clean.

Measured on test1/test2 (DRBD 8.4.11), the released build
547CEA42C1BF30E540E7F3B:

• The 2/net/mesh/drbd suite passed every row (31 PASS, none failed),
  crash_audit included. outage-test passed, and takeover-test self
  and foreign passed.
• death-test: a node crash with a writer running was fenced
  (P238-DRBD-FENCE-WITNESSED, P236-FENCE-CERTIFIED) and its slice
  recovered in 84 s. Every fsynced file of both nodes was intact and cold
  chk_mxfs clean.
• fence-test: a link cut settled in 11 s and the survivor kept writing.
• split-test: exactly one winner, in 11 s.
• remount-test: three unmount/remount cycles, a crash-cut retirement and a
  whole-cluster restart, with nothing stalled and cold chk_mxfs clean.
• resolve-test: dm over /dev/drbd0 was refused, a loop device was
  refused, and both devices were left byte-identical.

Released on the same build, at 2, 4 and 8 nodes:

• Release boards: 2/net/mesh/direct and 2/disk/caw/direct,
  4/net/mesh/direct and 4/disk/caw/direct, and 8/net/mesh/direct and
  8/disk/caw/direct each passed 31 of 31.
• Platforms: packaged rounds and the hung-node test passed at 8 nodes on
  both 8-node configurations, on:
  • Proxmox VE 9, kernels 6.17 and 7.0
  • RHEL 9.8, SELinux enforcing with sVirt
  • Ubuntu 24.04
  • Debian 13
• DRBD: 2/net/mesh/drbd stays a trial attachment, outside the release
  matrix.

Removed from the defect queue as fixed and verified:
D-BOOTSTRAP-RESUME-AFTER-PARTIAL-COMPLETION-ABORTS,
D-DRBD-BOOTSTRAP-TAKEOVER-REFUSED-SO-A-DEAD-OWNERS-TERM-CANNOT-BE-FINISHED,
D-BOOTSTRAP-TAKEOVER-STALE-CONTENDER-OF-THIS-BOOT-IS-NEVER-EXCLUDED,
D-CAS-FALLS-BACK-TO-A-PLAIN-WRITE-ON-A-DEVICE-WITHOUT-COMPARE-AND-WRITE
(its last site class, the takeover journal and its clear, now exercised with
its swaps refused), D-PAIR-OUTAGE-BOOTSTRAP-REFUSED-WHEN-A-VICTIM-HOST-REMOUNTS
and D-TCP-MOUNT-CONTINUES-PAST-A-BOOTSTRAP-IN-PROGRESS-AND-RECOVERS-A-SEALED-VICTIM.

Release verification:

• A board that never ran no longer reads as passed.
  tests/board_4node_chain.sh read back the board on file after each
  run.sh. When the host preflight refused all six release boards
  (run.sh rc=3), it read 0.90.40's 31/31 boards and logged the step as
  rc=0 in 14 s instead of ~27 min. The chain now fails a board whose
  run.sh did not succeed.
• The host's free-space gate is one limit: 10 G free. The preflight
  refused runs at 87.08% used with 225 G free, because df rounds that to
  88% against an 88% ceiling. A percentage of a fixed-size disk restates
  the free-space limit, so only the free-space floor remains
  (scripts/clyde_preflight.sh, docs/host-safety.md).
• Platform rounds run on two nodes of each platform. Each round uses a
  node and a peer to fence. Every fault a platform VM has caught showed at
  two nodes: a kernel API, the fence on that kernel, packaging. Scale is
  verified on the development rig by the release boards.
  tests/full_verify.sh runs the 2-node configurations of the release
  matrix on each platform's first two nodes and powers up only those.
  The -3..-8 nodes of the four platform sets (24 VMs, 134 G) are deleted;
  the lab files and data/platforms.json describe two-node sets. 0.90.41
  itself was verified on eight nodes of each platform, before this change.

Tests: `scripts/drbd_rig.sh takeover-test [self|foreign...

MeshXFS 0.90.39

2026-10-01 — 0.90.39 — mkfs.mxfs formats a 20 GB LUN in 0.24 s instead of 18–31 s; a drain that cannot claim its inode is handed to the work queue instead of run unclaimed; slot depth changes only under the slot lock; test LUNs come from a pool, and the release boards run side by side

mkfs.mxfs is no longer slow. It wrote every empty page of the lock
authority ledger as its own synchronous 4 KiB write, about 2.3 ms each: 10,571
writes for a 20 GB LUN and 67,650 for 128 GiB, so a format took tens of
seconds on a small LUN and minutes on a large one. It now writes the ledger
in large batched writes, flushes once at the end instead of opening the device
O_SYNC, and computes CRC32C in hardware: 0.13 s at 10 GB (was 12.4 s) and
0.24 s at 20 GB (was 17.9–30.9 s), with a byte-for-byte identical result
(details under "Defects disposed" below).

0.90.38 was built but never released. Its changes below ship in this
version, which replaces its unclaimed-drain fallback and closes one more
race in the claim slots that its own self-test found.

A release drain that holds no demoter claim no longer runs. 0.90.38
waited 50 ms for a slot and, if both stayed held, ran the drain anyway, which
is the self-deadlock D-PINNED-RELEASE-RUNS-BAST-PROCESS-WITH-NO-DEMOTER-CLAIM
describes. Now:

• The BAST worker, the MHT dwork and mxfs_clayer/pinned_resource.c hand
  the release to the MHT dwork (mxfs_dlm_drain_defer). The inode's release
  stays owed (i_dlm_bast_pending), its state is left as it is so local
  non-demoters keep waiting as they would during the drain, and the dwork
  retries the claim a few ms later. The caller's inode reference goes with
  the arm. When a dwork is already queued it holds its own reference, so the
  caller's duplicate is dropped and is never the last one.
• Where no work can be queued (an inode being evicted, or the arm gate closed
  for teardown) the drain waits for a slot (mxfs_dlm_claim_demoter_sync).
  That covers the freeing-inode inline drain in mxfs_dlm_ilock_begin.
  Both slot holders are then live drains of the inode, so the wait ends.
• mxfs_bast_arm_queue_delayed_gated tells "queued", "already queued" and
  "gate closed" apart under one lock.
• P75-DEMOTER-DRAIN reports the deferred and synchronous paths.

An owner changes its slot's depth only under the slot lock, and only while
the slot still names it (D-DEMOTER-OWNER-WRITES-DEPTH-AFTER-ITS-CLAIM-WAS-SWEPT).
MXFS_CLEAR_DEMOTER read "this slot is mine" outside the reap lock and then
wrote the depth; the nest path of MXFS_SET_DEMOTER did the same. The punt
sweep can empty a retained claim between that read and the write, and another
task can claim the slot, so the stale owner's write landed on the new owner's
depth and its nested claim was released by its first unnest.
mxfs_demoter_slot_take now sets depth and stamps under the lock when it
wins, mxfs_demoter_slot_nest raises depth only while the slot names the
caller, and mxfs_demoter_slot_release lowers or empties it the same way.
A task that holds slot 2 now nests there instead of also taking slot 1.

Test-only knobs, all off by default: demoter_claim_fail_inject=N makes
every Nth claim wait report no slot, to drive the deferred path;
demoter_test_window_us stalls inside the slot race windows;
demoter_slot_selftest=N races claimants, the punt sweep, dead-claim
retirement and threads that exit holding a claim for N seconds on detached
inodes and prints a verdict. tests/demoter_selftest_node.sh runs that on
one idle node; tests/demoter_claim_fix_verify.sh runs it and the
injected-claim board rows on a group.

Every fixed test LUN is gone. The rig's :shared image (128 GiB, sparse,
72 GB written), the per-group and per-platform LUNs, seven snapshots of
corrupt group LUNs and a stale /etc/scst.conf were deleted, with
scripts/rig_groups.sh and scripts/scst_platform_targets.sh. A sparse
image grows into whatever a test writes, on the filesystem that also holds
every guest image and the host journal.

tools/lun_pool.sh keeps a pool of fixed-size test LUNs (12 × 20 GiB on
clyde: six board groups, four platform sets and the yardstick capture can
each hold one; with 8, debian13's platform round found none free), each allocated in full at create so it cannot grow, each its own SCST
target with no default LUN, visible only to the nodes it is allocated to.
alloc binds a LUN to a node set for the life of an owner process; when the
owner exits the allocation stays bound, and the next run on the same nodes
adopts it without rebinding. It is released when another allocation names any
of its nodes, or when no LUN is free. snapshot copies a LUN's platter out
of the pool before the next holder formats it. up re-registers the pool
after a host reboot.

• run.sh borrows a pool LUN for every direct-attachment and --group run
  once it holds its locks, and exports the LUN's WWID and host image. The
  log slice count goes with the size class. mkfs caps the AG count at what
  fits one AG's log, so a 20 GiB LUN formatted with 32 slices had 9 AGs; with
  16 it has 19. Runs of up to 8 nodes use 2N slices on 20 GiB, up to 16 use
  32 on 80 GiB, beyond that 32 on 144 GiB.
• Throughput rows (fio_perf, scaling_curve, dlm_scaling,
  rsync_paired) take a host-wide lock, so two side-by-side runs never grade
  each other's load. The first parallel 8-node pair ran fio_perf in the
  same 25 s, and one board read randW at 27% of the native-XFS yardstick
  while the other read 358%.
• tests/full_verify.sh runs the release matrix's suites side by side on rig
  groups g<N> and g<N>b, and gives every platform set a pool LUN and a lab
  file written from it. tests/board_4node_chain.sh takes
  <configuration>@<group> and runs such boards side by side.
  tests/release_verify_chain.sh runs the claimed node count's boards and
  every smaller count's in one side-by-side step, between the clean build
  and the packages, and powers down rig guests a stage does not use.
  SIDE_BY_SIDE=0 keeps the old order: the claimed count's boards inside
  full_verify, then the smaller counts'.
• Every rig guest (test1–test32) has 2 vCPUs instead of 4. All six boards of
  the release matrix side by side are 28 guests: 56 vCPUs on clyde's 56
  cores and about 70 GiB of its 94. At 4 vCPUs the same set was 112 vCPUs.
• scripts/lab_power.sh takes group:<name>. scripts/clyde_preflight.sh
  checks the pool's filesystem. data/rigs.json declares the SCST rig as a
  pool with no fixed LUN. tests/lib/rig.sh resolves a harness's LUN from
  its run, or from the pool allocation bound to its nodes.
• tests/tcp_peer_freeze_death.sh ends its window once the death and the
  survivor's first write have both been seen, or at the 180 s write budget.
  It used to run 300 s whatever happened, though the verdict was known at
  about +73 s; that was 4 idle minutes in each of a release's eight hung-node
  tests. The other survivors are probed when the window ends, so they now
  have to be serving inside the write budget, as the check always said.
• tests/full_verify.sh takes PLATFORMS to re-run only some platform sets
  on a version whose others already ran. The verdict still reads every
  platform's log, so a set left out is graded on its own last run of that
  version, and one with no run of it refuses to be left out.
• tests/packaged_round.sh logs each mount's own duration (mount_ms), and
  repeats its reboot section POSTBOOT_LAPS times (default 1): marker
  written, every node unmounted and rebooted, every node mounted again with
  no mkfs, the marker read back everywhere.
• tests/packaged_round.sh saves the kernel log of any node whose mount
  fails or does not return within its budget, before the round stops. The
  next round reboots the set and a platform's journal is not persistent, so
  without it the log of D-POSTBOOT-JOINER-MOUNT-OVERRAN-60S-NO-KERNEL-LOG
  was lost.
• tools/lun_pool.sh tries a node's iSCSI initiator name three times, 5 s
  apart, before failing an allocation. A set is allocated right after it is
  powered up, and one missed ssh to debian13-1 cost that platform its whole
  round.
• tests/release_verify_chain.sh records the host's CPU, memory and I/O
  every 5 s for the whole chain (release_verify_hostcpu_<version>.log), so
  how far its stages can overlap is decided from what the host actually
  spends rather than from the vCPUs the guests were given.
• tests/full_verify.sh builds the packages unless dist/<version> holds a
  SHA256SUMS its packages match. It used to skip the build whenever the
  directory existed, and a release.sh stopped part way leaves it behind
  empty: this version's first full chain built nothing and failed all twelve
  packaged rounds on a missing .deb. Its exit status is now the platform
  steps' verdict; it used to be that of the summary grep, so that chain
  still returned 0.
• fio_perf reports the best of 4 measured passes per workload, after
  the warm-up pass it already discarded, and prints every pass. The clyde
  LUN's write throughput is bimodal from run to run: ten native-XFS runs on
  test1, same LUN and workload, read seqW from 783 to 2188 MiB/s, and three
  of them sat under 70% of their own median. A single pass graded against
  any native figure therefore failed whenever it landed in the slow regime:
  4/net/mesh/direct read 68% that way. The native capture runs the same
  script, so both sides of fio_perf_vs_xfs are measured alike.
• The native-XFS fio yardsticks were measured again on a pool LUN, on test1
  at 2 vCPUs with every other rig guest powered off and clyde otherwise
  idle. Each is the median of 5 runs, and each run is the best of 4 passes:
  • net/mesh/direct: 1324 MiB/s seqW, 1331 MiB/s seqR, 39,196 randW and
    47,216 randR iops. The five seqW runs read 983–1452.
  • disk/caw/direct: 1133, 1291, 44,043...

MeshXFS 0.90.37

2026-10-01 — 0.90.37 — directory data-loss fixes; {2,4,8}/net/mesh/direct and {2,4,8}/disk/caw/direct released on Proxmox VE 9, RHEL 9.8, Ubuntu 24.04 and Debian 13

Upgrade from 0.90.36. 0.90.36 can lose directory entries and link counts
when one node removes directories other nodes are writing into, and a
symlink into such a directory can shut nodes down. 0.90.37 fixes each of
these; the fixes and their verification are below.

Release verification (tests/release_verify_chain.sh 0.90.37, CLAIM=8,
POWER=1, log tests/evidence/release_verify_0.90.37.log): window laps of
8/net/mesh/direct and 2/net/mesh/direct, the full 8/net/mesh/direct and
8/disk/caw/direct boards, the packaged rounds and freeze-death tests of both
configurations on every platform (tests/evidence/full_verify_0.90.37*.log),
then the 4/* and 2/* boards, all on module srcversion
85AEE4625B3EA2BBDBA2B96.

Configurations replace condition codes in every tool, record and document.

What the rig tests is now named by a configuration,
<nodes>/<class>/<method>/<attach>: how many nodes, where the DLM keeps lock
state (net or disk), which DLM implementation (mesh, caw), and how the
shared LUN reaches each node (direct, mpath, pass). 8/net/mesh/direct
is what the rig called 8-node tcp. The design is
docs/attachment-methods.md; the definitions are data/configurations.json;
the only parser is the new tools/configuration.py, which every tool, harness
and record now goes through. No kernel code and no module behaviour changed.

Why. The condition codes tcp, cawd, cawp and caw each fused the
lock manager with the attachment. That hid that every release so far was
verified on direct single-path iSCSI only, left net/mesh with no multipath
or passthrough cell although it fences through the LUN exactly as disk/caw
does, and
let exact string matches drop records silently:

• a defect tagged caw never appeared in a cawd view, or the reverse; the
  view the loop's session prompt printed was cawd, so every disk/caw
  record was missing from it;
• the board's declared configuration list had no cawd or cawp column, so
  the whole-board roll-up and --gaps never looked at them;
• a criterion that applies only to disk/caw was compared to the whole column
  name, so dlm_lock_correctness was hidden from the disk/caw/direct and
  disk/caw/pass boards although the rig wrote it there. With the pattern
  match it appears, and on the 0.90.36 module it reads PASS: the
  disk/caw/direct boards are 31 of 31, not the 30 of 30 the old view showed;
• a key the board could not parse made every criterion count as applicable.
  It now applies to nothing.

The retired codes map one to one and are refused by name with their
replacement, never translated: tcp → net/mesh/direct, cawd →
disk/caw/direct, caw → disk/caw/mpath, cawp → disk/caw/pass,
tcpmp → net/mesh/mpath. The native-XFS baseline is 1/xfs.

Records migrated, not restarted, each through its own writer where it has
one:

• the board (tools/criteria.py migrate-keys): 27 columns renamed with their
  cells and history (cell and history totals identical before and after,
  checked per column); each criterion's transport became an applies_to
  pattern (net/mesh, disk/caw, any). The 2/net/mesh/direct,
  4/net/mesh/direct and 8/net/mesh/direct boards read identical row for
  row before and after;
• the defect queue (tools/defects.py migrate-config): each record's dlm
  became a config pattern, 23 any → */*/*, 37 caw → disk/caw/*, 30
  tcp → net/mesh/*. The attachment is left open on every record, because
  none recorded which attachment its evidence came from. The views
  32/disk/caw/mpath, 2/net/mesh/direct and 32/net/mesh/direct hold
  exactly the records 32 caw, 2 tcp and 32 tcp did, and nothing blocks any
  of the six released configurations;
• scripts/migrate_configuration_keys.py: 658 bench.json entries (dlm →
  configuration, labels to the slug form), the eleven per-rig native-XFS
  baseline and raw-ceiling files renamed one to one
  (.xfs_fio_baseline.net-mesh-direct.scst-fio.json), and the cluster marker
  and last-run record.

History carried over includes runs from before 2026-09-26, when the rig's
net/mesh laps reached their LUN through LIO or the QNAP rather than direct
iSCSI; they keep the net/mesh/direct name. A failure among them still counts in a flake
window, so the carry-over can only make a board stricter.

Tools and harnesses.

• run.sh <configuration> (./run.sh 8/net/mesh/direct); the device default
  and the restore after a power cycle come from the attachment.
  net/mesh/direct now gets the same single-portal iSCSI restore as
  disk/caw/direct, because both use the same device; before, tcp got no
  restore, a leftover from when its disk was wired into the VM XML.
• criteria.py and defects.py take one key; defects.py also takes a
  partial view (8, 8/disk, 8/disk/caw). -D/--dlm became
  -C/--config.
• The release matrix (data/configurations.json) drives tests/full_verify.sh
  and tests/release_verify_chain.sh, instead of tcp and cawd spelled out
  in each. Its content is unchanged: {2,4,8}/net/mesh/direct and
  {2,4,8}/disk/caw/direct.
• tests/packaged_round.sh and tests/tcp_peer_freeze_death.sh take
  CONFIG=<configuration>. Each refuses a configuration whose node count is
  not the set it runs on, and any attachment but direct, which is the only one
  either builds.
• scripts/rig.sh and scripts/verify_infra.sh take the attachment names
  (direct, mpath, pass). The LIO/tcm_loop XML wiring they used to bring
  up as a fourth rig fakes COMPARE AND WRITE and is no configuration's
  attachment; switching still unwires it.
• Harnesses read MXFS_CONFIG where they read MXFS_TRANSPORT and
  MXFS_DLM. scripts/rekey_invocations.py rewrote 1092 lines in 460
  scripts (run.sh 2 tcp → run.sh 2/net/mesh/direct, "prep 2/tcp first",
  criteria.py 32 caw); the harnesses with their own transport handling were
  converted by hand.
• tools/criteria_window.py takes one configuration and reads the board
  MXFS_CRIT names, instead of always the primary one.
• Every tool and script that selects what to run takes the same key, the
  full configuration, and nothing takes a node count and a DLM or attachment
  separately any more:
  • scripts/rig.sh <configuration> and scripts/verify_infra.sh <configuration>
    wire and verify its attachment on its node count;
  • scripts/ladder.sh and scripts/raw_fio_ceiling.sh take a list of
    configurations;
  • tests/board_4node_chain.sh takes <configuration>[:rows];
  • the release chain's LAPS entries are <configuration>:<rows>:<count>;
  • about fifty harnesses that took <N> (or NNODES=/N=) with a
    hard-coded DLM now take a configuration, positionally or as CONFIG=.
    Tools that act on node counts alone (lab_power.sh rig:N, the release
    chain's CLAIM/NODES, which select entries of the release matrix) are
    unchanged.

Documents. conditions.md is gone, replaced by
docs/attachment-methods.md (the grammar; both DLM classes and every method,
implemented or not; each attachment and what it needs from the device; DRBD
dual-primary; fabric; device layers MeshXFS refuses) and docs/fencing.md (the
design for pluggable fencing methods). docs/condition4_multipath_scope.md
is now docs/multipath-attach.md. The README names the six released
configurations. The man page (mxfs.5) and docs/iscsi_setup.md no longer
list NVMe-oF, Fibre Channel and SAS as supported storage, or call disk/caw
usable on LIO, or say 2 nodes is the only released size.

ROADMAP.md is new. It is the list of wanted, unbuilt work, and an item
leaves it when done. It holds the lock methods named but not implemented
(net/server, disk/reserve, disk/pr, disk/fused, disk/paxos),
pluggable fencing, death-threshold validation at 16+ nodes, DRBD dual-primary,
NVMe, net/mesh/mpath and net/mesh/pass, and multipath and passthrough in
the release matrix. It replaces docs/future.md, which is deleted. That file
still listed disk/caw/mpath as not started, although it has run for months, and most of its other entries named frontend/ and
libmxfs/, which the move to the XFS fork removed. Platforms stay in
data/platforms.json.

tools/mxfs_rig_tag.sh identifies the rig behind a multipath map. It
read the SCSI vendor from /sys/block/<dev>/device/vendor, which a
device-mapper node does not have. On /dev/mapper/mpatha (dm-1 on test1) it
therefore printed nothing, the 2/disk/caw/mpath prep recorded an empty rig,
and the fio tests that score against a per-rig yardstick could not be scored
on any multipath configuration. It now reads the vendor of the map's first
slave path. The paths of one map are one LUN, and on test1 both slaves, sda
and sdb, report SCST_FIO. Verified: the multipath map and the single-path
by-path device both resolve to scst-fio.

Rig groups: several configurations run on the rig at once. A rig group
is a disjoint slice of test1..test32, named by a new group line in the lab
file (tools/mxfs_lab.sh group|groups). The shape comes from the host: each
rig VM has 4 GiB and 4 vCPUs, so the 28 nodes that all six release boards need
at once do not fit in 94 GiB. Groups g2, g4 and g8 (test1..14, 56 GiB)
hold one DLM class's three release boards at a time.

• scripts/rig_groups.sh setup gives each group a LUN of its own, built the
  same way as the platform targets: a sparse 20G image, a target whose LUN 0
  only the group's initiators can see, and each node logged out of everything
  and into its own group's target alone.
• run.sh <configuration> --group <name> runs on the group's nodes and LUN.
  It keeps its own cluster mar...

MeshXFS 0.90.36

2026-09-30 — 0.90.36 — 8-node TCP and 8-node CAW are released on Proxmox VE 9, RHEL 9.8, Ubuntu 24.04 and Debian 13

Clusters of eight nodes are now released on both transports, alongside
clusters of two and of four, on exactly the kernels the earlier releases
name: Proxmox VE 9 (6.17.2-1-pve and 7.0.14-19-pve), RHEL / AlmaLinux / Rocky
9.8 (5.14.0-687.49.1.el9_8), Ubuntu 24.04 LTS (6.8.0-101-generic) and Debian
13 (6.12.107+deb13-amd64). Three nodes, five to seven, and more than eight
are not claimed. Nothing in the defect queue blocks any of the six released
configurations (tools/defects.py 8 tcp --release, 8 caw --release, and the
same at 4 and 2).

How the LUN reached each node. Every node of every verification ran its
own iSCSI initiator and logged into the shared target on one portal: a
single-path SCSI disk, with nothing between MeshXFS and the target. Neither
dm-multipath nor a hypervisor's SCSI passthrough carried the lock or fencing
commands in any run of this release, on either transport. What the rig names
cawd is CAW on that single-path attachment; the 2- and 4-node CAW releases
(0.90.7, 0.90.24) were verified the same way. Multipath and passthrough are
the next verification work, on both transports.

This is the first release since 0.90.24. It carries the changes recorded
below under 0.90.25 to 0.90.35, none of which was released on its own.

What 0.90.36 passed (module srcversion 591CDF015CE73AC096D0E79; logs
tests/evidence/release_verify_0.90.36.log and
tests/evidence/full_verify_0.90.36.log):

• a build from a clean copy of the tree, 0 warnings, with the deployed
  module's srcversion; the userspace tools, the user-mode ledger tests (0
  failures), the extern-declaration audit and the inode flag audit
• every rig suite, every row PASS on this module, none FLAKY and none
  SKIPped: 8-node TCP 31 of 31, 8-node CAW 30 of 30, 4-node TCP 31 of 31,
  4-node CAW 30 of 30, 2-node TCP 31 of 31, 2-node CAW 30 of 30
  (tools/criteria.py 8 tcp --build 591CDF015CE73AC096D0E79, and the same
  for 8 cawd, 4 tcp, 4 cawd, 2 tcp, 2 cawd). Before the 8-node
  boards, ten laps each of the rows whose eleven-run window still held an
  earlier build's failure (8/tcp guard_census; 8/cawd dirent_durability
  and guard_census) passed every row
• the release packages, built in the oldest container each targets, after
  each platform's kernel build check
• on every platform, on eight nodes sharing a LUN of their own, on each
  transport: the packaged round (install from the packages, DKMS build, mount
  with no options on all eight, cross-node checksums, create and remote
  delete, chk_mxfs clean, peer= with multicast dropped, every node
  rebooted with the configured transport, data intact), Proxmox on both
  kernels; the hung-node test (a node frozen mid-write, declared dead,
  fenced, replayed, the other seven writing again); on RHEL, SELinux
  enforcing with sVirt. 39 of 39 steps passed
• the hung-node test with eight members: on TCP the frozen node was declared
  dead at +59-60 s and the measured survivor wrote again at +72-73 s on three
  platforms and at +99 s on RHEL; on CAW at +65-66 s and +73-75 s (budgets
  120 s and 180 s)

The platforms ran one at a time. An earlier run of the same verification put
two eight-node sets on the host at once; there, six of eight RHEL nodes
overran the 660 s install budget compiling the module beside Ubuntu's
compile, 64 vCPUs of compile on a 56-core host. Every other step of that run
passed, and the one-at-a-time run above is the one this release cites.

The two- and four-node claims were earned again on this module rather than
carried over from 0.90.24.

The modify-path directory evict reads its keep inputs in checkpoint order

Predictions were written before the run, in tests/evidence/lapq_x36a.queue.

• What was measured. Queue w35a lap 3 (8/tcp, two victims, 0.90.35)
  ended with chk_mxfs rc=4: a survivor's directory held five names whose
  inodes the inode btree holds free. The read-side instrument printed once,
  on that survivor: its rm read the directory's block from the platter
  while the block carried 25 logged, unwritten changes (lseq 563, wseq 538),
  and the buffer's last invalidator was the modify-path evict's DONE clear
  in mxfs_dir_evict_data_blocks, with the block in the AIL and undestaged
  and every stale-base arm clear, a state the keep rule keeps.
• Cause. The evict read the log item's AIL membership first and the
  buffer's pin count second. Checkpoint completion does the reverse, AIL
  insert then unpin (xlog_cil_ail_insert_batch), without the buffer lock
  the evict holds. After xfsaild wrote the block, the rm's next removal put
  a fresh item in the CIL, pinned and not yet in the AIL; an evict that read
  "not in AIL" before the checkpoint's insert and "not pinned" after its
  unpin took the block for a clean one and cleared DONE. The rm's next
  unlink read the platter's older image, undoing the removals it had made,
  and the directory kept the names of inodes it went on to free.
• Change (xfs/xfs_mxfs_dir_evict.c). The pin count is read first,
  then AIL membership behind a read barrier, and the keep test uses those
  two reads only; unpinned then means any checkpoint that carried the item
  has already put it in the AIL. P-EVICT-AIL-ARRIVED prints if an item
  reaches the AIL between the decision and the clear with changes unwritten,
  which only the old order allows. Control switches
  -DMXFS_TEST_EVICT_AIL_FIRST (old order, 5 ms sleep between the reads
  while pinned) and -DMXFS_TEST_EVICT_WIDEN (the same sleep in the new
  order) build the arms from the same source.
• Verified by control and fix arms (queues x36a, x36b). Control
  (-DMXFS_TEST_EVICT_AIL_FIRST): within 4 s of the load, every one of the
  eight nodes printed P-EVICT-AIL-ARRIVED on a block its load had just
  logged 70 changes into and never written, read that block back as corrupt
  (error 117) about 110 ms later and shut down. Fix with the same widening
  (-DMXFS_TEST_EVICT_WIDEN): detector 0 on every node, no read error, no
  shutdown, chk_mxfs rc=0, PASS.

The same read order at the other keep-or-drop decisions

• Seven more decisions read AIL membership before the pin count and keep on
  it without also keeping on logged-but-unwritten: the owned-block evict
  (mxfs_dir_evict_owned_dir_blocks), the release-side stale
  (mxfs_dir_stale_data_blocks), the cached-block drop
  (mxfs_dlm_drop_clean_cached_blocks), the gen-lagging discard of the
  AG-metadata invalidate (mxfs_ag_meta_invalidate_stale), the two
  epoch-stale refreshes in xfs_dir2_node_addname_int and the one in
  xfs_dir2_leaf_addname. Each now reads through
  mxfs_buf_read_pin_then_ail (xfs/xfs_mxfs_ag_meta.c), which reads the
  pin count, then AIL membership behind a read barrier; the invalidate's
  other arm shares those reads. What each decision keeps is unchanged.
  Found by a sweep of the source and not measured one by one: the
  mechanism is the one the arms above proved.

The fresh-grant walk keeps an inode cluster buffer that carries inode items

• What was measured. The first 8/cawd board on the 0.90.36 candidate
  (inside tests/full_verify.sh 0.90.36) lost test2 during
  dirent_durability. The fresh-grant AG walk (mxfs_dlm_invalidate_ag_meta)
  staled an inode cluster buffer with one inode item attached
  (P-STALE-WITH-ITEMS items=1 delwri=0). About 10 ms later xfsaild's write
  pass skipped it as locked. After that the buffer was on no list and never
  written, while inode 16799147's item sat FLUSHING at the AIL minimum. The
  no-inode release fence found the minimum frozen and shut the node down
  rather than release undrained. Every name the row counted as lost was
  test2's, in the directories that release held.
• Cause, as far as it is proven. An attached inode item keeps a pointer
  to that buffer object and flushes through it. A flush into the staled
  buffer queues a stale buffer. The next lookup's stale reset
  (xfs_buf_find_lock) wipes _XBF_DELWRI_Q, and the write pass then drops
  it unwritten. The two middle steps are what the new probes
  P-DELWRI-QUEUE-STALE (a stale buffer queued) and P-STALE-DELWRI-WIPE (a
  queued write reset by a lookup) print.
• Change (xfs/xfs_mxfs_buf.c). Both of the walk's inode-buffer keeps,
  unlocked and locked, now also keep a buffer whose b_li_list is non-empty,
  as they already kept one queued for write. -DMXFS_TEST_WALK_STALES_ATTACHED
  builds the walk as it was.

The acquire re-drive no longer skips a single-node mount

The 0.90.35 re-drive returned at once on a single-node mount, and the 8-node
guard_census rows failed on it as an unclassified single-node fast path.
The skip was wrong on its merits. An EX without its certificate is refused
by whichever node replays this one's journal, and a node that is alone now
may not be alone when it dies. The test is gone; the loop already leaves at
once when nothing is stale.

Removed from the defect queue as fixed and verified

• D-SURVIVOR-DIRECTORY-KEEPS-NAMES-OF-FREED-INODES-AFTER-TWO-VICTIM-LAP
  (8/tcp, integrity): the read order above. After the control and fix arms,
  seven laps of the failing two-victim configuration passed, three on the
  fix-only build (x36b) and four on the build with every decision reordered
  (x36c, srcversion 859A861B9DECC27AB1B4B2D). Each ended with
  chk_mxfs rc=0 and no platter read over unwritten content, no dropped
  block and no dangling name on any node. The rate before was one failing
  lap in three to nine.
• D-TWO-NODE-KILL-UNDER-LOAD-SLICE-REPLAY-REFUSED-AG-QUARANTINED-EIO
  (stability): the 0.90.35 re-drive of an acquire whose certificate a
  release of the same node made stale. After its control arm (14
  uncertified acquires in one lap), 15 laps of the two-vi...