vms-ec75: DLM rung H11 — cluster-wide distributed deadlock detection (the finale) - #930
Merged
Conversation
Edge-chasing deadlock search over the distributed wait-for graph: a new SCS_DLM_OP_DLKSRCH probe follows the wait-for edges the master already computes (blocking_csid/blocking_req_lkid), detects a REAL cross-node cycle, and aborts one deterministic victim (the initiator) with SS$_DEADLOCK via the existing GRANT-reply path. Clean-room: structure from standard edge-chasing + public OpenVMS lock-management docs; the byte-level frame is an OVMX design choice (Rule 8). Implementation + 2-node cycle harness follow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…or (double-victim hazard) 'Victim = the initiator' is unsafe: in a 2-node cycle both ends can initiate a search concurrently, each probe returns to its own initiator, and BOTH abort -> two processes killed when exactly one should be (the data-integrity failure this rung prevents). Fixed: the probe carries a running MIN (csid,lkid) over every request it chases; edge-chasing visits the whole cycle before closing, so every probe computes the SAME global-min victim regardless of who initiated. Cycle-close tests the initiator; the ABORTED request is the global min. Victim delivery is idempotent (a second VICTIM for an already-aborted request is a no-op). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-based (H11 IS buildable)
The Opus impl pass correctly found the proc-based chase framing (find the holder's
process P, does P have a waiting request) unbuildable: a cross-node lock has no
owning process. But that walled framing was the design's error, not a real
prerequisite. The distributed wait-for graph is READABLE today, CSID-keyed and
split across two authorities that existing state already holds:
- HOME: grant_recv creates a vms_dlm_origin even for a QUEUED reply
(granted_mode==NL, carrying resnam+master_csid) — so a node's pending
cross-node waits are enumerable (confirmed: scsd's queued-reply path,
dlm_pend_seen). New readback ioctl reads the existing origin list.
- MASTER: res->granted is keyed by req_csid; H10b's VMS_IOCTL_DLM_GET_GRANTED
already returns holder_csid/holder_req_lkid for a resource — reuse it.
The chase queries home (what does CSID H wait for) then master (who holds it),
hop by hop, reading only real state. No P2 app-acquisition path, no stored blocker
edge — the 2-node cycle is daemon-choreographed + CSID-keyed like every DLM rung.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The LAST DLM rung. Extends the local wait-for-graph detector to a
cluster-wide one: a deadlock CYCLE whose edges cross nodes is detected by
an edge-chasing search (Chandy-Misra-Haas) over REAL executive state, and
EXACTLY ONE deterministic victim's queued $ENQ is aborted with SS$_DEADLOCK
(3594), breaking the cycle.
Executive (shared kernel-core + both backends):
- SCS_DLM_OP_DLKSRCH = 6 wire op, registered in all N sites: scs_dlm.h
enum + scs_dlm.c codec/op_name, vms_ioctl.h VMS_DLM_OP_* mirror AND the
NetBSD mirror vms_lock_nb.h (#928 twin trap), scsd.c static_assert,
vms_lock.c dispatch. Phase (SEARCH-HOLDER / SEARCH-RESOURCE / VICTIM)
rides the frame `flags`; ttl rides `status`; the running global-min
victim rides the valblk slots.
- New readback ioctl VMS_IOCTL_DLM_ENUM_WAITS (0x38): the HOME authority --
enumerate this node's pending (NL) cross-node origins (resnam + master +
handle), one outgoing wait-for edge each. Reuses GET_GRANTED (0x37) as
the MASTER authority ("who holds R?"). Both wired on Linux + NetBSD; a
vms_kif_dlm_enum_waits wrapper holds the kernel-floor.
- The VICTIM leg is dispatched into the executive (vms_lock_dlm_xnode_
dlksrch): find the queued cross-node waiter (req_csid,req_lkid) on
res->waiting, remove it, complete with SS$_DEADLOCK. Idempotent.
Daemon (scsd): the initiate (at the master where a cross-node $ENQ queues
behind a REMOTE holder), the two-phase chase orchestration bouncing
home<->master by CSID (peer_by_csid) over the two readback authorities, the
GLOBAL-min victim selection (not local-initiator -- avoids the double-abort
hazard), and the abort + unprompted GRANT(SS$_DEADLOCK) to the victim's
$ENQ. Two new CHOKED senders (scsd_dlm_send_dlksrch probe forward +
scsd_dlm_send_victim_grant victim signal), both in the SEND SITE TABLE.
Harness (3-node A/B/C, C masters both contended resources so both holds and
both waits are cross-node): init_dlm_ec75.sh + run_dlm_harness_ec75.sh +
Dockerfile.dlm-harness-ec75, and the dlm-harness-ec75 CI job.
PROVEN ON A REAL 3-NODE QEMU CLUSTER (KVM): a genuine cross-node cycle
(A waits-for B on RES_E, B waits-for A on RES_D) was DETECTED --
SCSD-I-DLKCYCLE on C (initiator=1030, closed on holder 1030) -- and EXACTLY
ONE victim aborted: node A's WAIT $ENQ returned status=0x00000E0A
(SS$_DEADLOCK), node B's request stayed queued. Every hop reads real state
(enum/get_granted return SS$_NORMAL off live ioctls); INV-6 holds -- a
dropped/ttl-expired probe reports no deadlock, never a fabricated cycle.
Censuses green: send-sites 117/0, kif caller PASS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
baron-3dl
added a commit
that referenced
this pull request
Aug 30, 2026
…der is COMPLETE (H0→H11) (#931) The cluster-dlm.yaml was last_reviewed 2026-08-28, before H10b/e84/H11 landed — it still read remaster/LVB/deadlock 'absent' and BLKAST 'deferred', which the merges (#918 BLKAST H6, #923/#925 LVB H8/H9, #927/#928 remaster H10a/H10b, #929 e84 directory-guard, #930 H11 deadlock — all on main) made false. Update all four items partial→verified with verified_against pointing at the multi-node /dev/vms harnesses that prove them, rewrite the summary to the complete ladder + the honest remaining DEPTH (vms-04f concurrent-initiate proof, vms-b96 directory REDIRECT, vms-d1f post-1.0 app-acquisition), add REBUILD=5/DLKSRCH=6 to the opcode set, and render_compat.py (docs/compatibility-surface.md regenerated, drift gate clean). Closes the vms-7fa epic on the compat surface. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DLM rung H11 (vms-ec75) — the last rung
A cross-node deadlock CYCLE is detected and exactly one deterministic victim's queued
$ENQis aborted withSS$_DEADLOCK(3594), breaking the cycle; the other request stays queued (grants once the victim releases — the VMS back-off contract). After this the DLM ladder (H0→H11 + remaster + LVB + directory-guard) is complete.Approach — CSID-keyed edge-chasing over real state (design:
docs/design-dlm-distributed-deadlock.md)Chandy-Misra-Haas edge-chasing (how VMS does distributed deadlock search; clean-room structure from public lock-mgmt docs, byte frame an OVMX Rule-8 choice). The distributed wait-for graph is read from existing executive state, split across two authorities — no new stored graph, no fabricated edges (INV-6):
VMS_IOCTL_DLM_ENUM_WAITS, new): enumerate a node's pendingvms_dlm_originrecords (granted_mode==NL, from H5's queued-reply path) → what a CSID waits for + where.VMS_IOCTL_DLM_GET_GRANTED, reused from H10b): who holds a resource (req_csid/req_lkid).The chase (in scsd) bounces home→master→home→master; each SEARCH-HOLDER hop updates a running global-min
(csid, lkid)victim; a hop whose holder is the initiator closes the cycle. Victim = the global-min (NOT the initiator), so concurrent bidirectional searches pick the same victim; VICTIM delivery is idempotent (abort of an already-gone request is a no-op) — no double-victim. The victim's queued request is aborted through the real GRANT-replySS$_DEADLOCKpath.Proof — real 3-node QEMU/KVM cluster, exit 0
A/B in the cycle, C the neutral master of both resources (a genuine cycle needs each blocker to be a remote holder, so both edges cross the wire — 3-node like the e84/h10 family):
The runner asserts victim identity (initiator/victim = 1030), that A's
$ENQreally returnsSS$_DEADLOCK, and that B is NOT aborted (exactly one victim — the double-victim guard). Every value read verbatim off live ioctls (enum_rc/get_granted_rc/abort_rc).Changes (16 files)
SCS_DLM_OP_DLKSRCH=6 (all N places incl the NetBSD mirrorvms_lock_nb.h);VMS_IOCTL_DLM_ENUM_WAITS=0x38; the VICTIM-abort handler (vms_lock_dlm_xnode_dlksrch, idempotent, real removal).SS$_DEADLOCK); two new CHOKED senders in the SEND SITE TABLE.tests/qemu/*_dlm_ec75*+dlm-harness-ec75job + path filters.Reconciliation — green
test_scsd_send_sites117 checks / 0 fail;kif_caller_census54/54 (vms_kif_dlm_enum_waits); NetBSD mirror symbol-diffed; ci.yml YAML parses; codec ctests 2/2; nodocs/compat/*.yamltouched.Deferred (Rule 5, filed)
SS$_UNSUPPORTED"0.4" deferral); the named successor to that honest stub.🤖 Generated with Claude Code