fix lvol creation - #2
Merged
Merged
Conversation
hamdy-simplyblock
added a commit
that referenced
this pull request
Mar 15, 2024
hamdy-simplyblock
added a commit
that referenced
this pull request
Mar 15, 2024
hamdy-simplyblock
added a commit
that referenced
this pull request
Mar 23, 2024
fix device add bug set version: 1.8.0
hamdy-simplyblock
added a commit
that referenced
this pull request
Apr 4, 2024
hamdy-simplyblock
added a commit
that referenced
this pull request
Apr 16, 2024
hamdy-simplyblock
added a commit
that referenced
this pull request
Apr 25, 2024
hamdy-simplyblock
pushed a commit
that referenced
this pull request
May 7, 2024
hamdy-simplyblock
pushed a commit
that referenced
this pull request
May 7, 2024
hamdy-simplyblock
added a commit
that referenced
this pull request
May 8, 2024
Add API path to cluster API /cluster/gracefulshutdown/
hamdy-simplyblock
added a commit
that referenced
this pull request
May 23, 2024
hamdy-simplyblock
added a commit
that referenced
this pull request
May 23, 2024
schmidt-scaled
added a commit
that referenced
this pull request
Jun 6, 2026
…RPC name Three defects in the JM shared_placement support (654f941) caused placement-map/journal conflicts and a runaway migration loop: 1. bdev_jm_create hard-coded shared_placement=True (JM_SHARED_PLACEMENT_DEFAULT), so every JM created on a node restart came up shared-placement even on a cluster that had NOT been migrated -- while distrib (which honors cluster.shared_placement) stayed legacy. The mismatch produced a conflict on the placement maps when reading journal records. Now bdev_jm_create takes a shared_placement arg (default off, omitted when False, mirroring bdev_distrib_create) and the three storage_node_ops call sites pass cluster.shared_placement -- so a restart reproduces the cluster's actual mode and JM/distrib stay consistent. 2. jm_set_shared_placement was called with no name, but the data-plane RPC requires one (there is exactly one JM per node, jm_<node_id>); the call errored. The client method now makes name mandatory and cluster_ops.set_shared_placement passes jm_<node_id>. 3. Because the JM RPC always errored (#2), set_shared_placement returned False, the monitor never cleared shared_placement_migration_pending, and it re-fired the migration on every tick (the "every few seconds" symptom). Fixing #2 lets the one-shot migration complete and clear the flag. Tests updated to match: JM create emits the flag only when opted in, and the JM runtime RPC requires a name. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
schmidt-scaled
added a commit
that referenced
this pull request
Jun 6, 2026
…RPC name Three defects in the JM shared_placement support (654f941) caused placement-map/journal conflicts and a runaway migration loop: 1. bdev_jm_create hard-coded shared_placement=True (JM_SHARED_PLACEMENT_DEFAULT), so every JM created on a node restart came up shared-placement even on a cluster that had NOT been migrated -- while distrib (which honors cluster.shared_placement) stayed legacy. The mismatch produced a conflict on the placement maps when reading journal records. Now bdev_jm_create takes a shared_placement arg (default off, omitted when False, mirroring bdev_distrib_create) and the three storage_node_ops call sites pass cluster.shared_placement -- so a restart reproduces the cluster's actual mode and JM/distrib stay consistent. 2. jm_set_shared_placement was called with no name, but the data-plane RPC requires one (there is exactly one JM per node, jm_<node_id>); the call errored. The client method now makes name mandatory and cluster_ops.set_shared_placement passes jm_<node_id>. 3. Because the JM RPC always errored (#2), set_shared_placement returned False, the monitor never cleared shared_placement_migration_pending, and it re-fired the migration on every tick (the "every few seconds" symptom). Fixing #2 lets the one-shot migration complete and clear the flag. Tests updated to match: JM create emits the flag only when opted in, and the JM runtime RPC requires a name. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
schmidt-scaled
added a commit
that referenced
this pull request
Jun 6, 2026
…RPC name Three defects in the JM shared_placement support (654f941) caused placement-map/journal conflicts and a runaway migration loop: 1. bdev_jm_create hard-coded shared_placement=True (JM_SHARED_PLACEMENT_DEFAULT), so every JM created on a node restart came up shared-placement even on a cluster that had NOT been migrated -- while distrib (which honors cluster.shared_placement) stayed legacy. The mismatch produced a conflict on the placement maps when reading journal records. Now bdev_jm_create takes a shared_placement arg (default off, omitted when False, mirroring bdev_distrib_create) and the three storage_node_ops call sites pass cluster.shared_placement -- so a restart reproduces the cluster's actual mode and JM/distrib stay consistent. 2. jm_set_shared_placement was called with no name, but the data-plane RPC requires one (there is exactly one JM per node, jm_<node_id>); the call errored. The client method now makes name mandatory and cluster_ops.set_shared_placement passes jm_<node_id>. 3. Because the JM RPC always errored (#2), set_shared_placement returned False, the monitor never cleared shared_placement_migration_pending, and it re-fired the migration on every tick (the "every few seconds" symptom). Fixing #2 lets the one-shot migration complete and clear the flag. Tests updated to match: JM create emits the flag only when opted in, and the JM runtime RPC requires a name. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
schmidt-scaled
added a commit
that referenced
this pull request
Jun 6, 2026
…RPC name Three defects in the JM shared_placement support (654f941) caused placement-map/journal conflicts and a runaway migration loop: 1. bdev_jm_create hard-coded shared_placement=True (JM_SHARED_PLACEMENT_DEFAULT), so every JM created on a node restart came up shared-placement even on a cluster that had NOT been migrated -- while distrib (which honors cluster.shared_placement) stayed legacy. The mismatch produced a conflict on the placement maps when reading journal records. Now bdev_jm_create takes a shared_placement arg (default off, omitted when False, mirroring bdev_distrib_create) and the three storage_node_ops call sites pass cluster.shared_placement -- so a restart reproduces the cluster's actual mode and JM/distrib stay consistent. 2. jm_set_shared_placement was called with no name, but the data-plane RPC requires one (there is exactly one JM per node, jm_<node_id>); the call errored. The client method now makes name mandatory and cluster_ops.set_shared_placement passes jm_<node_id>. 3. Because the JM RPC always errored (#2), set_shared_placement returned False, the monitor never cleared shared_placement_migration_pending, and it re-fired the migration on every tick (the "every few seconds" symptom). Fixing #2 lets the one-shot migration complete and clear the flag. Tests updated to match: JM create emits the flag only when opted in, and the JM runtime RPC requires a name. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
schmidt-scaled
added a commit
that referenced
this pull request
Jul 2, 2026
…ver force-strands devices Fix #2's sibling guard (skip the global device force when a sibling on the same home node also reached IO-error quorum) checked sibling quorum at the *instant* the current device tripped. But the two devices on a node do not trip together: their events are processed on independent per-node collector threads and cross quorum a few seconds apart (observed ~2s), and the first to trip clears its own votes on force. So neither sibling ever observes the other's quorum -- the first tripper doesn't yet see the second, and after it forces+clears, the second can't see the first. The guard fired 0x while both siblings were forced globally UNAVAILABLE, stranding the devices until a full node restart. Live incident (6-node mixed-churn soak, iodepth 128): iter-4 network_outage_20 on node b9a6aad8 force-marked both its devices (storage_id 6,7) at 12:15:37/39; they were never re-admitted, leaving a phantom offline_devices=2 baseline that iter-5's dual outage stacked on top of -> 5 devices offline -> cluster SUSPENDED. Fix: remember which devices were quorum-forced (_recent_quorum_forced, stamped before votes are cleared). Treat a sibling that reached quorum OR was force-marked within SIBLING_FORCE_EVIDENCE_SEC as node-outage evidence, and re-admit a sibling that was wrongly forced in the race (only when status=UNAVAILABLE, io_error=False, not retries_exhausted -- a genuine local fault is left to the restart/removal path). Now the second sibling to trip recognizes the node outage and undoes the first force, so offline_devices returns to 0 on node recovery instead of wedging. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
schmidt-scaled
added a commit
that referenced
this pull request
Jul 2, 2026
…ver force-strands devices Fix #2's sibling guard (skip the global device force when a sibling on the same home node also reached IO-error quorum) checked sibling quorum at the *instant* the current device tripped. But the two devices on a node do not trip together: their events are processed on independent per-node collector threads and cross quorum a few seconds apart (observed ~2s), and the first to trip clears its own votes on force. So neither sibling ever observes the other's quorum -- the first tripper doesn't yet see the second, and after it forces+clears, the second can't see the first. The guard fired 0x while both siblings were forced globally UNAVAILABLE, stranding the devices until a full node restart. Live incident (6-node mixed-churn soak, iodepth 128): iter-4 network_outage_20 on node b9a6aad8 force-marked both its devices (storage_id 6,7) at 12:15:37/39; they were never re-admitted, leaving a phantom offline_devices=2 baseline that iter-5's dual outage stacked on top of -> 5 devices offline -> cluster SUSPENDED. Fix: remember which devices were quorum-forced (_recent_quorum_forced, stamped before votes are cleared). Treat a sibling that reached quorum OR was force-marked within SIBLING_FORCE_EVIDENCE_SEC as node-outage evidence, and re-admit a sibling that was wrongly forced in the race (only when status=UNAVAILABLE, io_error=False, not retries_exhausted -- a genuine local fault is left to the restart/removal path). Now the second sibling to trip recognizes the node outage and undoes the first force, so offline_devices returns to 0 on node recovery instead of wedging. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
schmidt-scaled
pushed a commit
that referenced
this pull request
Jul 20, 2026
…oot loop FD-0 whole-domain reboot (2026-07-20) looped restart->abort->offline and dropped fio to EIO on the surviving mirror. Root cause: 16-way parallel FD recovery saturated the Python GIL, inflating the CP-local work between port-block RPCs so the client-port block on the serving counterpart exceeded the 6s nvmf ack-timeout and reject-converted the live client; and the coverage gate required down same-FD siblings' devices, so every recovering node aborted waiting on its down peers (circular dependency). Fixes (all unit-tested, tests/unit/test_restart_cpu_fixes.py): - A: async (queue-based) logging handler (utils.make_async_handler) removes the process-global handler-lock + stdout contention under 100+ threads. - B: bounded connect/reconnect workers (_restart_worker_sem / _bounded_thread, RESTART_WORKER_MAX_CONCURRENCY=24) cap GIL-bound fan-out. - C-safe: _connect_device_thread fast-skips (single attempt, no backoff) when the owning peer is not ONLINE — cuts ~6s/device on down peers. - #1: replace the process-global _recreate_lvstore_gate with a per-LVS lock registry; same-LVS recreates serialize, different-LVS run concurrently; activation_mode bypasses the lock. - #2: coverage gate excludes down same-failure-domain siblings from the required set only under a sanctioned dead-FD recovery (fd_dead_recovery_allowed = FTT floor), breaking the restart loop while preserving the 2026-07-16 silent-connect-failure protection. - Per-LVS port-block-duration print right after unblock in both recreate paths. - collect_logs.py: drop the O(N^2) per-node `sn check` block. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
schmidt-scaled
pushed a commit
that referenced
this pull request
Jul 21, 2026
Window decomposition (2026-07-21, 3.3s window): ~45 RPCs x ~50ms CP-observed + 1.46s CP gaps. This removes the movable content: #1 pre-stage hublvol NVMf subsystems + listeners BEFORE the block: StorageNode.prestage_hublvol_subsystem() (subsystem/listener half of expose_bdev — no lvstore/bdev dependency; identical parameters incl. the disjoint-cntlid requirement). Wired pre-block in both impls (leader hublvol, transferhub when persisted, sec_1 shared-NQN subsystem; non-leader: own secondary-hublvol subsystem). In-window expose reduces to probe+add_ns. Idempotent; failure falls back to in-window creation. #2 coordinator success stamp deferred: externally-managed locks mark pending_stamp instead of paying the FDB txn in-window; the impls' release helpers stamp post-unblock (cooldown shifts later — strictly more conservative). #3 port deny/allow event emission (FDB+graylog, ~190ms measured in-window) deferred to post-unblock/abort flush; block-span logs carry the true timing, events stay complete for audit. #4 examine-idempotency probes (raid/lvstore exists) computed pre-block in both impls — they read only snode's fresh SPDK, built pre-block. Expected: in-window content drops by ~8-10 RPCs + the event write + the stamp txn per window (~3.3s -> ~2s); reaching <1s additionally needs the monitor-baseline trim (follow-up #5) since examine + leadership + attach are the intrinsic in-window floor at the current ~50ms/RPC GIL tax. 7 new tests; full unit suite: 784 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mxsrc
pushed a commit
to mxsrc/sbcli
that referenced
this pull request
Jul 27, 2026
…in_shutdown self-heal A concurrent `sn shutdown` of two nodes wedged a node in `in_shutdown` permanently (incident 2026-06-18). set_node_status committed in_shutdown->offline via atomic CAS, but a concurrent full-object write_to_db() (which serializes the WHOLE row) re-wrote a stale in_shutdown. No offline->in_shutdown event existed — a silent lost update. The node then deadlocked: the monitor skips in_shutdown nodes, re-shutdown is rejected, and the peer's concurrent shutdown is rejected. Root cause: atomic_update only prevents lost updates if EVERY writer of a key uses it. Any `x=db.get(); x.field=v; x.write_to_db()` on a concurrently-written row clobbers other fields (notably status). Device status lives inside the StorageNode row, so every device write during a shutdown full-writes the node and reverts node.status. Fix simplyblock#1 — route clobbering writers through db.atomic_update (CAS): - device_controller: _atomic_device_set helper; device state/io_error/ retries/JM state/peer remote_devices/recreate/add-from-failed - health_check_service: node + device health, remote_jm_devices - health_controller: remote_devices rebuild - mgmt_node_monitor: MgmtNode status - storage_node_monitor: cluster is_re_balancing + shared_placement flags - lvol_monitor: status/health/io_error/deletion_status - lvol_stat_collector: io_error clear - cluster_ops: set_shared_placement, add_replication - backup_controller: backup_source Fix simplyblock#2 — self-heal: new StorageNode.shutdown_since (stamped in set_node_status, mirrors down_since); the monitor reconciles a node stuck in_shutdown past a 120s grace with SPDK confirmed dead back to OFFLINE. auto_restart_disabled stays set so an intentionally-stopped node is not auto-restarted. Only in_shutdown needed a backstop — cluster/lvol/device status self-correct each monitor cycle, and restarting is owned by a retrying task runner. test_shared_placement: DB fake updated to model atomic_update faithfully. Full suite: 1288 passed, 257 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
schmidt-scaled
pushed a commit
that referenced
this pull request
Aug 13, 2026
replication_commit took one fire-and-forget internal snapshot, selected a cutover base that was 1-2 replication intervals old, and enqueued the final task — which froze immediately, WITHOUT waiting for anything to replicate. The writes between the last replicated snapshot and the pre-commit snapshot were covered neither by the final step's delta (top blob only = writes after the pre-commit snapshot) nor by anything on the target: every cutover silently lost up to ~2 intervals of data. Invisible to the harness because fio's sequential sweep rewrites its whole working set and verify_backlog only checks recent writes. The cutover now shrinks the delta iteratively before freezing: snapshot #1 (at commit) -> wait until replicated AND converted on the target -> IMMEDIATELY snapshot #2 (delta = just the wait window) -> wait again -> IMMEDIATELY build the target clone on that last replicated snapshot and run the freeze + ANA flip. replication_commit is now thin (validate, shrink snapshot #1, enqueue); the final-task runner owns the shrink state machine (bounded by REPL_CUTOVER_SHRINK_TIMEOUT_SEC, waiting does not burn task retries) and the clone/map-id/replication-record preparation, so the base is always the freshly replicated shrink snapshot. Old-style tasks with tgt_* params still run unchanged (skip shrink + prepare). Harness: case 1 now writes a baseline fio never touches and md5-verifies it through the cutover volume after a remount — the ONLY assertion that exercises the replicated snapshot history (fio's own data always arrives via the final step); target paths are connected inside the cutover wait loop as soon as the runner creates the target volume, keeping multipath ahead of the ANA flip. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RaunakJalan
pushed a commit
that referenced
this pull request
Aug 31, 2026
replication_commit took one fire-and-forget internal snapshot, selected a cutover base that was 1-2 replication intervals old, and enqueued the final task — which froze immediately, WITHOUT waiting for anything to replicate. The writes between the last replicated snapshot and the pre-commit snapshot were covered neither by the final step's delta (top blob only = writes after the pre-commit snapshot) nor by anything on the target: every cutover silently lost up to ~2 intervals of data. Invisible to the harness because fio's sequential sweep rewrites its whole working set and verify_backlog only checks recent writes. The cutover now shrinks the delta iteratively before freezing: snapshot #1 (at commit) -> wait until replicated AND converted on the target -> IMMEDIATELY snapshot #2 (delta = just the wait window) -> wait again -> IMMEDIATELY build the target clone on that last replicated snapshot and run the freeze + ANA flip. replication_commit is now thin (validate, shrink snapshot #1, enqueue); the final-task runner owns the shrink state machine (bounded by REPL_CUTOVER_SHRINK_TIMEOUT_SEC, waiting does not burn task retries) and the clone/map-id/replication-record preparation, so the base is always the freshly replicated shrink snapshot. Old-style tasks with tgt_* params still run unchanged (skip shrink + prepare). Harness: case 1 now writes a baseline fio never touches and md5-verifies it through the cutover volume after a remount — the ONLY assertion that exercises the replicated snapshot history (fio's own data always arrives via the final step); target paths are connected inside the cutover wait loop as soon as the runner creates the target volume, keeping multipath ahead of the ANA flip. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.