fix iostats device not found error - #1
Closed
boddumanohar wants to merge 1 commit into
Closed
Conversation
hamdy-simplyblock
added a commit
that referenced
this pull request
Mar 2, 2024
hamdy-simplyblock
added a commit
that referenced
this pull request
Mar 14, 2024
hamdy-simplyblock
added a commit
that referenced
this pull request
Mar 27, 2024
hamdy-simplyblock
added a commit
that referenced
this pull request
Apr 21, 2024
hamdy-simplyblock
added a commit
that referenced
this pull request
Apr 29, 2024
hamdy-simplyblock
added a commit
that referenced
this pull request
May 3, 2024
hamdy-simplyblock
added a commit
that referenced
this pull request
May 6, 2024
set version: sbcli-mig==1.0.85
hamdy-simplyblock
added a commit
that referenced
this pull request
May 6, 2024
set version: sbcli-mig==1.0.86
hamdy-simplyblock
added a commit
that referenced
this pull request
May 6, 2024
set version: sbcli-mig==1.0.87
hamdy-simplyblock
pushed a commit
that referenced
this pull request
May 7, 2024
hamdy-simplyblock
pushed a commit
that referenced
this pull request
May 7, 2024
hamdy-simplyblock
pushed a commit
that referenced
this pull request
May 7, 2024
hamdy-simplyblock
added a commit
that referenced
this pull request
May 8, 2024
geoffrey1330
pushed a commit
that referenced
this pull request
May 8, 2024
set version: sbcli-mig==1.0.87
hamdy-simplyblock
added a commit
that referenced
this pull request
May 10, 2024
hamdy-simplyblock
added a commit
that referenced
this pull request
May 23, 2024
schmidt-scaled
added a commit
that referenced
this pull request
Apr 21, 2026
…host-disjointness Three independent fixes that unblock cluster_activate on HA clusters with max_fault_tolerance >= 2: 1. _connect_to_remote_jm_devs: remove the _peer_reachable_via_jm_quorum gate. During activation peer JC quorums are still bootstrapping, so the probe returns False for every intended member of the new jm_vuid and causes all remote_jm attaches to be silently skipped. The gate was introduced for runtime re-attach paths (rejoin/restart-task) which carry their own reachability checks; keeping it out of the activation path is correct. 2. StorageNode.connect_to_hublvol: call bdev_lvol_connect_hublvol BEFORE bdev_lvol_set_lvs_opts. set_lvs_opts needs the lvstore to exist in SPDK's local registry. On secondary the registry usually contains it via bdev_examine of the raid0 superblock, but on tertiary the raid0 has never seen IO so its superblock is empty and set_lvs_opts then returns -19 "No such device". Doing connect_hublvol first registers the lvstore regardless of raid0 state. 3. get_secondary_nodes_2: enforce host-disjointness against the already picked secondary. The previous logic only excluded the primary's mgmt_ip, so with two storage nodes per physical host the tertiary could land on the same host as the secondary, and a single host outage would take out two of the four HA journal members. Caller in cluster_ops.cluster_activate now passes exclude_mgmt_ips=[sec_node.mgmt_ip]; if no host-disjoint candidate exists, activation fails fast with the pre-existing "not enough nodes for dual fault tolerance" error. Tests: new test_activation_fixes.py covers #1 and #3 (8 tests). test_hublvol_unit.py updated for the reversed ordering and adds a test that set_lvs_opts is not called when connect_hublvol fails (30 tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
schmidt-scaled
added a commit
that referenced
this pull request
Apr 21, 2026
…host-disjointness Three independent fixes that unblock cluster_activate on HA clusters with max_fault_tolerance >= 2: 1. _connect_to_remote_jm_devs: remove the _peer_reachable_via_jm_quorum gate. During activation peer JC quorums are still bootstrapping, so the probe returns False for every intended member of the new jm_vuid and causes all remote_jm attaches to be silently skipped. The gate was introduced for runtime re-attach paths (rejoin/restart-task) which carry their own reachability checks; keeping it out of the activation path is correct. 2. StorageNode.connect_to_hublvol: call bdev_lvol_connect_hublvol BEFORE bdev_lvol_set_lvs_opts. set_lvs_opts needs the lvstore to exist in SPDK's local registry. On secondary the registry usually contains it via bdev_examine of the raid0 superblock, but on tertiary the raid0 has never seen IO so its superblock is empty and set_lvs_opts then returns -19 "No such device". Doing connect_hublvol first registers the lvstore regardless of raid0 state. 3. get_secondary_nodes_2: enforce host-disjointness against the already picked secondary. The previous logic only excluded the primary's mgmt_ip, so with two storage nodes per physical host the tertiary could land on the same host as the secondary, and a single host outage would take out two of the four HA journal members. Caller in cluster_ops.cluster_activate now passes exclude_mgmt_ips=[sec_node.mgmt_ip]; if no host-disjoint candidate exists, activation fails fast with the pre-existing "not enough nodes for dual fault tolerance" error. Tests: new test_activation_fixes.py covers #1 and #3 (8 tests). test_hublvol_unit.py updated for the reversed ordering and adds a test that set_lvs_opts is not called when connect_hublvol fails (30 tests). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
schmidt-scaled
added a commit
that referenced
this pull request
Jun 5, 2026
…eration settle Fix #1: the 2026-06-04 concurrent-shutdown fix made only the pre-outage gate (wait_for_outage_ready) tolerant of a peer the CP is autonomously restarting/shutting down; the inter-iteration settle, NIC-chaos, and post-outage waits still used the strict wait_for_all_online and could abort with "Unaffected nodes are not online" on the same CP-heal race. Add a tolerate_transient flag (new TRANSIENT_NODE_STATUSES = in_restart/in_shutdown) and enable it at all three inter-iteration sync points. The active-outage wait inside run_outage_pair stays strict -- there a wobbling third node IS an FTT-budget breach. Fix #4: remove the double inter-iteration settle. settle_between_iterations was called twice per iteration (once under the lock before NIC chaos, once unlocked after), starving the churn thread with two blind sleeps. Keep a single unlocked settle; NIC chaos stays under the lock and relies on its own all-online + cluster-stable waits. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
schmidt-scaled
pushed a commit
that referenced
this pull request
Jul 20, 2026
…oot loop FD-0 whole-domain reboot (2026-07-20) looped restart->abort->offline and dropped fio to EIO on the surviving mirror. Root cause: 16-way parallel FD recovery saturated the Python GIL, inflating the CP-local work between port-block RPCs so the client-port block on the serving counterpart exceeded the 6s nvmf ack-timeout and reject-converted the live client; and the coverage gate required down same-FD siblings' devices, so every recovering node aborted waiting on its down peers (circular dependency). Fixes (all unit-tested, tests/unit/test_restart_cpu_fixes.py): - A: async (queue-based) logging handler (utils.make_async_handler) removes the process-global handler-lock + stdout contention under 100+ threads. - B: bounded connect/reconnect workers (_restart_worker_sem / _bounded_thread, RESTART_WORKER_MAX_CONCURRENCY=24) cap GIL-bound fan-out. - C-safe: _connect_device_thread fast-skips (single attempt, no backoff) when the owning peer is not ONLINE — cuts ~6s/device on down peers. - #1: replace the process-global _recreate_lvstore_gate with a per-LVS lock registry; same-LVS recreates serialize, different-LVS run concurrently; activation_mode bypasses the lock. - #2: coverage gate excludes down same-failure-domain siblings from the required set only under a sanctioned dead-FD recovery (fd_dead_recovery_allowed = FTT floor), breaking the restart loop while preserving the 2026-07-16 silent-connect-failure protection. - Per-LVS port-block-duration print right after unblock in both recreate paths. - collect_logs.py: drop the O(N^2) per-node `sn check` block. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
schmidt-scaled
pushed a commit
that referenced
this pull request
Jul 21, 2026
Window decomposition (2026-07-21, 3.3s window): ~45 RPCs x ~50ms CP-observed + 1.46s CP gaps. This removes the movable content: #1 pre-stage hublvol NVMf subsystems + listeners BEFORE the block: StorageNode.prestage_hublvol_subsystem() (subsystem/listener half of expose_bdev — no lvstore/bdev dependency; identical parameters incl. the disjoint-cntlid requirement). Wired pre-block in both impls (leader hublvol, transferhub when persisted, sec_1 shared-NQN subsystem; non-leader: own secondary-hublvol subsystem). In-window expose reduces to probe+add_ns. Idempotent; failure falls back to in-window creation. #2 coordinator success stamp deferred: externally-managed locks mark pending_stamp instead of paying the FDB txn in-window; the impls' release helpers stamp post-unblock (cooldown shifts later — strictly more conservative). #3 port deny/allow event emission (FDB+graylog, ~190ms measured in-window) deferred to post-unblock/abort flush; block-span logs carry the true timing, events stay complete for audit. #4 examine-idempotency probes (raid/lvstore exists) computed pre-block in both impls — they read only snode's fresh SPDK, built pre-block. Expected: in-window content drops by ~8-10 RPCs + the event write + the stamp txn per window (~3.3s -> ~2s); reaching <1s additionally needs the monitor-baseline trim (follow-up #5) since examine + leadership + attach are the intrinsic in-window floor at the current ~50ms/RPC GIL tax. 7 new tests; full unit suite: 784 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mxsrc
pushed a commit
to mxsrc/sbcli
that referenced
this pull request
Jul 27, 2026
…in_shutdown self-heal A concurrent `sn shutdown` of two nodes wedged a node in `in_shutdown` permanently (incident 2026-06-18). set_node_status committed in_shutdown->offline via atomic CAS, but a concurrent full-object write_to_db() (which serializes the WHOLE row) re-wrote a stale in_shutdown. No offline->in_shutdown event existed — a silent lost update. The node then deadlocked: the monitor skips in_shutdown nodes, re-shutdown is rejected, and the peer's concurrent shutdown is rejected. Root cause: atomic_update only prevents lost updates if EVERY writer of a key uses it. Any `x=db.get(); x.field=v; x.write_to_db()` on a concurrently-written row clobbers other fields (notably status). Device status lives inside the StorageNode row, so every device write during a shutdown full-writes the node and reverts node.status. Fix simplyblock#1 — route clobbering writers through db.atomic_update (CAS): - device_controller: _atomic_device_set helper; device state/io_error/ retries/JM state/peer remote_devices/recreate/add-from-failed - health_check_service: node + device health, remote_jm_devices - health_controller: remote_devices rebuild - mgmt_node_monitor: MgmtNode status - storage_node_monitor: cluster is_re_balancing + shared_placement flags - lvol_monitor: status/health/io_error/deletion_status - lvol_stat_collector: io_error clear - cluster_ops: set_shared_placement, add_replication - backup_controller: backup_source Fix simplyblock#2 — self-heal: new StorageNode.shutdown_since (stamped in set_node_status, mirrors down_since); the monitor reconciles a node stuck in_shutdown past a 120s grace with SPDK confirmed dead back to OFFLINE. auto_restart_disabled stays set so an intentionally-stopped node is not auto-restarted. Only in_shutdown needed a backstop — cluster/lvol/device status self-correct each monitor cycle, and restarting is owned by a retrying task runner. test_shared_placement: DB fake updated to model atomic_update faithfully. Full suite: 1288 passed, 257 skipped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
schmidt-scaled
pushed a commit
that referenced
this pull request
Aug 13, 2026
replication_commit took one fire-and-forget internal snapshot, selected a cutover base that was 1-2 replication intervals old, and enqueued the final task — which froze immediately, WITHOUT waiting for anything to replicate. The writes between the last replicated snapshot and the pre-commit snapshot were covered neither by the final step's delta (top blob only = writes after the pre-commit snapshot) nor by anything on the target: every cutover silently lost up to ~2 intervals of data. Invisible to the harness because fio's sequential sweep rewrites its whole working set and verify_backlog only checks recent writes. The cutover now shrinks the delta iteratively before freezing: snapshot #1 (at commit) -> wait until replicated AND converted on the target -> IMMEDIATELY snapshot #2 (delta = just the wait window) -> wait again -> IMMEDIATELY build the target clone on that last replicated snapshot and run the freeze + ANA flip. replication_commit is now thin (validate, shrink snapshot #1, enqueue); the final-task runner owns the shrink state machine (bounded by REPL_CUTOVER_SHRINK_TIMEOUT_SEC, waiting does not burn task retries) and the clone/map-id/replication-record preparation, so the base is always the freshly replicated shrink snapshot. Old-style tasks with tgt_* params still run unchanged (skip shrink + prepare). Harness: case 1 now writes a baseline fio never touches and md5-verifies it through the cutover volume after a remount — the ONLY assertion that exercises the replicated snapshot history (fio's own data always arrives via the final step); target paths are connected inside the cutover wait loop as soon as the runner creates the target volume, keeping multipath ahead of the ANA flip. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RaunakJalan
pushed a commit
that referenced
this pull request
Aug 31, 2026
replication_commit took one fire-and-forget internal snapshot, selected a cutover base that was 1-2 replication intervals old, and enqueued the final task — which froze immediately, WITHOUT waiting for anything to replicate. The writes between the last replicated snapshot and the pre-commit snapshot were covered neither by the final step's delta (top blob only = writes after the pre-commit snapshot) nor by anything on the target: every cutover silently lost up to ~2 intervals of data. Invisible to the harness because fio's sequential sweep rewrites its whole working set and verify_backlog only checks recent writes. The cutover now shrinks the delta iteratively before freezing: snapshot #1 (at commit) -> wait until replicated AND converted on the target -> IMMEDIATELY snapshot #2 (delta = just the wait window) -> wait again -> IMMEDIATELY build the target clone on that last replicated snapshot and run the freeze + ANA flip. replication_commit is now thin (validate, shrink snapshot #1, enqueue); the final-task runner owns the shrink state machine (bounded by REPL_CUTOVER_SHRINK_TIMEOUT_SEC, waiting does not burn task retries) and the clone/map-id/replication-record preparation, so the base is always the freshly replicated shrink snapshot. Old-style tasks with tgt_* params still run unchanged (skip shrink + prepare). Harness: case 1 now writes a baseline fio never touches and md5-verifies it through the cutover volume after a remount — the ONLY assertion that exercises the replicated snapshot history (fio's own data always arrives via the final step); target paths are connected inside the cutover wait loop as soon as the runner creates the target volume, keeping multipath ahead of the ANA flip. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.