Skip to content

26.4.0-beta2

Pre-release
Pre-release

Choose a tag to compare

View all tags
@schmidt-scaled schmidt-scaled released this 19 Oct 18:58
· 39 commits to R26.4-Beta since this release
26.4.0-beta2
35c9c26

simplyblock 26.4.0-beta2 — release notes

Released 2026-10-19 (pre-release). Validated by the full two-node AWS bed regression c13 (18 runs, all green) and the two-node simulator in sbcli CI.

Changes since 26.4.0-beta1 (2026-10-06): sbcli main +77 commits, operator main +87, documentation main +55, ultra main (#206–#221), SPDK new_md_journal (#76–#81). simplyblock-dr main has only 4 PRs since beta1. AI discovery lives on feature/ai-discovery and does not ship in beta2 (see "Not in this release").

Highlights

Two-node HA with arbitration (edge / small clusters)

A two-storage-node cluster can now fail over safely, with the control plane as the witness (arbiter). It needs two_node_arbitration on the cluster.

  • Node side (ultra JC + SPDK lvol):
    • hold, lease and epoch;
    • grant / fence / unfence RPCs (jc_grant_solo, jc_fence, jc_unfence, jc_lease_renew, jc_ha_status, jc_wait_events long-poll) (ultra #206, SPDK #76);
    • the fence blocks only the client port and saves/restores every ANA group (SPDK #80);
    • a grant stops a non-leader's hublvol redirect (SPDK #81).
  • Control plane (sbcli #1458/#1459/#1466):
    • the TwoNodeArbiter service: leases, fence-then-grant verdicts, healing, the arbitration API;
    • a Go TwoNodeEventCollector (long-poll → FDB event queue).
  • Failure behaviour:
    • Crash, reboot, shutdown, client-path loss: the survivor is granted and serves IO.
    • Partition: both nodes fence (no preferred node), and recovery runs automatically after the heal.
    • A node waking from a freeze longer than its 1.5 s lease aborts (ultra #211) and recovers through restart.
  • Recovery: the arbiter re-admits nodes and restores leadership and ANA after an unfence. A data rebuild runs after re-admission; a node with a stale copy is never granted (sbcli #1472/#1476).
  • Freeze-case fixes (ultra #214, #215, #217, #219):
    • Journal reader: it rescans the remaining safe JMs instead of aborting when its source JM is lost (#214). A solo volume group's reader skips the dead peer JM and abandons a history read in flight on it (#217/#219, F21/F26).
    • Release on grant: held client writes are released on every grant (#215).
    • Solo grant (#219): isolates the lost peer's data devices (F23/F24) and doesn't wait for the append lock on the frozen peer's JM (F25).
  • Client defaults on arbitrated two-node clusters: NVMe keep-alive 2 s (--keep-alive-tmo=2 --fast_io_fail_tmo=8 --ctrl-loss-tmo=3600 --reconnect-delay=2) (sbcli #1494).
  • Operator (#721): see the Operator section.
  • Regular clusters (≥3 nodes) and single-node clusters are unaffected: every two-node change is gated at runtime; restart, failover and failback paths are byte-identical (sbcli #1475, ultra #209, SPDK #79).
  • Validation: a two-node simulator runs in sbcli CI (#1488 and follow-ups #1490–#1508), plus a 13-scenario AWS bed regression (c13, 18 runs): all green. Crash, freeze (4 s and 8 s), reboot, graceful shutdown, client-path loss and partition scenarios pass; worst client stall 2 s (limit 3 s); 02c (8 s freeze of the primary) 5/5 at 1–2 s; no IO errors outside the by-design partition cases, no verify failures, natural recovery with ANA restored on all groups.

Small clusters

  • Single-node clusters run every erasure-coding scheme (operator #717).
  • Relaxed distrib migrations on 1–2-node clusters, and on clusters with fewer nodes than chunks per stripe (sbcli #1460).

pNFS: shared file storage (ReadWriteMany) at block performance (new feature)

Volumes with fstype: pnfs are served by a pNFS export. A metadata server (MDS) handles the metadata, and clients read and write directly over NVMe/TCP to the simplyblock volume, in parallel.

  • Operator / CSI:
    • Core: ReadWriteMany volumes served by a pNFS export (#550); the nfsexport gRPC protocol (#576); design doc (#577).
    • MDS: a pod-hosted pNFS metadata server (#683), on OpenShift and on a storage node (#731). One Service per MDS, plus cleanup of dead pNFS mounts (#733). The guest's nfsd is watched and logged to the console (#710), zeroed pNFS write layouts can be enabled in the guest (#714), and the MDS StatefulSet updates when its pod template changes (#707).
    • Robustness:
      • #681: the CSI link is served in every install path, and nfsd is started v4-only;
      • #682: pNFS is served from the plugin container on a host with no nfs-utils;
      • #697: survives an MDS restart without EACCES, and unstages a deleted volume;
      • #706: adds the first export to an empty root file without panicking;
      • #711: withdraws a departing MDS's address, and the nodes forget its flows;
      • #715: never truncates the layout probe, and probes one mount at a time.
    • Test tooling (sbtest): restarts/churn/recovery evidence (#698), expand and snapshot under load (#704), metadata churn across nodes (#705), cross-node write verification with the MDS restarted in every run (#713), conntrack/df fixes (#709).
  • SPDK (new_md_journal): extended support for pNFS features / NVMe-oF reservations (#77).
  • Docs:
    • initial pNFS / ReadWriteMany documentation (#238);
    • #240: the distributed file storage concept with diagrams, pNFS in Performance and QoS, the deployment architectures, and the prerequisites (KVM-capable node with /dev/kvm, the storage.simplyblock.io/kvm-capable label, a client kernel with CONFIG_PNFS_BLOCK and nvme_get_unique_id — RHEL 9.8 validated — no nfs-utils needed, NFSv4.1/4.2 over TCP 2049 to a per-export ClusterIP, spec.pnfs.mds ≥ 512Mi).
  • Limitations:
    • striped pNFS volumes: design draft only;
    • creating a pNFS volume from a snapshot or clone isn't supported;
    • DR of pNFS volumes isn't supported.

Operator (simplyblock-operator, main since v26.4.0-beta1)

  • Two-node HA and small clusters (#721, which brings in #712, #716, #717, #719): spec.twoNode and the arbitration controller (storage-fenced taint, remediation status). A declared two-node cluster activates 1+1 on two nodes; single-node clusters run any EC scheme; preferredNode is deprecated and ignored.
  • Console:
    • PREVIEW: the AI-assisted discovery and DR proposals pages (#722); the backend isn't in this release, see "Not in this release";
    • choose the test network's DHCP server per DR path (#679);
    • console fixes: no empty add-selectors, compact tiers editor, the hub namespace taken out of the form (#678).
  • pNFS: see above (#550, #576, #577, #681–#683, #697, #698, #704–#707, #709–#711, #713–#715, #731, #733).
  • CSI driver:
    • #384: recover NodeStageVolume when staging was interrupted before the stash was written;
    • #687: route ReplicationClient through TLSConnect;
    • #676: deploy the csi-addons sidecar only when enableCSIAddons is set;
    • #465: remove the bundled csi-driver-host-path and its local-hostpath StorageClass.
  • Backup: on-demand backup requests and cross-namespace restore (#708); a backup request asks the control plane for one backup (#724).
  • Pinned volumes / migration:
    • #685: pinned volumes, one migration per cluster, and validation results;
    • #684: a new move is raised when a pin returns to a node it moved to before;
    • #677: the webhook sends every pin-annotation spelling to the validator.
  • Rebalancer:
    • #689: read the baseline from the StorageNode;
    • #688: pin the baseline Job to the worker;
    • #691: scrape the probe sidecar;
    • #736: the webhook sends the rebalancer injector only to SPDK pods.
  • Control plane / chart / install:
    • #720: build in-cluster names from a configurable cluster DNS domain;
    • #693: verify ControlPlaneOps rollouts from the Deployment;
    • #726: render a StorageClass for each install-time storage cluster;
    • #729: a maintenance window waits for the worker to be uncordoned;
    • #703, #700: cleanup deletes the storage clusters' StorageClasses and releases volumes before uninstalling the CSI driver;
    • #668: drop the unmounted restart-script ConfigMap;
    • #672: keep every released helm chart in the repo;
    • #674: base image revert;
    • #680, #723, #730: openapi.json sync.
  • Security / CI:
    • #696: golang.org/x/net v0.60.0;
    • #728: publish to the internal registry;
    • #742: daily CVE scan of the published 26.3/26.4 images;
    • #699: tests skip udev's temporary by-id links;
    • #735: sbtest says a client hung when its writes reached neither path.

Backup

  • Direct backup creation from a volume without a snapshot (sbcli #1464).
  • On-demand backup requests and cross-namespace restore (operator #708, #724; docs #241).

Replication

  • Replication node failover: send from any online source member, transfer only to a settled target leader (sbcli #1441).
  • Failback quiesce, listeners created in their ANA state (#1439).
  • Fixes #1461 and #1463.

Other control-plane changes (sbcli)

  • Port blocking goes only through the SPDK RPC; the FirewallClient fallback is removed (#1497).
  • Fenced bdev_lvol_set_leader gets its own 2 s budget (#1495).
  • Monitor and migration:
    • a skipped ANA failover is retried (#1480);
    • a non-operable cluster defers the migration runners (#1479);
    • the ANA failback can't abort an lvstore recreate (#1453);
    • the lvstore poller group runs on the nvmf poller cores (#1452).
  • RPC client refactor (_request2 / _request3 removed, #1430–#1457).
  • Security: CVE fixes (#1437, #1473, #1499).
  • k8s: spdk pods self-describe their port (#1451).

Storage plane (ultra / SPDK)

  • ultra: SPDK no longer flushes the host's iptables on start (#218 on R26.3, also on main).
  • SPDK new_md_journal: cluster allocation speed-up (#78).

Documentation

  • Edge / two-node: clusters, device-failure tolerance, journal RAID, relaxed migration (#236, #239); the arbiter / control-plane availability requirement for two-node (#242).
  • pNFS: concepts, performance and prerequisites (#238, #240).
  • DR: the DR overview, plus discovery / environment promotion / SRM import pages labelled preview (#237); the backend doesn't ship in this release.
  • Backup: #241.
  • Upgrade: the R25→R26 multi-cluster upgrade notes (#243).

Known limitations

  1. Two-node needs the arbiter. A survivor only continues after a grant from the control plane. If the control plane / arbiter is unreachable when a node fails, the survivor fences itself at the 2.5 s hold deadline, so the volume is unavailable until the arbiter returns. Run the control plane HA, or in a third failure domain. An arbiter outage alone, with both nodes healthy, has no effect.
  2. Freeze stalls / to be confirmed: if a storage node's SPDK freezes without crashing, client IO pauses briefly (≤ 2 s in c13), with no IO error and no data loss. The F26 path (abandoning a journal history read already in flight on the frozen peer's JM) is covered by unit tests and the simulator, but was not exercised on the bed (no read was in flight in the 5 freeze runs; the race is intermittent). A generic "quarantine all connections to the lost peer at the grant" mechanism is planned to replace the per-path fixes.
  3. Not root-caused: F9 (distribs rejecting the cluster map after a reboot, seen once in c3) and F11 (journals not reconnecting after both links healed, seen once in c3). Neither recurred in c5–c12b.
  4. Partition with a lossy link (10b): both nodes fence by design, so the client sees IO errors until the link heals. Recovery is automatic.
  5. PREVIEW: AI-assisted discovery, blueprints (UAT→PRD) and the SRM import. The Control Center pages (#722) and the documentation pages (#237) ship as a preview; the backend (dr-hub discovery, DRBlueprint, SRMImport) is on simplyblock-dr feature/ai-discovery and is not part of this release.

Upgrade notes

  • Two-node clusters: enable arbitration (two_node_arbitration / operator spec.twoNode). Existing two-node clusters without it keep the plain dual-node behaviour (no split-brain protection).
  • The client KATO change (2 s) applies to new connections on arbitrated two-node clusters only. Reconnect clients to pick it up.
  • Port fencing now goes only through the SPDK RPC (#1497): the FirewallClient fallback and the firewall-port plumbing are removed: the storage node model's firewall_port (default 5001) and the firewall_port parameter of the SPDK container start are gone, and the SPDK deployment template no longer exposes the firewall web app (see also ultra #218, which removed iptables from the SPDK image).
  • The operator's preferredNode is ignored (deprecated).
  • Storage-plane image: pinned to public.ecr.aws/simply-block/ultra:main-48b669e4 (digest sha256:b145ceba08e5953ec2fce01eb461b27194059c605c510aacd6c627f66c71978d; ultra main 48b669e4, code identical to the tested 37b8ccec), built on spdk-core new_md_journal 5d449ef7a (not master). Do not use ultra:main-latest: SPDK new_md_journal has a newer, untested commit (6473b954).

Not in this release

  • simplyblock-dr AI-assisted discovery (ADR 0023), blueprints (UAT→PRD) and the SRM import (ADR 0024): merged on simplyblock-dr feature/ai-discovery only. simplyblock-dr main has only the design doc (#14).
    • The console pages (operator #722) and docs pages (#237) are included and labelled preview; they need dr-hub from feature/ai-discovery.
  • sbcli #1471 (Go event collector → Python) is open, not included; the user decided to keep Go for now.
  • The generic peer-IO quarantine is planned.
Assets 2
Loading