feat: design ClusterOps CR to decouple cluster operations from the StorageCluster reconciler

[geoffrey1330]

Fixes: issue-392

Summary

Introduces StorageClusterOps, a new one-shot CRD for driving cluster-level operations (activate, expand, shutdown, restart, node-recycle) to completion and recording the result. Modelled after the existing StorageNodeOps pattern.

• New CRD storageclusterops.storage.simplyblock.io (scops short name) with Pending → Running → Succeeded/Failed lifecycle phases
• Mutual exclusion via activeOpsRef on StorageClusterStatus — only one StorageClusterOps active per cluster at a time; concurrent ops requeue until the lock is free
• Triggered guard on StorageClusterOpsStatus prevents duplicate backend POSTs when a status patch fails after a successful mutation
• Controller dispatches to per-action reconcilers (reconcileActivate, reconcileExpand, reconcileSimplePost, reconcileNodeRecycle), each polling until the backend confirms completion
• Helm CRD added to simplyblock-operator/crds/ and example CR added to operator_customresources.yaml
• RBAC markers and config/rbac/role.yaml updated to storageclusterops (all three subresources: base, /status, /finalizers)
• Unit tests (12 cases) covering terminal phase no-ops, cluster not found, mutual exclusion requeue, lock acquire/release ownership, nil-cluster safety, unknown action, and missing nodeUUID for node-recycle
• Design doc at operator/docs/designs/design-storageclusterops.md

Test plan

• go test ./internal/controller/ -run TestStorageClusterOps -v — all 12 tests pass
• make manifests generates storage.simplyblock.io_storageclusterops.yaml with correct kind, plural, and short name (scops)
• Apply two StorageClusterOps CRs targeting the same cluster simultaneously — second one should stay Pending and requeue until the first completes
• Verify kubectl get scops works via short name

Script Based Tests

# Tests:
#   1  — Short name: kubectl get scops works
#   2  — Unknown action rejected at admission by CRD enum validation
#   3  — node-recycle: initialises (Triggered=true) and eventually Succeeds (E2E)
#   4  — Reference to non-existent cluster → Failed phase
#   5  — Mutual exclusion: second ops requeues while first holds activeOpsRef
#   6  — activeOpsRef cleared after ops completes (Succeeded or Failed)
#   7  — Events emitted on StorageClusterOps CR
#   8  — Activate: cluster transitions to active (E2E, requires inactive cluster)
#   9  — Expand: cluster expands capacity (E2E)
#  10  — Shutdown: cluster shuts down (E2E, destructive)
#  11  — Restart: cluster restarts (E2E, destructive)

Test Result

user@users-MacBook-Pro-2 ~ % kubectl -n simplyblock get scops -w
NAME                CLUSTER               ACTION         PHASE     MESSAGE                                                       AGE
test-node-recycle   simplyblock-cluster   node-recycle   Running   Node 3/3 (43c684c4-f26f-4349-8ea0-78bdcddec18c): restarting   4m44s

test-node-recycle   simplyblock-cluster   node-recycle   Running   Node 3/3 (43c684c4-f26f-4349-8ea0-78bdcddec18c): rebalancing   5m29s

test-node-recycle   simplyblock-cluster   node-recycle   Succeeded   All nodes recycled successfully                                6m37s

user@users-MacBook-Pro-2 ~ % kubectl -n simplyblock get scops   
NAME           CLUSTER               ACTION   PHASE     MESSAGE                                           AGE
test-start   simplyblock-cluster   start    Running   Starting — waiting for cluster to become active   62s

test-start   simplyblock-cluster   start    Succeeded   Cluster started successfully                      3m45s

user@users-MacBook-Pro-2 ~ % kubectl -n simplyblock get scops
test-restart   simplyblock-cluster   restart   Running   shutting-down   0s
test-restart   simplyblock-cluster   restart   Running   starting        5s

test-restart   simplyblock-cluster   restart   Succeeded   Cluster restarted successfully   3m29s

user@users-MacBook-Pro-2 ~ % kubectl -n simplyblock get scops -w
NAME            CLUSTER               ACTION     PHASE     MESSAGE                                         AGE
test-activate   simplyblock-cluster   activate   Running   activate POST sent, polling for active status   5s
test-activate   simplyblock-cluster   activate   Succeeded   Cluster activated successfully                  11s

user@users-MacBook-Pro-2 ~ % kubectl -n simplyblock get events     
LAST SEEN   TYPE     REASON      OBJECT                            MESSAGE
15m         Normal   Activated   storageclusterops/test-activate   Cluster simplyblock-cluster activated successfully
17s         Warning   Failed             storageclusterops/test-events                StorageClusterOps test-events failed: cluster "does-not-exist" not found

 Test 5 — Mutual exclusion: second ops requeues while first holds activeOpsRef
══════════════════════════════════════════
storageclusterops.storage.simplyblock.io/test-mutex-a created
storageclusterops.storage.simplyblock.io/test-mutex-b created
[PASS] activeOpsRef='test-node-recycle' — only one ops active at a time
[PASS] Second ops is still Pending (phase=) while first holds the lock

[boddumanohar]

It also seems like we are removing the fields spec.action and spec.nodeRecycle. This will be a breaking change.

From 26.4 we should be careful not to include any breaking changes.

[geoffrey1330]

It also seems like we are removing the fields spec.action and spec.nodeRecycle. This will be a breaking change.

From 26.4 we should be careful not to include any breaking changes.

I think this is acceptable as no one is using the operator in production now. thus why the rush to implement this. cc @noctarius

[boddumanohar]

Okay, didn't know that this is going to be added to the release branch (26.3). Generally it's good to have this in this release make the CRDs more complete. Then in that case, it should be fine.

[Copilot]

Pull request overview

This PR introduces a dedicated StorageClusterOps one-shot CRD + controller to execute long-running, cluster-level operations (activate/expand/shutdown/start/restart/node-recycle) outside of the StorageCluster steady-state reconciler, aligning the design with the existing StorageNodeOps pattern.

Changes:

• Added StorageClusterOps API types, CRDs (operator + Helm), controller implementation (including node-recycle state machine), and controller registration.
• Removed legacy imperative action handling (spec.action, actionStatus, node-recycle logic) from StorageCluster API + reconciler and deleted the old actions implementation file.
• Updated RBAC/manifests and added unit tests, docs, and a shell regression test script for the new API.

Reviewed changes

Copilot reviewed 21 out of 24 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
operator/test/utils/test_storageclusterops.sh	Adds a manual regression/E2E script for StorageClusterOps behaviors (shortname, mutual exclusion, events, operations).
operator/internal/utils/constants.go	Adds a new finalizer constant for StorageClusterOps.
operator/internal/controller/storageclusterops_noderecycle.go	Implements node-recycle state machine for StorageClusterOps (including pod refresh helpers).
operator/internal/controller/storageclusterops_controller.go	New controller handling locking via activeOpsRef, action dispatch, finalizers, and operation lifecycles.
operator/internal/controller/storageclusterops_controller_unit_test.go	Adds unit tests for basic reconcile paths and lock behavior.
operator/internal/controller/simplyblockstoragecluster_controller.go	Removes inline cluster action dispatch from the StorageCluster reconciler.
operator/internal/controller/simplyblockstoragecluster_controller_unit_test.go	Removes unit tests tied to legacy inline cluster actions and node-recycle write-ahead behavior.
operator/internal/controller/simplyblockstoragecluster_actions.go	Deletes the legacy action-handling implementation previously used by StorageCluster.
operator/docs/tests/test-plan-storageclusterops.md	Adds a test plan document for StorageClusterOps.
operator/docs/designs/design-storageclusterops.md	Adds a design doc describing the CRD/controller architecture and migration strategy.
operator/dist/install.yaml	Updates bundled install manifests: adds StorageClusterOps CRD and removes legacy StorageCluster action fields; updates RBAC resources.
operator/config/rbac/role.yaml	Updates generated ClusterRole to include storageclusterops resources/subresources.
operator/config/crd/kustomization.yaml	Adds the new storageclusterops CRD base to kustomize inputs.
operator/config/crd/bases/storage.simplyblock.io_storageclusters.yaml	Removes legacy StorageCluster action-related schema and adds status.activeOpsRef.
operator/config/crd/bases/storage.simplyblock.io_storageclusterops.yaml	Adds the generated CRD for StorageClusterOps.
operator/cmd/main.go	Registers the new StorageClusterOpsReconciler with the manager.
operator/api/v1alpha1/zz_generated.deepcopy.go	Updates generated deepcopy code for new/removed API types.
operator/api/v1alpha1/storagenodeset_types.go	Removes the legacy ActionStatus type (now unused after action refactor).
operator/api/v1alpha1/storageclusterops_types.go	Adds the StorageClusterOps API types and status structs.
operator/api/v1alpha1/storagecluster_types.go	Removes legacy action fields/status from StorageCluster and adds ActiveOpsRef to status.
helm-charts/charts/simplyblock-operator/templates/roles/manager_role.yaml	Updates Helm role template to include storageclusterops resources/subresources.
helm-charts/charts/simplyblock-operator/crds/storage.simplyblock.io_storageclusters.yaml	Updates Helm-shipped StorageCluster CRD schema (remove action fields; add activeOpsRef).
helm-charts/charts/simplyblock-operator/crds/storage.simplyblock.io_storageclusterops.yaml	Adds Helm-shipped StorageClusterOps CRD.
atlas-lib/internal/cpapi/gen/main_test.go	Minor formatting-only adjustment (newline/indent).

Files not reviewed (1)

• operator/api/v1alpha1/zz_generated.deepcopy.go: Generated file

Suppressed comments (2)

operator/internal/controller/storageclusterops_controller.go:566

• releaseClusterLock drops errors from the status patch (_ = r.Status().Patch(...)). If this patch fails (conflict, transient apiserver issue), activeOpsRef can remain stuck and block all future ops with no logs to explain why.

	patch := client.MergeFrom(cluster.DeepCopy())
	cluster.Status.ActiveOpsRef = ""
	_ = r.Status().Patch(ctx, cluster, patch)

operator/docs/tests/test-plan-storageclusterops.md:98

• The Node Recycle E2E scenarios describe nodeUUID-scoped API calls (/node-recycle/{nodeUUID}), but the implementation recycles all nodes by listing them and then calling per-node /storage-nodes/{uuid}/shutdown|restart endpoints. The test plan should be updated to reflect the implemented behavior.

| Scenario | Expected | Classification |
|---|---|---|
| Create `StorageClusterOps(action=node-recycle, nodeUUID=<uuid>)` | Backend `/cluster/{uuid}/node-recycle/{nodeUUID}` called; ops `Succeeded` | E2E |
| Provide an invalid `nodeUUID` | Backend returns non-2xx; ops transitions to `Failed` with error message | E2E |

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[geoffrey1330]

Okay, didn't know that this is going to be added to the release branch (26.3). Generally it's good to have this in this release make the CRDs more complete. Then in that case, it should be fine.

I don't know if it will be added in this release. we can discuss it during the sync

[noctarius]

We should describe it the same way as Kubernetes names this operation, a "rolling restart" or "rollout restart"

[geoffrey1330]

what do you mean here?

[noctarius]

What happens when a storage node goes offline during a restart? In FTT=2, this would be valid but how would the code handle this situation?

[geoffrey1330]

Good question. The current code doesn't handle this. it only checks the status of the node it's actively restarting, not the status of the rest of the nodes. i will add this condition to pause the restart until the rest of the nodes are online

[geoffrey1330]

Added a pre-shutdown health check that verifies all other nodes are online before proceeding with each shutdown. If any peer is not online, the rolling restart holds and requeues every 30s until the cluster recovers, preventing us from pushing past the cluster's FTT.