fix(rook-ceph): disable rook mgr module to stop crash storm - #1384
Merged
Conversation
The prometheus mgr module unconditionally calls node_proxy_fullreport() on every scrape, which the rook orchestrator backend doesn't implement. The resulting NotImplementedError is recorded as a crash at the C++ dispatch boundary every 15s, which races the crash module's unlocked do_post() into HEALTH_ERR. Disabling the rook module stops orch dispatch entirely, removing the crash generator. This repo doesn't use ceph orch (docs/ceph/toolbox.md already documents it as inapplicable here), so the module isn't needed. Upstream: rook/rook#18124, ceph/ceph#71041, ceph/ceph#71180 (all open, unfixed even on ceph main as of 2026-08-22).
Contributor
--- HelmRelease: rook-ceph/rook-ceph-cluster CephCluster: rook-ceph/rook-ceph
+++ HelmRelease: rook-ceph/rook-ceph-cluster CephCluster: rook-ceph/rook-ceph
@@ -97,13 +97,13 @@
- enabled: true
name: diskprediction_local
- enabled: true
name: insights
- enabled: true
name: pg_autoscaler
- - enabled: true
+ - enabled: false
name: rook
- enabled: true
name: nfs
- enabled: true
name: rgw
placement: |
Contributor
--- kubernetes/apps/rook-ceph/rook-ceph/cluster Kustomization: rook-ceph/rook-ceph-cluster HelmRelease: rook-ceph/rook-ceph-cluster
+++ kubernetes/apps/rook-ceph/rook-ceph/cluster Kustomization: rook-ceph/rook-ceph-cluster HelmRelease: rook-ceph/rook-ceph-cluster
@@ -126,13 +126,13 @@
- enabled: true
name: diskprediction_local
- enabled: true
name: insights
- enabled: true
name: pg_autoscaler
- - enabled: true
+ - enabled: false
name: rook
- enabled: true
name: nfs
- enabled: true
name: rgw
placement: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Disable the rook mgr module to stop a Ceph crash storm that is currently pinning cluster health at HEALTH_ERR/HEALTH_WARN and blocks every future Talos and Kubernetes node upgrade.
Root cause (from completed investigation, primary-source verified against the running mgr pod and upstream Ceph/Rook trackers): the prometheus mgr module unconditionally calls node_proxy_fullreport() on every scrape (15s), which the rook orchestrator backend does not implement, raising NotImplementedError. The mgr's C++ dispatch layer records a crash report for module 'rook' before the exception reaches prometheus's own try/except. This produces a crash every 15s. The crash module's do_post() mutates self.crashes without taking crashes_lock (the only mutator missing this), and the serve() thread iterating that same dict under the lock eventually collides with it, raising RuntimeError -> MGR_MODULE_ERROR -> HEALTH_ERR. Both are confirmed, open, unfixed upstream bugs (rook/rook#18124, ceph/ceph#71041, ceph/ceph#71180) with no released fix, including on ceph main.
Change: in kubernetes/apps/rook-ceph/rook-ceph/cluster/helmrelease.yaml, set the mgr.modules 'rook' entry from enabled: true to enabled: false, with a comment linking rook/rook#18124, ceph/ceph#71041, ceph/ceph#71180, noting this should be reverted once a tentacle point release ships containing both fixes.
Why safe: nothing in this repo uses ceph orch (grepped; only docs/ceph/toolbox.md documents it as the wrong control plane for this Rook-based cluster). The Ceph dashboard stays functional for everything else. The Rook operator reconciles from its own CRDs, not through this mgr module. This repo already sets mgr/crash/warn_recent_interval to 7200s (2h), so once the crash generator stops, RECENT_MGR_MODULE_CRASH self-clears within 2 hours with no manual archive-all/cleanup needed.
Cluster access is read-only for inspection/verification only; nothing was applied, patched, deleted, scaled, or annotated directly on the live cluster - this ships through Git and Flux only. No attempt was made to speed up the self-heal window via ceph crash archive-all or a manual mgr restart.
This is a live cluster mutation once merged: it only takes effect when Flux reconciles the HelmRelease change. Post-merge 'fixed' means: ceph mgr module ls no longer dispatches to rook for orchestrator calls; no new crash records accumulate over a several-minute window (ceph crash ls-new count/timestamps flat, not climbing at the ~4/minute rate previously measured); ceph -s returns to HEALTH_OK within the 2-hour self-heal window with no manual intervention.
Out of scope: do not touch kubernetes/apps/system-upgrade/tuppr/upgrades/kubernetesupgrade.yaml or talosupgrade.yaml. Do not attempt to fix the separately-flagged, lower-priority CephCrashesDetected alert's wrong health-check name in prometheusrules.yaml (RECENT_CRASH vs RECENT_MGR_MODULE_CRASH) - note only as a candidate follow-up, not part of this change.
What Changed
rookentry undermgr.modulesinkubernetes/apps/rook-ceph/rook-ceph/cluster/helmrelease.yamlfromenabled: truetoenabled: false, with an inline comment explaining the crash storm root cause and linking Tentacle v20.2.3 on Rook: permanent mgr RECENT_MGR_MODULE_CRASH — prometheus get_hardware_metrics() → node_proxy_fullreport() NotImplementedError rook/rook#18124, tentacle: mgr: don't record a crash dump for NotImplementedError from dispatch_remote ceph/ceph#71041, mgr/crash: fix race in do_post() ceph/ceph#71180, noting it should be reverted once a Tentacle point release ships both fixes.[2026-08-22]entry todocs/ceph-cluster-changelog.mddocumenting the change, why, risk assessment, rollback steps, and verification criteria.Risk Assessment
✅ Low: Single-line config flip (mgr module enabled: true -> false) with a well-documented rationale and upstream tracker links; scoped exactly to the stated intent, touches no other files, and is a reversible GitOps change that Flux applies declaratively.
Testing
All verification was static/manifest-level since this is a GitOps repo with no live cluster access in the test environment and the fix only takes effect on post-merge Flux reconciliation; I confirmed the diff is exactly the single intended file/field change with the required comment and no out-of-scope edits, that it survives the repo's actual Kustomize build path cleanly, and by pulling the real pinned upstream Rook Helm chart and templating it with the actual HelmRelease values at both commits, that the only difference in the entire rendered CephCluster custom resource (1849 lines) is
spec.mgr.modules[].enabledfor therookmodule flipping true→false - i.e. exactly the change the Rook operator will apply to the live cluster once Flux reconciles. Live-cluster evidence (ceph -s HEALTH_OK, crash rate flattening) is inherently unavailable pre-merge and is called out in the intent itself as a post-merge observation.Evidence: Base-vs-target rendered CephCluster CR diff (real upstream chart, real HelmRelease values)
Evidence: Full rendered manifest set at target commit (helm template output, 1849 lines, from the real oci://ghcr.io/rook/rook-ceph-cluster:v1.20.4 chart)
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
git diff --stat a161d503..2014395e (confirms single file, 7 insertions/1 deletion)git diff scoped to kubernetes/apps/system-upgrade/tuppr/upgrades/{kubernetesupgrade,talosupgrade}.yaml and **/prometheusrules.yaml between base and target (empty, confirms out-of-scope files untouched)yq '.spec.values.cephClusterSpec.mgr.modules' on the target helmrelease.yaml (confirms rook: enabled: false with the required comment/issue links present)kustomize build kubernetes/apps/rook-ceph/rook-ceph/cluster --load-restrictor LoadRestrictionsNone (the same manifest-build step task flux:test:all/CI exercises; renders cleanly, and yq on the output confirms the rendered HelmRelease carries enabled: false through)helm pull oci://ghcr.io/rook/rook-ceph-cluster --version v1.20.4 (the exact chart version pinned in ocirepository.yaml) thenhelm templateit against the realspec.valuesblock extracted from the HelmRelease at both base and target commits, diffing the two full rendered manifest sets✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.