Skip to content

fix: fix nil map panic in CompareConfigmapAndTriggerRestart when dcr.Annotations is nil#493

Merged
morningman merged 1 commit into
apache:masterfrom
JinVei:master
Jun 22, 2026
Merged

fix: fix nil map panic in CompareConfigmapAndTriggerRestart when dcr.Annotations is nil#493
morningman merged 1 commit into
apache:masterfrom
JinVei:master

Conversation

@JinVei

@JinVei JinVei commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:

When CompareConfigmapAndTriggerRestart detects a configmap change and triggers
a rolling restart, it assigns directly to dcr.Annotations without checking
whether the map is initialized. If a DorisCluster CR is created without any
annotations, dcr.Annotations is nil, causing a panic that crashes the reconcile
loop.

Additionally, a copy-paste bug caused the log line to print dcr.Namespace twice
instead of dcr.Name for the CRD name field.

Observed error:

I0609 06:16:32.410860       1 doriscluster_controller.go:119] DorisClusterReconciler reconcile the update crd name adb namespace doris
I0609 06:16:32.410963       1 controller.go:221] Doris cluster is not have cn
I0609 06:16:32.411228       1 sub_controller.go:719] CompareConfigmapAndTriggerRestart TriggerRestart fe for CRD doris , namespace: doris
2026-06-09T06:16:32Z	ERROR	Observed a panic	{"controller": "doriscluster", "controllerGroup": "doris.selectdb.com", "controllerKind": "DorisCluster", "DorisCluster": {"name":"adb","namespace":"doris"}, "namespace": "doris", "name": "adb", "reconcileID": "5803f29d-1f4c-4a83-8dd7-46b31b54bb31", "panic": "assignment to entry in nil map", "panicGoValue": "\"assignment to entry in nil map\"", "stacktrace": "goroutine 234 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x233a1d8, 0xc0032e8cf0}, {0x1d2a8a0, 0x2310bb0})\n\t/go/pkg/mod/k8s.io/apimachinery@v0.32.0/pkg/util/runtime/runtime.go:107 +0xbc\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile.func1()\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:107 +0x112\npanic({0x1d2a8a0?, 0x2310bb0?})\n\t/usr/local/go/src/runtime/panic.go:791 +0x132\ngithub.com/apache/doris-operator/pkg/controller/sub_controller.(*SubDefaultController).CompareConfigmapAndTriggerRestart(0x0?, 0xc0027aeea0, {{0xc0014c9ff0, 0xe}, {0xc001b8e780, 0x40}, {0x0, 0x0, 0x0}, {0x0, ...}, ...}, ...)\n\t/workspace/pkg/controller/sub_controller/sub_controller.go:720 +0x285\ngithub.com/apache/doris-operator/pkg/controller/sub_controller/fe.(*Controller).Sync(0xc00028cba0, {0x233a1d8, 0xc0032e8cf0}, 0xc0027aeea0)\n\t/workspace/pkg/controller/sub_controller/fe/controller.go:93 +0x16c\ngithub.com/apache/doris-operator/pkg/controller.(*DorisClusterReconciler).Reconcile(0xc00006b580, {0x233a1d8, 0xc0032e8cf0}, {{{0xc000920c83, 0x5}, {0xc000920c80, 0x3}}})\n\t/workspace/pkg/controller/doriscluster_controller.go:149 +0x762\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile(0xc0032e8c60?, {0x233a1d8?, 0xc0032e8cf0?}, {{{0xc000920c83?, 0x0?}, {0xc000920c80?, 0x0?}}})\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:118 +0xbf\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler(0x2352ea0, {0x233a210, 0xc00068dc70}, {{{0xc000920c83, 0x5}, {0xc000920c80, 0x3}}})\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:319 +0x3a5\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem(0x2352ea0, {0x233a210, 0xc00068dc70})\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:279 +0x20e\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2()\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:240 +0x85\ncreated by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2 in goroutine 130\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:236 +0x538\n"}
github.com/apache/doris-operator/pkg/controller/sub_controller.(*SubDefaultController).CompareConfigmapAndTriggerRestart
	/workspace/pkg/controller/sub_controller/sub_controller.go:720
github.com/apache/doris-operator/pkg/controller/sub_controller/fe.(*Controller).Sync
	/workspace/pkg/controller/sub_controller/fe/controller.go:93
github.com/apache/doris-operator/pkg/controller.(*DorisClusterReconciler).Reconcile
	/workspace/pkg/controller/doriscluster_controller.go:149
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:319
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:279
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:240
2026-06-09T06:16:32Z	ERROR	Reconciler error	{"controller": "doriscluster", "controllerGroup": "doris.selectdb.com", "controllerKind": "DorisCluster", "DorisCluster": {"name":"adb","namespace":"doris"}, "namespace": "doris", "name": "adb", "reconcileID": "5803f29d-1f4c-4a83-8dd7-46b31b54bb31", "error": "panic: assignment to entry in nil map [recovered]"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:332
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:279
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.0/pkg/internal/controller/controller.go:240

Fix:

  • Initialize dcr.Annotations before assignment when it is nil.
  • Fix log format arg: use dcr.Name instead of dcr.Namespace for the CRD name field.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Al-assad

Copy link
Copy Markdown
Member

Thanks for the fix. The nil map guard looks right to me, and the log arg correction also makes sense.

I had one question about the restart annotation update path. In the current reconcile flow, the object passed into CompareConfigmapAndTriggerRestart seems to be a DeepCopy of the fetched CR:

https://github.com/apache/doris-operator/blob/master/pkg/controller/doriscluster_controller.go#L131
dcr := edcr.DeepCopy()

Then CompareConfigmapAndTriggerRestart updates the annotation on that copied object:

https://github.com/apache/doris-operator/blob/master/pkg/controller/sub_controller/sub_controller.go#L758
dcr.Annotations[dorisv1.GetRestartAnnotationKey(componentType)] = time.Now().Format(time.RFC3339)

After Sync, I only see ApplyDorisCluster being called from revertDorisClusterSomeFields when FE replicas differ, while the normal path only updates status:

https://github.com/apache/doris-operator/blob/master/pkg/controller/doriscluster_controller.go#L177-L180
https://github.com/apache/doris-operator/blob/master/pkg/controller/doriscluster_controller.go#L212-L214

So I may be missing something, but could you help confirm where this annotation change gets persisted back to the apiserver? If there isn't an existing path, maybe it would be helpful to add a small test covering the configmap-change restart flow.

@morningman morningman merged commit ded74d2 into apache:master Jun 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants