Panic in reconciler when daemonsets.annotations contain value with colons

[sole0bserver]

Hi,

I want to set an ArgoCD tracking annotation for daemonsets that are spawn by the operator. The intention is to see all operator-related objects in ArgoCD UI.

To do so, I put the following to the Helm values:

daemonsets:
       annotations:
           argocd.argoproj.io/tracking-id: 'gpu-operator:/Application:gpu-operator/gpu-operator'

This breaks the operator with the following log entries:

{"level":"info","ts":1678704637.7298467,"msg":"Observed a panic in reconciler: assignment to entry in nil map","controller":"clusterpolicy-controller","object":{"name":"cluster-policy"},"namespace":"","name":"cluster-policy","reconcileID":"db939a16-9340-4f44-8c8c-5a09636915ed"}
panic: assignment to entry in nil map [recovered]
	panic: assignment to entry in nil map

goroutine 1095 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:118 +0x1f4
panic({0x1902300, 0x1df3cf0})
	/usr/local/go/src/runtime/panic.go:884 +0x212
github.com/NVIDIA/gpu-operator/controllers.applyCommonDaemonsetMetadata(...)
	/workspace/controllers/object_controls.go:589
github.com/NVIDIA/gpu-operator/controllers.preProcessDaemonSet(0xc003802480, {{0x1e0e8f8, 0xc002711950}, 0xc000851900, {0xc00004a053, 0xc}, {0xc000c2e000, 0x10, 0x10}, {0xc002713b00, ...}, ...})
	/workspace/controllers/object_controls.go:567 +0xab8
github.com/NVIDIA/gpu-operator/controllers.DaemonSet({{0x1e0e8f8, 0xc002711950}, 0xc000851900, {0xc00004a053, 0xc}, {0xc000c2e000, 0x10, 0x10}, {0xc002713b00, 0x10, ...}, ...})
	/workspace/controllers/object_controls.go:3099 +0x4a5
github.com/NVIDIA/gpu-operator/controllers.(*ClusterPolicyController).step(0x2b80c40)
	/workspace/controllers/state_manager.go:885 +0x136
github.com/NVIDIA/gpu-operator/controllers.(*ClusterPolicyReconciler).Reconcile(0xc000846090, {0x1e0e8f8, 0xc002711950}, {{{0x0, 0x0}, {0xc0007db5a0, 0xe}}})
	/workspace/controllers/clusterpolicy_controller.go:135 +0x4e5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1e0e850?, {0x1e0e8f8?, 0xc002711950?}, {{{0x0?, 0x1a78ee0?}, {0xc0007db5a0?, 0x4045d4?}}})
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:121 +0xc8
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0003780a0, {0x1e0e850, 0xc001268dc0}, {0x1982860?, 0xc0008a3140?})
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:320 +0x33c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0003780a0, {0x1e0e850, 0xc001268dc0})
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:273 +0x1d9
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:234 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/workspace/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:230 +0x333

I've tried to double quote and even triple quote the value without success.

[shivamerla]

@sole0bserver thanks for reporting this issue, we have a bug in handling custom annotations. Will fix this as part of v23.3.0 release.

[cdesiniotis]

@sole0bserver I am closing this issue as a fix was introduced in GPU Operator v23.3.0 for this. Please re-open if you are still facing issues.