Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Operator crashing when trying to assign Annotation to the deployment spec #147

Closed
gidesh opened this issue Jan 18, 2021 · 2 comments
Closed

Operator crashing when trying to assign Annotation to the deployment spec #147

gidesh opened this issue Jan 18, 2021 · 2 comments

Comments

@gidesh
Copy link
Contributor

gidesh commented Jan 18, 2021 •
edited

First of all great work.

We are using the operator to create VM components. We have seen the operator crashing when trying to reconcile/create vminsert. Please see the logs

vm-operator-dfb9d694c-v8tpf manager E0118 11:08:05.271870       1 runtime.go:78] Observed a panic: "assignment to entry in nil map" (assignment to entry in nil map)
vm-operator-dfb9d694c-v8tpf manager goroutine 1206 [running]:
vm-operator-dfb9d694c-v8tpf manager k8s.io/apimachinery/pkg/util/runtime.logPanic(0x15c0300, 0x19252f0)
vm-operator-dfb9d694c-v8tpf manager 	k8s.io/apimachinery@v0.18.6/pkg/util/runtime/runtime.go:74 +0xa6
vm-operator-dfb9d694c-v8tpf manager k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
vm-operator-dfb9d694c-v8tpf manager 	k8s.io/apimachinery@v0.18.6/pkg/util/runtime/runtime.go:48 +0x89
vm-operator-dfb9d694c-v8tpf manager panic(0x15c0300, 0x19252f0)
vm-operator-dfb9d694c-v8tpf manager 	runtime/panic.go:969 +0x1b9
vm-operator-dfb9d694c-v8tpf manager github.com/VictoriaMetrics/operator/controllers/factory.createOrUpdateVMInsert(0x1970ca0, 0xc000040040, 0xc0000d6540, 0x1980d00, 0xc0006b18f0, 0xc0003ba000, 0xc001b227e8, 0x14, 0xc001b22800)
vm-operator-dfb9d694c-v8tpf manager 	github.com/VictoriaMetrics/operator/controllers/factory/vmcluster.go:298 +0x437
vm-operator-dfb9d694c-v8tpf manager github.com/VictoriaMetrics/operator/controllers/factory.CreateOrUpdateVMCluster(0x1970ca0, 0xc000040040, 0xc0000d6540, 0x1980d00, 0xc0006b18f0, 0xc0003ba000, 0x0, 0x0, 0x0, 0x0)
vm-operator-dfb9d694c-v8tpf manager 	github.com/VictoriaMetrics/operator/controllers/factory/vmcluster.go:156 +0x38a
vm-operator-dfb9d694c-v8tpf manager github.com/VictoriaMetrics/operator/controllers.(*VMClusterReconciler).Reconcile(0xc000547710, 0xc00056a780, 0x10, 0xc00056a74c, 0x4, 0xc000b96da0, 0xc000c9eab0, 0xc0009d8cf8, 0xc0009d8cf0)
vm-operator-dfb9d694c-v8tpf manager 	github.com/VictoriaMetrics/operator/controllers/vmcluster_controller.go:50 +0x2a5
vm-operator-dfb9d694c-v8tpf manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00058fa70, 0x1615780, 0xc0004c0e40, 0x0)
vm-operator-dfb9d694c-v8tpf manager 	sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:235 +0x2a9
vm-operator-dfb9d694c-v8tpf manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00058fa70, 0x203000)
vm-operator-dfb9d694c-v8tpf manager 	sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:209 +0xb0
vm-operator-dfb9d694c-v8tpf manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc00058fa70)
vm-operator-dfb9d694c-v8tpf manager 	sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:188 +0x2b
vm-operator-dfb9d694c-v8tpf manager k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc001883110)
vm-operator-dfb9d694c-v8tpf manager 	k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155 +0x5f
vm-operator-dfb9d694c-v8tpf manager k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc001883110, 0x193c360, 0xc0006e4210, 0x1, 0xc0005a7860)
vm-operator-dfb9d694c-v8tpf manager 	k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156 +0xad
vm-operator-dfb9d694c-v8tpf manager k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc001883110, 0x3b9aca00, 0x0, 0x1, 0xc0005a7860)
vm-operator-dfb9d694c-v8tpf manager 	k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133 +0x98
vm-operator-dfb9d694c-v8tpf manager k8s.io/apimachinery/pkg/util/wait.Until(0xc001883110, 0x3b9aca00, 0xc0005a7860)
vm-operator-dfb9d694c-v8tpf manager 	k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90 +0x4d
vm-operator-dfb9d694c-v8tpf manager created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
vm-operator-dfb9d694c-v8tpf manager 	sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:170 +0x3fa
vm-operator-dfb9d694c-v8tpf manager panic: assignment to entry in nil map [recovered]
vm-operator-dfb9d694c-v8tpf manager 	panic: assignment to entry in nil map
vm-operator-dfb9d694c-v8tpf manager 
vm-operator-dfb9d694c-v8tpf manager goroutine 1206 [running]:
vm-operator-dfb9d694c-v8tpf manager k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
vm-operator-dfb9d694c-v8tpf manager 	k8s.io/apimachinery@v0.18.6/pkg/util/runtime/runtime.go:55 +0x10c
vm-operator-dfb9d694c-v8tpf manager panic(0x15c0300, 0x19252f0)
vm-operator-dfb9d694c-v8tpf manager 	runtime/panic.go:969 +0x1b9
vm-operator-dfb9d694c-v8tpf manager github.com/VictoriaMetrics/operator/controllers/factory.createOrUpdateVMInsert(0x1970ca0, 0xc000040040, 0xc0000d6540, 0x1980d00, 0xc0006b18f0, 0xc0003ba000, 0xc001b227e8, 0x14, 0xc001b22800)
vm-operator-dfb9d694c-v8tpf manager 	github.com/VictoriaMetrics/operator/controllers/factory/vmcluster.go:298 +0x437
vm-operator-dfb9d694c-v8tpf manager github.com/VictoriaMetrics/operator/controllers/factory.CreateOrUpdateVMCluster(0x1970ca0, 0xc000040040, 0xc0000d6540, 0x1980d00, 0xc0006b18f0, 0xc0003ba000, 0x0, 0x0, 0x0, 0x0)
vm-operator-dfb9d694c-v8tpf manager 	github.com/VictoriaMetrics/operator/controllers/factory/vmcluster.go:156 +0x38a
vm-operator-dfb9d694c-v8tpf manager github.com/VictoriaMetrics/operator/controllers.(*VMClusterReconciler).Reconcile(0xc000547710, 0xc00056a780, 0x10, 0xc00056a74c, 0x4, 0xc000b96da0, 0xc000c9eab0, 0xc0009d8cf8, 0xc0009d8cf0)
vm-operator-dfb9d694c-v8tpf manager 	github.com/VictoriaMetrics/operator/controllers/vmcluster_controller.go:50 +0x2a5
vm-operator-dfb9d694c-v8tpf manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00058fa70, 0x1615780, 0xc0004c0e40, 0x0)
vm-operator-dfb9d694c-v8tpf manager 	sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:235 +0x2a9
vm-operator-dfb9d694c-v8tpf manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00058fa70, 0x203000)
vm-operator-dfb9d694c-v8tpf manager 	sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:209 +0xb0
vm-operator-dfb9d694c-v8tpf manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(0xc00058fa70)
vm-operator-dfb9d694c-v8tpf manager 	sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:188 +0x2b
vm-operator-dfb9d694c-v8tpf manager k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc001883110)
vm-operator-dfb9d694c-v8tpf manager 	k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155 +0x5f
vm-operator-dfb9d694c-v8tpf manager k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc001883110, 0x193c360, 0xc0006e4210, 0x1, 0xc0005a7860)
vm-operator-dfb9d694c-v8tpf manager 	k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156 +0xad
vm-operator-dfb9d694c-v8tpf manager k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc001883110, 0x3b9aca00, 0x0, 0x1, 0xc0005a7860)
vm-operator-dfb9d694c-v8tpf manager 	k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133 +0x98
vm-operator-dfb9d694c-v8tpf manager k8s.io/apimachinery/pkg/util/wait.Until(0xc001883110, 0x3b9aca00, 0xc0005a7860)
vm-operator-dfb9d694c-v8tpf manager 	k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90 +0x4d
vm-operator-dfb9d694c-v8tpf manager created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
vm-operator-dfb9d694c-v8tpf manager 	sigs.k8s.io/controller-runtime@v0.6.2/pkg/internal/controller/controller.go:170 +0x3fa

We have a mutating webhook which adds an annotation to deployment spec template when a deployment is applied. When the webhook adds the annotation the vminsert deployment created by the operator, the operator crashes when trying to reconcile because of line 298

newDeployment.Spec.Template.Annotations[annotation] = value

A quick fix was to check if Annotation is nil and then initialize a map

if newDeployment.Spec.Template.Annotations == nil {
      newDeployment.Spec.Template.Annotations = make(map[string]string)
}
for annotation, value := range currentDeployment.Spec.Template.Annotations {
      newDeployment.Spec.Template.Annotations[annotation] = value
}

I have created PR , adding the if condition to check the Annotation is nil at all the required placed - #148
@gidesh gidesh mentioned this issue Jan 18, 2021
Merged
@gidesh
Copy link
Contributor Author

gidesh commented Jan 18, 2021

Thanks for merging the PR, I will close the issue

@gidesh gidesh closed this as completed Jan 18, 2021
@f41gh7
Copy link
Collaborator

f41gh7 commented Jan 20, 2021 •
edited

https://github.com/VictoriaMetrics/operator/releases/tag/v0.7.3 was released with fix, feel free to reopen if it doesnt help.

@renovate renovate bot mentioned this issue Feb 15, 2022
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@f41gh7 @gidesh