Observed a panic: "invalid memory address or nil pointer dereference"

[aleksey005]

After upgrade from 0.5.0 to 0.6.0

CrashLoopBackOff: Back-off 5m0s restarting failed container=longhorn-manager pod=longhorn-manager-78jk6_longhorn-system(99bf8884-b48f-48dd-a8cc-4b718fff5818)

time="2019-09-23T07:37:23Z" level=info msg="Start overwriting built-in settings with customized values"
E0923 07:37:23.694897 1 node_controller.go:502] Couldn't get node backup01 for replica pvc-35fd6640-8dfc-411d-97e6-5d38caf5643f-r-47d04266: node.longhorn.rancher.io "backup01" not found
time="2019-09-23T07:37:23Z" level=info msg="Start Longhorn node controller"
time="2019-09-23T07:37:23Z" level=info msg="Start Longhorn websocket controller"
time="2019-09-23T07:37:23Z" level=info msg="Start Longhorn Setting controller"
time="2019-09-23T07:37:23Z" level=info msg="Start kubernetes controller"
time="2019-09-23T07:37:23Z" level=debug msg="Waiting for engine image longhornio/longhorn-engine:v0.6.0 to be ready"
time="2019-09-23T07:37:23Z" level=info msg="Starting Longhorn instance manager controller"
time="2019-09-23T07:37:23Z" level=info msg="Start Longhorn engine controller"
time="2019-09-23T07:37:23Z" level=info msg="Start Longhorn volume controller"
time="2019-09-23T07:37:23Z" level=info msg="Start Longhorn Engine Image controller"
time="2019-09-23T07:37:23Z" level=info msg="Start Longhorn replica controller"
E0923 07:37:23.938105 1 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:522
/usr/local/go/src/runtime/panic.go:513
/usr/local/go/src/runtime/panic.go:82
/usr/local/go/src/runtime/signal_unix.go:390
/go/src/github.com/longhorn/longhorn-manager/controller/kubernetes_controller.go:227
/go/src/github.com/longhorn/longhorn-manager/controller/kubernetes_controller.go:179
/go/src/github.com/longhorn/longhorn-manager/controller/kubernetes_controller.go:167
/go/src/github.com/longhorn/longhorn-manager/controller/kubernetes_controller.go:160
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:1333
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x127461b]
goroutine 363 [running]:
github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x108
panic(0x1464940, 0x25c1a80)
/usr/local/go/src/runtime/panic.go:513 +0x1b9
github.com/longhorn/longhorn-manager/controller.(*KubernetesController).syncKubernetesStatus(0xc00017a1a0, 0xc0004b07a8, 0x8, 0x0, 0x0)
/go/src/github.com/longhorn/longhorn-manager/controller/kubernetes_controller.go:227 +0x18b
github.com/longhorn/longhorn-manager/controller.(*KubernetesController).processNextWorkItem(0xc00017a1a0, 0xc000157900)
/go/src/github.com/longhorn/longhorn-manager/controller/kubernetes_controller.go:179 +0xdf
github.com/longhorn/longhorn-manager/controller.(*KubernetesController).worker(0xc00017a1a0)
/go/src/github.com/longhorn/longhorn-manager/controller/kubernetes_controller.go:167 +0x2b
github.com/longhorn/longhorn-manager/controller.(*KubernetesController).worker-fm()
/go/src/github.com/longhorn/longhorn-manager/controller/kubernetes_controller.go:160 +0x2a
github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc000b88b70)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000b88b70, 0x3b9aca00, 0x0, 0x1, 0xc000086a20)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbe
github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc000b88b70, 0x3b9aca00, 0xc000086a20)
/go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by github.com/longhorn/longhorn-manager/controller.(*KubernetesController).Run
/go/src/github.com/longhorn/longhorn-manager/controller/kubernetes_controller.go:160 +0x1ce

[yasker]

To reproduce the issue, create the following yaml file and apply using kubectl create -f

apiVersion: v1
kind: PersistentVolume
metadata:
  name: test-pv-volume
spec:
  storageClassName: manual
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/tmp/123"

All the managers would crash at this point.

[courcelm]

Also got this one!

[yasker]

@courcelm OK, sounds like this is the culprit!

We will release v0.6.1 today to fix this.

[yasker]

Fixed. Longhorn v0.6.1 is coming soon.

[fredleger]

Still getting this one with Longhorn v1.0.2 installed through apps in Rancher v2.4.8 in a digital ocean managed kubernetes offer (latest 1.18 release).
Had to downgrade to v1.0.0 to make it work

[yasker]

@fredleger This issue should have been fixed long ago. Can you create a new issue and post the crash log?