New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kubelet 1.13.0 panic when run against 1.12.x kube-apiserver #71749

Open
nerzhul opened this Issue Dec 5, 2018 · 11 comments

Comments

Projects
None yet
4 participants
@nerzhul

nerzhul commented Dec 5, 2018

What happened:

Kubelet panics after registering node

What you expected to happen:

Kubelet should not panic

How to reproduce it (as minimally and precisely as possible):

Use kubelet on an ARM64 machine (i'm on Scaleway provider)

Anything else we need to know?:

I upgraded from 1.12.3 to 1.13.0

here is the statup log (only the end):

Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: I1205 14:17:18.151924    5523 reconciler.go:154] Reconciler: start to sync state
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: I1205 14:17:18.327832    5523 kubelet_node_status.go:114] Node scw-xxxxxx was previously registered
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: I1205 14:17:18.327924    5523 kubelet_node_status.go:75] Successfully registered node scw-xxxxxx
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: E1205 14:17:18.356785    5523 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:76
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /usr/local/go/src/runtime/asm_arm64.s:412
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /usr/local/go/src/runtime/panic.go:513
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /usr/local/go/src/runtime/panic.go:82
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /usr/local/go/src/runtime/signal_unix.go:390
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_pods.go:546
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet_pods.go:449
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container.go:192
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_container.go:115
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kuberuntime/kuberuntime_manager.go:741
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1689
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/kubelet.go:824
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:174
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:183
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /workspace/anago-v1.13.0-rc.2.1+ddf47ac13c1a94/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubelet/pod_workers.go:221
Dec 05 14:17:18 scw-xxxxxx kubelet[5523]: /usr/local/go/src/runtime/asm_arm64.s:1114

Environment:

  • Kubernetes version (use kubectl version): 1.13.0
  • Cloud provider or hardware configuration: Scaleway
  • OS (e.g. from /etc/os-release): Debian GNU/Linux 9 (stretch)
  • Kernel (e.g. uname -a): 4.15.11-mainline-rev1
  • Install tools:
  • Others:

/kind bug

@nerzhul

This comment has been minimized.

nerzhul commented Dec 5, 2018

/sig node

@k8s-ci-robot k8s-ci-robot added sig/node and removed needs-sig labels Dec 5, 2018

@liggitt liggitt added this to the v1.13 milestone Dec 6, 2018

@liggitt

This comment has been minimized.

Member

liggitt commented Dec 6, 2018

a few questions:

  • can you provide full output of kubectl version, including the server version?
  • did you upgrade kube-apiserver to 1.13.0?
  • are you running static pod manifests on the kubelet?
@nerzhul

This comment has been minimized.

nerzhul commented Dec 6, 2018

hello @liggitt

  • i had rollbacked this node cannot currently
  • no i didn't. I just upgraded on one server kube-proxy and kubelet. Kube-proxy works fine.
  • i don't have static pods, just daemonsets on this node (metrology)
@liggitt

This comment has been minimized.

Member

liggitt commented Dec 6, 2018

kubelet must not be newer than kube-apiserver

be sure to upgrade kube-apiserver to 1.13 before upgrading kubelet

@liggitt

This comment has been minimized.

Member

liggitt commented Dec 6, 2018

/close

@k8s-ci-robot

This comment has been minimized.

Contributor

k8s-ci-robot commented Dec 6, 2018

@liggitt: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@nerzhul

This comment has been minimized.

nerzhul commented Dec 6, 2018

i understand but why not just have a proper error instead of a ugly stack trace then ? it's more proper than an unhandled error. Here it's a memory error, a memory error is a grave error.

@liggitt liggitt changed the title from Kubelet 1.13.0 panic on ARM64 to kubelet 1.13.0 panic when run against 1.12.x kube-apiserver Dec 7, 2018

@liggitt

This comment has been minimized.

Member

liggitt commented Dec 7, 2018

/reopen
cc @kubernetes/sig-node-bugs for disposition

@k8s-ci-robot

This comment has been minimized.

Contributor

k8s-ci-robot commented Dec 7, 2018

@liggitt: Reopened this issue.

In response to this:

/reopen
cc @kubernetes/sig-node-bugs for disposition

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot reopened this Dec 7, 2018

@liggitt

This comment has been minimized.

Member

liggitt commented Dec 7, 2018

code in question is here:

serviceEnv, err := kl.getServiceEnvVarMap(pod.Namespace, *pod.Spec.EnableServiceLinks)

note that this is an unsupported configuration

@feiskyer

This comment has been minimized.

Member

feiskyer commented Dec 9, 2018

Refer the discussion here: #69061 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment