Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kube-apiserver panic caused by etcd-cient trying to close a closed channel #87135

Open
libnux opened this issue Jan 13, 2020 · 4 comments
Open

kube-apiserver panic caused by etcd-cient trying to close a closed channel #87135

libnux opened this issue Jan 13, 2020 · 4 comments
Assignees

Comments

@libnux
Copy link
Contributor

@libnux libnux commented Jan 13, 2020

What happened:
kube-apiserver crashed due to etcd-cient trying to close a closed channel
What you expected to happen:
kube-apiserver keep running instead of crashing
How to reproduce it (as minimally and precisely as possible):
Not easy to reproduce
Anything else we need to know?:

apiserver logs

I0109 13:30:21.445237       1 controller.go:116] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Requeue.
E0109 13:30:24.522260       1 compact.go:124] etcd: endpoint ([https://etcd-03.kube-system.svc.cluster.local:2379]) compact failed: etcdserver: permission denied
I0109 13:30:37.238973       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.custom.metrics.k8s.io
E0109 13:30:37.251397       1 controller.go:111] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0109 13:30:37.251413       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.custom.metrics.k8s.io: Rate Limited Requeue.
I0109 13:31:20.424527       1 controller.go:105] OpenAPI AggregationController: Processing item v1alpha1.incidents.monitoring.appscode.com
I0109 13:31:20.442875       1 controller.go:116] OpenAPI AggregationController: action for item v1alpha1.incidents.monitoring.appscode.com: Requeue.
I0109 13:31:20.442905       1 controller.go:105] OpenAPI AggregationController: Processing item v1alpha1.admission.monitoring.appscode.com
I0109 13:31:20.443540       1 controller.go:116] OpenAPI AggregationController: action for item v1alpha1.admission.monitoring.appscode.com: Requeue.
I0109 13:31:21.503763       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0109 13:31:21.505904       1 controller.go:116] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Requeue.
I0109 13:31:37.251555       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.custom.metrics.k8s.io
E0109 13:31:37.260175       1 controller.go:111] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: OpenAPI spec does not exists
I0109 13:31:37.260189       1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.custom.metrics.k8s.io: Rate Limited Requeue.
I0109 13:32:20.443030       1 controller.go:105] OpenAPI AggregationController: Processing item v1alpha1.incidents.monitoring.appscode.com
I0109 13:32:20.457138       1 controller.go:116] OpenAPI AggregationController: action for item v1alpha1.incidents.monitoring.appscode.com: Requeue.
I0109 13:32:20.457161       1 controller.go:105] OpenAPI AggregationController: Processing item v1alpha1.admission.monitoring.appscode.com
I0109 13:32:20.503895       1 controller.go:116] OpenAPI AggregationController: action for item v1alpha1.admission.monitoring.appscode.com: Requeue.
I0109 13:32:21.506050       1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I0109 13:32:21.508304       1 controller.go:116] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Requeue.
panic: close of closed channel

goroutine 25472782 [running]:
k8s.io/kubernetes/vendor/github.com/coreos/etcd/clientv3.(*watchGrpcStream).run(0xc43fcac000)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/clientv3/watch.go:477 +0x8a0
created by k8s.io/kubernetes/vendor/github.com/coreos/etcd/clientv3.(*watcher).newWatcherGrpcStream
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/clientv3/watch.go:236 +0x372

Environment:

  • Kubernetes version (use kubectl version): k8s v1.10.5, etcd client v3.2.13(commit: 95a726a27e09030f9ccbd9982a1508f5a6d25ada)
  • Cloud provider or hardware configuration:
  • OS (e.g: cat /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Network plugin and version (if this is a network-related bug):
  • Others:
@libnux

This comment has been minimized.

Copy link
Contributor Author

@libnux libnux commented Jan 13, 2020

/sig api-machinery

@libnux

This comment has been minimized.

Copy link
Contributor Author

@libnux libnux commented Jan 13, 2020

Is this problem related to or fixed by this PR of etcd? etcd-io/etcd#8471

@liggitt

This comment has been minimized.

Copy link
Member

@liggitt liggitt commented Jan 13, 2020

cc @jpbetz

@fedebongio

This comment has been minimized.

Copy link
Contributor

@fedebongio fedebongio commented Jan 14, 2020

/assign @wenjiaswe

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.