Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Too many watch update events for endpoints object #41635

Closed
shashidharatd opened this issue Feb 17, 2017 · 3 comments
Closed

Too many watch update events for endpoints object #41635

shashidharatd opened this issue Feb 17, 2017 · 3 comments

Comments

@shashidharatd
Copy link

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version): HEAD (latest)

Environment: dind

What happened:
Enpoints object is constantly getting updated every 2 seconds, even though nothing has changed in the object. It will produce lot of traffic on any informer watching on endpoints. An update event is produced every 2 second per endpoint.

What you expected to happen:
Endpoints(any object) should be updated only when there is change in either spec or status.

How to reproduce it (as minimally and precisely as possible):
Just watch on endpoints

# kubectl --namespace=kube-system get ep -w |adddate
Fri Feb 17 17:11:39 IST 2017 NAME                         ENDPOINTS                     AGE
Fri Feb 17 17:11:39 IST 2017 kube-controller-manager      <none>                        14m
Fri Feb 17 17:11:39 IST 2017 kube-dns                     172.17.1.1:53,172.17.1.1:53   19m
Fri Feb 17 17:11:39 IST 2017 kube-scheduler               <none>                        14m
Fri Feb 17 17:11:39 IST 2017 kubernetes-dashboard         172.17.1.0:9090               19m
Fri Feb 17 17:11:40 IST 2017 NAME             ENDPOINTS   AGE
Fri Feb 17 17:11:40 IST 2017 kube-scheduler   <none>      14m
Fri Feb 17 17:11:40 IST 2017 kube-controller-manager   <none>    14m
Fri Feb 17 17:11:42 IST 2017 kube-scheduler   <none>    14m
Fri Feb 17 17:11:42 IST 2017 kube-controller-manager   <none>    14m
Fri Feb 17 17:11:44 IST 2017 kube-scheduler   <none>    14m
Fri Feb 17 17:11:44 IST 2017 kube-controller-manager   <none>    14m
Fri Feb 17 17:11:46 IST 2017 kube-scheduler   <none>    15m
Fri Feb 17 17:11:46 IST 2017 kube-controller-manager   <none>    15m
Fri Feb 17 17:11:46 IST 2017 kube-dns   172.17.1.1:53,172.17.1.1:53   20m
Fri Feb 17 17:11:48 IST 2017 kube-scheduler   <none>    15m
Fri Feb 17 17:11:48 IST 2017 kube-controller-manager   <none>    15m
Fri Feb 17 17:11:50 IST 2017 kube-scheduler   <none>    15m
Fri Feb 17 17:11:50 IST 2017 kube-controller-manager   <none>    15m
Fri Feb 17 17:11:52 IST 2017 kube-scheduler   <none>    15m
Fri Feb 17 17:11:52 IST 2017 kube-controller-manager   <none>    15m

Also see the logs from federation controller manager, which is watching on endpoints updates from federated cluster

I0217 11:25:52.628669       1 servicecontroller.go:294] Delivering endpoint notification from federated cluster federation-dind :&Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-scheduler,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,UID:444ca9fe-f503-11e6-a7ee-0242ac110004,ResourceVersion:557,Generation:0,CreationTimestamp:2017-02-17 11:21:46 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"scheduler","leaseDurationSeconds":15,"acquireTime":"2017-02-17T11:21:45Z","renewTime":"2017-02-17T11:25:52Z","leaderTransitions":0},},OwnerReferences:[],Finalizers:[],ClusterName:,},Subsets:[],}
I0217 11:25:52.642019       1 servicecontroller.go:294] Delivering endpoint notification from federated cluster federation-dind :&Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-controller-manager,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,UID:444ca7a0-f503-11e6-a7ee-0242ac110004,ResourceVersion:558,Generation:0,CreationTimestamp:2017-02-17 11:21:46 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"controller","leaseDurationSeconds":15,"acquireTime":"2017-02-17T11:21:45Z","renewTime":"2017-02-17T11:25:52Z","leaderTransitions":0},},OwnerReferences:[],Finalizers:[],ClusterName:,},Subsets:[],}
I0217 11:25:54.630217       1 servicecontroller.go:294] Delivering endpoint notification from federated cluster federation-dind :&Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-scheduler,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,UID:444ca9fe-f503-11e6-a7ee-0242ac110004,ResourceVersion:559,Generation:0,CreationTimestamp:2017-02-17 11:21:46 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"scheduler","leaseDurationSeconds":15,"acquireTime":"2017-02-17T11:21:45Z","renewTime":"2017-02-17T11:25:54Z","leaderTransitions":0},},OwnerReferences:[],Finalizers:[],ClusterName:,},Subsets:[],}
I0217 11:25:54.645491       1 servicecontroller.go:294] Delivering endpoint notification from federated cluster federation-dind :&Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-controller-manager,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,UID:444ca7a0-f503-11e6-a7ee-0242ac110004,ResourceVersion:560,Generation:0,CreationTimestamp:2017-02-17 11:21:46 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"controller","leaseDurationSeconds":15,"acquireTime":"2017-02-17T11:21:45Z","renewTime":"2017-02-17T11:25:54Z","leaderTransitions":0},},OwnerReferences:[],Finalizers:[],ClusterName:,},Subsets:[],}
I0217 11:25:56.638414       1 servicecontroller.go:294] Delivering endpoint notification from federated cluster federation-dind :&Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-scheduler,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,UID:444ca9fe-f503-11e6-a7ee-0242ac110004,ResourceVersion:561,Generation:0,CreationTimestamp:2017-02-17 11:21:46 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"scheduler","leaseDurationSeconds":15,"acquireTime":"2017-02-17T11:21:45Z","renewTime":"2017-02-17T11:25:56Z","leaderTransitions":0},},OwnerReferences:[],Finalizers:[],ClusterName:,},Subsets:[],}
I0217 11:25:56.649554       1 servicecontroller.go:294] Delivering endpoint notification from federated cluster federation-dind :&Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-controller-manager,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,UID:444ca7a0-f503-11e6-a7ee-0242ac110004,ResourceVersion:562,Generation:0,CreationTimestamp:2017-02-17 11:21:46 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"controller","leaseDurationSeconds":15,"acquireTime":"2017-02-17T11:21:45Z","renewTime":"2017-02-17T11:25:56Z","leaderTransitions":0},},OwnerReferences:[],Finalizers:[],ClusterName:,},Subsets:[],}

Anything else we need to know:
Looking at the endpoints controller code https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/endpoint/endpoints_controller.go#L433
. it looks like reflect.DeepEqual is not working for endpoints labels and Subsets (may be need sorting, just a guess).

I also see another issue raised by @justinsb some time back #37151, related but slightly different.
I think this is a bug and if fixed will reduce lot of traffic to etcd.
@liggitt
Copy link
Member

liggitt commented Feb 18, 2017

Those two endpoints specifically are used as shared lock objects by controllers. Agree that's a bad object to use because so many things watch them for updating iptables, etc

@liggitt
Copy link
Member

liggitt commented Feb 18, 2017

code in question:
https://github.com/kubernetes/kubernetes/blob/master/plugin/cmd/kube-scheduler/app/server.go#L89-L111

@liggitt
Copy link
Member

liggitt commented Feb 18, 2017

dupe of #34627

@liggitt liggitt closed this as completed Feb 18, 2017
concaf added a commit to concaf/teleproxy that referenced this issue Feb 26, 2019
@concaf
Ignore endpoint updates from kube-system
8f3274c 
kube-scheduler and kube-controller-manager endpoints are
updated almost every second, leading to terrible noise,
and hence constant listener invokation. So, here we
ignore endpoint updates from kube-system namespace. More:
kubernetes/kubernetes#41635
kubernetes/kubernetes#34627
@concaf concaf mentioned this issue Feb 26, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@liggitt @shashidharatd