Too many watch update events for endpoints object #41635

Closed
shashidharatd opened this Issue Feb 17, 2017 · 3 comments

Projects

None yet

2 participants

@shashidharatd
Member

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version): HEAD (latest)

Environment: dind

What happened:
Enpoints object is constantly getting updated every 2 seconds, even though nothing has changed in the object. It will produce lot of traffic on any informer watching on endpoints. An update event is produced every 2 second per endpoint.

What you expected to happen:
Endpoints(any object) should be updated only when there is change in either spec or status.

How to reproduce it (as minimally and precisely as possible):
Just watch on endpoints

# kubectl --namespace=kube-system get ep -w |adddate
Fri Feb 17 17:11:39 IST 2017 NAME                         ENDPOINTS                     AGE
Fri Feb 17 17:11:39 IST 2017 kube-controller-manager      <none>                        14m
Fri Feb 17 17:11:39 IST 2017 kube-dns                     172.17.1.1:53,172.17.1.1:53   19m
Fri Feb 17 17:11:39 IST 2017 kube-scheduler               <none>                        14m
Fri Feb 17 17:11:39 IST 2017 kubernetes-dashboard         172.17.1.0:9090               19m
Fri Feb 17 17:11:40 IST 2017 NAME             ENDPOINTS   AGE
Fri Feb 17 17:11:40 IST 2017 kube-scheduler   <none>      14m
Fri Feb 17 17:11:40 IST 2017 kube-controller-manager   <none>    14m
Fri Feb 17 17:11:42 IST 2017 kube-scheduler   <none>    14m
Fri Feb 17 17:11:42 IST 2017 kube-controller-manager   <none>    14m
Fri Feb 17 17:11:44 IST 2017 kube-scheduler   <none>    14m
Fri Feb 17 17:11:44 IST 2017 kube-controller-manager   <none>    14m
Fri Feb 17 17:11:46 IST 2017 kube-scheduler   <none>    15m
Fri Feb 17 17:11:46 IST 2017 kube-controller-manager   <none>    15m
Fri Feb 17 17:11:46 IST 2017 kube-dns   172.17.1.1:53,172.17.1.1:53   20m
Fri Feb 17 17:11:48 IST 2017 kube-scheduler   <none>    15m
Fri Feb 17 17:11:48 IST 2017 kube-controller-manager   <none>    15m
Fri Feb 17 17:11:50 IST 2017 kube-scheduler   <none>    15m
Fri Feb 17 17:11:50 IST 2017 kube-controller-manager   <none>    15m
Fri Feb 17 17:11:52 IST 2017 kube-scheduler   <none>    15m
Fri Feb 17 17:11:52 IST 2017 kube-controller-manager   <none>    15m

Also see the logs from federation controller manager, which is watching on endpoints updates from federated cluster

I0217 11:25:52.628669       1 servicecontroller.go:294] Delivering endpoint notification from federated cluster federation-dind :&Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-scheduler,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,UID:444ca9fe-f503-11e6-a7ee-0242ac110004,ResourceVersion:557,Generation:0,CreationTimestamp:2017-02-17 11:21:46 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"scheduler","leaseDurationSeconds":15,"acquireTime":"2017-02-17T11:21:45Z","renewTime":"2017-02-17T11:25:52Z","leaderTransitions":0},},OwnerReferences:[],Finalizers:[],ClusterName:,},Subsets:[],}
I0217 11:25:52.642019       1 servicecontroller.go:294] Delivering endpoint notification from federated cluster federation-dind :&Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-controller-manager,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,UID:444ca7a0-f503-11e6-a7ee-0242ac110004,ResourceVersion:558,Generation:0,CreationTimestamp:2017-02-17 11:21:46 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"controller","leaseDurationSeconds":15,"acquireTime":"2017-02-17T11:21:45Z","renewTime":"2017-02-17T11:25:52Z","leaderTransitions":0},},OwnerReferences:[],Finalizers:[],ClusterName:,},Subsets:[],}
I0217 11:25:54.630217       1 servicecontroller.go:294] Delivering endpoint notification from federated cluster federation-dind :&Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-scheduler,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,UID:444ca9fe-f503-11e6-a7ee-0242ac110004,ResourceVersion:559,Generation:0,CreationTimestamp:2017-02-17 11:21:46 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"scheduler","leaseDurationSeconds":15,"acquireTime":"2017-02-17T11:21:45Z","renewTime":"2017-02-17T11:25:54Z","leaderTransitions":0},},OwnerReferences:[],Finalizers:[],ClusterName:,},Subsets:[],}
I0217 11:25:54.645491       1 servicecontroller.go:294] Delivering endpoint notification from federated cluster federation-dind :&Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-controller-manager,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,UID:444ca7a0-f503-11e6-a7ee-0242ac110004,ResourceVersion:560,Generation:0,CreationTimestamp:2017-02-17 11:21:46 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"controller","leaseDurationSeconds":15,"acquireTime":"2017-02-17T11:21:45Z","renewTime":"2017-02-17T11:25:54Z","leaderTransitions":0},},OwnerReferences:[],Finalizers:[],ClusterName:,},Subsets:[],}
I0217 11:25:56.638414       1 servicecontroller.go:294] Delivering endpoint notification from federated cluster federation-dind :&Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-scheduler,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-scheduler,UID:444ca9fe-f503-11e6-a7ee-0242ac110004,ResourceVersion:561,Generation:0,CreationTimestamp:2017-02-17 11:21:46 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"scheduler","leaseDurationSeconds":15,"acquireTime":"2017-02-17T11:21:45Z","renewTime":"2017-02-17T11:25:56Z","leaderTransitions":0},},OwnerReferences:[],Finalizers:[],ClusterName:,},Subsets:[],}
I0217 11:25:56.649554       1 servicecontroller.go:294] Delivering endpoint notification from federated cluster federation-dind :&Endpoints{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:kube-controller-manager,GenerateName:,Namespace:kube-system,SelfLink:/api/v1/namespaces/kube-system/endpoints/kube-controller-manager,UID:444ca7a0-f503-11e6-a7ee-0242ac110004,ResourceVersion:562,Generation:0,CreationTimestamp:2017-02-17 11:21:46 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{control-plane.alpha.kubernetes.io/leader: {"holderIdentity":"controller","leaseDurationSeconds":15,"acquireTime":"2017-02-17T11:21:45Z","renewTime":"2017-02-17T11:25:56Z","leaderTransitions":0},},OwnerReferences:[],Finalizers:[],ClusterName:,},Subsets:[],}

Anything else we need to know:
Looking at the endpoints controller code https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/endpoint/endpoints_controller.go#L433
. it looks like reflect.DeepEqual is not working for endpoints labels and Subsets (may be need sorting, just a guess).

I also see another issue raised by @justinsb some time back #37151, related but slightly different.
I think this is a bug and if fixed will reduce lot of traffic to etcd.

@liggitt
Member
liggitt commented Feb 18, 2017

Those two endpoints specifically are used as shared lock objects by controllers. Agree that's a bad object to use because so many things watch them for updating iptables, etc

@liggitt
Member
liggitt commented Feb 18, 2017

dupe of #34627

@liggitt liggitt closed this Feb 18, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment