-
Notifications
You must be signed in to change notification settings - Fork 39.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support skip watch Headless Service in server-side. #122394
Comments
/sig network api-machinery |
how many headless Services does that kubeflow task creates? |
/sig scalability |
Under steady-state conditions, the cluster maintains 30,000 headless services. During daily new tasks, an additional 10,000 headless services will be created. The number of nodes in the cluster is around 7,000 |
@aojea kube-apiserver needs to support field-selector with clusterIP firstly, and then kubelet/kube-proxy can be updated. Otherwise, clients performing List-watch operations will receive errors like this, which is not suitable for updating on existing clusters. |
@aojea If adding a field-selector for clusterIP is considered to be ok. I think we can add IndexFields for service spec.clusterIP as well (such as node metadata.name),it can reduce events-count converted from kube-apiserver common channel to every cacheWatcher,reduce cpu-usage for kube-apiserver. |
That is on sig-scalability area and better @wojtek-t to decide in that, I can see how Headless services are more used lately in Kubernetes , special fo AI/ML workloads, so I think is a fair request to implement this field selector and allow to filter server side headless service, but I do not know the tradeoffs of using IndexFields |
+1 I would be ok with adding IndexFields for headless services too [re tradeoff, it adds a bit of memory usage, but it is fairly low - there was a similar thing happening recently in https://github.com//issues/120778 that has a lot of good analysis done] |
I can see that IndexFields implementation is feature gated, I think we can iterate here and go with the fieldselector first and evaluate later the IndexFields addition |
The documentation currently says that headless services are labeled. Is this not accurate? https://kubernetes.io/docs/reference/labels-annotations-taints/#servicekubernetesioheadless |
The control plane adds this label to an Endpoints object when the owning Service is headless. |
/triage accepted |
/reopen We had to revert the fix https://github.com/kubernetes/kubernetes/pull/123896/files |
@aojea: Reopened this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
What would you like to be added?
Support watch service with field-selector spec.clusterIP!=None to skip headless service in kube-apiserver. And kubelet/kube-proxy can skip watch headless-service in the future.
![image](https://private-user-images.githubusercontent.com/42128207/291599493-55a3df12-e70d-4b09-b5a4-a5a6ebff890e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQwMDEyNDYsIm5iZiI6MTcyNDAwMDk0NiwicGF0aCI6Ii80MjEyODIwNy8yOTE1OTk0OTMtNTVhM2RmMTItZTcwZC00YjA5LWI1YTQtYTVhNmViZmY4OTBlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA4MTglMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwODE4VDE3MDkwNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWExMzRiZTYwODllZjljNzA2NDRiNzQ1NDMwODIwZDVkNGI5ZDNiNjVmNGU2ODJmYjY5MTkyZTJkZmIyZjk3YzEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.FnYDjksb6WN5CqPHKbQO5-sk-5A9KbyeQ3-NZcxmiIU)
Now, endpoints will be added labels service.kubernetes.io/headless for headless service by endpoints-controller, and kube-proxy will ignore event for it by label-selector. https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-proxy/app/server.go#L913. But headless service haven't be added label and kube-proxy still watch for it.
For kubelet, it watch service to add service-env for pods , but it ignores headless service when build service-env-map(https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_pods.go#L549). So kubelet don't need to watch headless service either.
Why is this needed?
In a large-cluster with more then 5000 nodes, when we run kubeflow task( tfjob:pod:headlessService=1:10:10) with qps 10, it can save huge network bandwidth and cpu for kube-apiserver.
The text was updated successfully, but these errors were encountered: