Support skip watch Headless Service in server-side.

[Sakuralbj]

What would you like to be added?

Support watch service with field-selector spec.clusterIP!=None to skip headless service in kube-apiserver. And kubelet/kube-proxy can skip watch headless-service in the future.

Now, endpoints will be added labels service.kubernetes.io/headless for headless service by endpoints-controller, and kube-proxy will ignore event for it by label-selector. https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-proxy/app/server.go#L913. But headless service haven't be added label and kube-proxy still watch for it.

For kubelet, it watch service to add service-env for pods , but it ignores headless service when build service-env-map(https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet_pods.go#L549). So kubelet don't need to watch headless service either.

Why is this needed?

In a large-cluster with more then 5000 nodes, when we run kubeflow task( tfjob:pod:headlessService=1:10:10) with qps 10, it can save huge network bandwidth and cpu for kube-apiserver.

[neolit123]

/sig network api-machinery

[aojea]

In a large-cluster with more then 5000 nodes, when we run kubeflow task( tfjob:pod:headlessService=1:10:10) with qps 1

how many headless Services does that kubeflow task creates?

[aojea]

/sig scalability
this falls into that area too

[Sakuralbj]

In a large-cluster with more then 5000 nodes, when we run kubeflow task( tfjob:pod:headlessService=1:10:10) with qps 1

how many headless Services does that kubeflow task creates?

Under steady-state conditions, the cluster maintains 30,000 headless services. During daily new tasks, an additional 10,000 headless services will be created. The number of nodes in the cluster is around 7,000

[aojea]

interesting, I can see lately a lot of consumption of headless Services specially in AI/ML workloads, instead of labelling Services we can add a field selector on the ClusterIP field, WDYT @wojtek-t @thockin ?

/cc @alculquicondor @ahg-g

[aojea]

Submitted #122541 , we are already having a selector for PodIP #50091 so I think is ok having a field selector for the clusterIP field, that will bring the benefit of being able to build informers that filter headless services by using spec.clusterIP!="None"
/assign

[Sakuralbj]

Submitted #122541 , we are already having a selector for PodIP #50091 so I think is ok having a field selector for the clusterIP field, that will bring the benefit of being able to build informers that filter headless services by using spec.clusterIP!="None" /assign

@aojea
We firstly consider support the function by field-selector for clusterIP, compared to add label to headless service as we discussed in #122502 , the main disadvantages is:

kube-apiserver needs to support field-selector with clusterIP firstly, and then kubelet/kube-proxy can be updated. Otherwise, clients performing List-watch operations will receive errors like this, which is not suitable for updating on existing clusters.

When using the way for adding labels for headless-service, kube-proxy does not need to be updated (As it always list-watch with label-selector service.kubernetes.io/headless not exists) .

[Sakuralbj]

@aojea If adding a field-selector for clusterIP is considered to be ok. I think we can add IndexFields for service spec.clusterIP as well (such as node metadata.name)，it can reduce events-count converted from kube-apiserver common channel to every cacheWatcher，reduce cpu-usage for kube-apiserver.

[aojea]

. I think we can add IndexFields for service spec.clusterIP as well (such as node metadata.name)，it can reduce events-count converted from kube-apiserver common channel to every cacheWatcher，reduce cpu-usage for kube-apiserver.

That is on sig-scalability area and better @wojtek-t to decide in that, I can see how Headless services are more used lately in Kubernetes , special fo AI/ML workloads, so I think is a fair request to implement this field selector and allow to filter server side headless service, but I do not know the tradeoffs of using IndexFields

[wojtek-t]

instead of labelling Services we can add a field selector on the ClusterIP field, WDYT @wojtek-t @thockin ?

+1

I would be ok with adding IndexFields for headless services too [re tradeoff, it adds a bit of memory usage, but it is fairly low - there was a similar thing happening recently in https://github.com//issues/120778 that has a lot of good analysis done]

[aojea]

I would be ok with adding IndexFields for headless services too [re tradeoff, it adds a bit of memory usage, but it is fairly low - there was a similar thing happening recently in https://github.com//issues/120778 that has a lot of good analysis done]

I can see that IndexFields implementation is feature gated, I think we can iterate here and go with the fieldselector first and evaluate later the IndexFields addition

[alculquicondor]

The documentation currently says that headless services are labeled. Is this not accurate?

https://kubernetes.io/docs/reference/labels-annotations-taints/#servicekubernetesioheadless

[aojea]

The documentation currently says that headless services are labeled. Is this not accurate?

https://kubernetes.io/docs/reference/labels-annotations-taints/#servicekubernetesioheadless

The control plane adds this label to an Endpoints object when the owning Service is headless.

[alexzielenski]

/triage accepted

[aojea]

/reopen

We had to revert the fix https://github.com/kubernetes/kubernetes/pull/123896/files

[k8s-ci-robot]

@aojea: Reopened this issue.

In response to this:

/reopen

We had to revert the fix https://github.com/kubernetes/kubernetes/pull/123896/files

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.