Service without selector but with matching Endpoint will cause timeout

[galexrt]

Bug report

General Information

• Cilium version (run cilium version)

Client: 1.8.1 5ce2bc7b3 2020-07-02T20:04:47+02:00 go version go1.14.4 linux/amd64
Daemon: 1.8.1 5ce2bc7b3 2020-07-02T20:04:47+02:00 go version go1.14.4 linux/amd64

Cilium is installed to replace the kube-proxy, kubeadm init has been run with the skip-phase kube-proxy flag!

• Kernel version (run uname -a)

Ubuntu 20.04 5.4.x and Fedora 32 5.7.7-200

• Orchestration system version in use (e.g. kubectl version, Mesos, ...)

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:47:41Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-06-26T03:39:24Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

• Link to relevant artifacts (policies, deployments scripts, ...)
• Generate and upload a system zip: Please let me know if sysdumps are needed in this case.

How to reproduce the issue

A Service without a selector with a matching Endpoint object created does not work (timeout when talking to the Service ClusterIP). cilium service list does not show the Service when the Service doesn't have a selector: but a matching Endpoint object.

Steps:

kubectl create -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-nginx
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      run: my-nginx
  template:
    metadata:
      labels:
        run: my-nginx
    spec:
      containers:
      - image: nginx
        imagePullPolicy: Always
        name: my-nginx
        ports:
        - containerPort: 80
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  name: my-nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  type: ClusterIP
  selector:
    run: my-nginx
status:
  loadBalancer: {}
EOF
kubectl rollout status deployment my-nginx
SERVICE_IP="$(kubectl get service/my-nginx -o jsonpath='{.spec.clusterIP}')"
curl -v $SERVICE_IP:80

curl should return "default" nginx index / welcome page.Run commands:

kubectl get svc my-nginx -o yaml > service.yaml
kubectl get endpoints my-nginx -o yaml > endpoints.yaml
kubectl delete -f service.yaml

Remove the label selector: from the service.yaml.

vim service.yaml endpoints.yaml
kubectl create -f service.yaml
kubectl create -f endpoints.yaml
SERVICE_IP="$(kubectl get service/my-nginx -o jsonpath='{.spec.clusterIP}')"
curl -v $SERVICE_IP:80

curl should timeout / fail here now, even though there is a "perfectly" valid Endpoints object with Pod target endpoints in it.

In minikube this works as expected that the Service without a selector but a matching Endpoint object is still reachable. (edited)

(see https://app.slack.com/client/T1MATJ4SZ/threads/thread/C53TG4J4R-1593434412.267800)

[galexrt]

Sysdump attached to this comment.

cilium-sysdump-20200707-103546.zip

[fristonio]

I was able to reproduce the issue and I have figured out the reason this is happening. In the K8s version where we have support for EndpointSlices we don't start a watcher for Endpoint objects.
https://github.com/cilium/cilium/blob/master/pkg/k8s/watchers/watcher.go#L439-L450
In the above example, we are creating the endpoint object that is associated with the service. This is in turn never picked by the watcher and the service remains without a backend due to which cilium does not have a service entry for the mentioned service. As a simple fix, instead of creating an Endpoint object, you should create EndpointSlices object associated with the service.

I will start a discussion on slack as to how we should handle these kinds of issues and will push a fix if we need one.

[fristonio]

Edit on how to reproduce this:
Make sure you have installed cilium with kube-proxy-replacement=strict mode and you are running a K8s version that has support for EndpointSlices.

[brb]

@fristonio Good catch! Is this a duplicate of #12513 then?

[galexrt]

@brb Yeah, seems like it from my point of view. I'll update my K8S cluster to v1.19.x and report back here / in #125123 till end of the week(end).

[fristonio]

Tested on Kubernetes 1.19 with Kubeadm, seems to work out fine in with the latest Kubernetes release. @brb

[galexrt]

I can confirm it too now, updating the K8S cluster to v1.19.0 fixed the issue thanks to the added EndpointSlices Mirroring controller.