Pod unable to reach itself through a service (unless --cni=true is set)

[arrawatia]

Minikube version (use minikube version):
minikube version: v0.16.0 and k8s version v1.6.4
(But I tried v0.17.1 and v0.19.1 too)
Environment:

• OS (e.g. from /etc/os-release):
• VM Driver :
  Virtualbox
• ISO version :
  minikube-v1.0.6.iso
• Install tools:
  curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.16.0/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
• Others:

What happened:
If a pod has a service which points to the pod, the pod cannot reach itself through the service IP. Other pods can reach the service and the pod itself can reach other services. This means all components (especially clustered & distributed systems) which expect to talk to themselves for leader election fail to startup properly.

What you expected to happen:
I expect the pod to be able to reach itself.

How to reproduce it (as minimally and precisely as possible):
It happens with all our services and pods but I can reproduce it with kube-system pods too.

Get service IP : kubectl describe svc kube-dns --namespace kube-system | grep IP:. I get 10.0.0.10
Get endpoint IP: kubectl describe svc kube-dns --namespace kube-system | grep Endpoints. I get 172.17.0.3

Exec into the pod:
kubectl --namespace kube-system exec -it kube-dns-v20-54536 sh

Run the following :
Using the service IP hangs

Name: kubernetes-dashboard.kube-system.svc.cluster.local
Address 1: 10.0.0.212 kubernetes-dashboard.kube-system.svc.cluster.local
/ # nslookup kubernetes-dashboard.kube-system.svc.cluster.local 10.0.0.10
Server: 10.0.0.10
^C

Using the endoint IP works

/ # nslookup kubernetes-dashboard.kube-system.svc.cluster.local 172.17.0.3
Server: 172.17.0.3
Address 1: 172.17.0.3 kube-dns-v20-54536

Name: kubernetes-dashboard.kube-system.svc.cluster.local
Address 1: 10.0.0.212 kubernetes-dashboard.kube-system.svc.cluster.local

Accessing a different service IP works. Using the kubernetes-dashboard IP from the last command
/ # telnet 10.0.0.212 80
get
HTTP/1.1 400 Bad Request
Content-Type: text/plain
Connection: close

400 Bad RequestConnection closed by foreign host

Anything else do we need to know:
minikube v0.17.1 works with K8S 1.5.3
I tried the following and it worked. So, I suspect it has something to do with upgrading minikube to v0.17.1 and K8S to v1.6.4

curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.17.1/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/
rm -rf ~/.minikube
minikube start --kubernetes-version 1.5.3 --cpus 4 --memory 6096 --v=8 --logtostderr

[ursuad]

It seems it's related to this: kubernetes/kubernetes#19930 and this kubernetes/kubernetes#20475

[ursuad]

For me, this helped to fix it kubernetes/kubernetes#20475 (comment)

So you can do:

minikube ssh
sudo ip link set docker0 promisc on

Maybe this fix can be merged directly in minikube so people won;t need to do custom things?

[arrawatia]

@ursuad It worked for me too (minikube 0.19.1 and k8s v1.6.4). Thanks a lot for your help. :)

[ursuad]

@arrawatia No problem, but maybe you should leave this issue open, so we'll have a longer term fix merged in minkube.
This is a bug, and my fix is just a workaround.

[rajasaur]

Not sure if there is a page to capture minikube gotchas. @ursuad's suggestion should be there

[yonatane]

Use Headless Service as a workaround (clusterIP: None). StatefulSets that talk to themselves like Kafka use headless services anyway.
This might still be considered a bug though.

[ursuad]

@kubernetes/sig-minikube

[r2d4]

I'll take a look at this

[medyagh]

@r2d4 I am pretty sure this is same as this #1690

[medyagh]

@arrawatia can you try with minikube with k8s 1.5.1 I am almost certain it will work in 1.5.1
I have that problem ! #1690

[wstrange]

Seeing this issue on minikube 0.21.0 and kube 1.70 and 1.7.2

The workaround posted by @ursuad seems to solve the issue for me

[r2d4]

I'm going to open this up again, since we ended up reverting the kubenet change.

[r2d4]

ref #1742

[wstrange]

Just updating the status for minikube 0.22.0 - the issue is still present

[chancez]

Still seeing this in minikube v0.23.0

[guilhermeblanco]

... and v0.24.0...

[manikantag]

... and v0.24.1

[YassineHk]

the same problem here 🤦

minikube version: v1.11.0

[tstromberg]

Fixing this by default appears to incur a ~30% performance penalty for startup, which makes me quite wary of imposing it on the users who do not care about CNI.

At a minimum though, we should document that it is possible to now say --cni=true to make this work.

[azhao155]

Have tested this with docker driver, running a service of two pods with simple nginx binary. Able to get response using service dns name ..svc.cluster.local from the pods.
Since docker driver will enable cni by default, so using docker driver won't encounter this issue.

[azhao155]

Also tested on kvm driver, doesn't encounter this issue when ping <service_name>..svc.cluster.local.

[azhao155]

./out/minikube start

create nginx service

kubectl apply -f template/nginx.yaml
kubectl expose deployment my-nginx

ssh into pods

kubectl get pods
Output:
NAMESPACE NAME READY STATUS RESTARTS AGE
default my-nginx-5b56ccd65f-hpktn 1/1 Running 0 45s
default my-nginx-5b56ccd65f-zqc72 1/1 Running 0 45s

kubectl exec -it my-nginx-5b56ccd65f-hpktn -- /bin/bash
curl my-nginx.default.svc.cluster.local

<title>Welcome to nginx!</title> <style> body { width: 35em; margin: 0 auto; font-family: Tahoma, Verdana, Arial, sans-serif; } </style>

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org.
Commercial support is available at nginx.com.

Thank you for using nginx.

[azhao155]

Nginx template:
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-nginx
spec:
selector:
matchLabels:
run: my-nginx
replicas: 2
template:
metadata:
labels:
run: my-nginx
spec:
containers:
- name: my-nginx
image: nginx
ports:
- containerPort: 80

[medyagh]

thanks @azhao155 for confirming this issue doesnt happen anymore.

closing please feel free to re-open if still see this problem

[belfo]

@medyagh @azhao155
I'm trying using driver=none but that don't work.
Could you help?

[afbjorklund]

@belfo : please open a new issue.

[Dan0x54]

I'm finding @azhao155's test works when the replicas are set to two but the test fails if I run
kubectl scale deploy/my-nginx --replicas=1 (or set the replicas manually in the YAML). It works if the service is headless. Is this the expected behaviour? I'm using the docker driver btw.

minikube version: v1.24.0
commit: 76b94fb3c4e8ac5062daf70d60cf03ddcc0a741b

[pchambre]

Still seeing this issue on Minikube 1.27 using hyperkit.

--cni=true set up the CNI, but still did not allow a Pod to connect to itself through a service.

sudo ip link set docker0 promisc in the Minikube VM did fix the behavior.

[jgoeres]

Also seeing this on 1.28 with Hyper-V driver.
The workaround (sudo ip link set docker0 promisc on) fixes it, but is rather inconvenient.