pod cant access nodeport

[gaoya16]

my cluster have three nodes，node1、node2、node3， i start a deployment and service， the
pod run on node2，in pod i can curl node1：port and node3：port，but cant curl node2：port。

i use flannel v0.10.0
kubernetes 0.10
Docker version 18.03.0-ce

thanks a lot！

[gaoya16]

/sig Network
/sig Node

[hyponet]

kubernetes 0.10？

can you check kube-proxy is running
or node2 has some firewall
or find some strange log from kube-proxy.

[gaoya16]

sorry, version is v1.10
kube-proxy is running well, and most service is ok, just the pod cant access the service itself by nodeport
pod ip:172.18.1.4, in pod, curl 172.18.1.4:8050 is ok, but curl 192.168.0.2:8050 will fail (192.168.0.2 is the ip of the host, the pod run on this host)
thanks a lot

[athenabot]

/triage unresolved

🤖 I am a bot run by @vllry. 👩‍🔬

[pswica]

What happens when you run:

kubectl get no

Also, where are the nodes hosted (VMware, AWS, Bare Metal, etc)?

One other thing to note:
Flannel, if you just run it with kubectl apply -f (e.g. you didn't edit the yaml), requires a default pod cidr of 10.244.0.0/16

[gaoya16]

thanks for your reply，

kubectl get no：
NAME STATUS ROLES AGE VERSION
128-195-0-1 Ready 166d v1.10.0
128-195-0-2 Ready 165d v1.10.0
128-195-0-3 Ready 166d v1.10.0

my nodes hosted VMware；

about the default pod cidr of 10.244.0.0/16, I have not config this, how should i config? thank you.

[pswica]

https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

I put the link to the yaml above. If your VMs just have one interface, it may be enough to edit or delete/re-deploy the ConfigMap flannel comes with (just look for "10.244.0.0" in the above link).

If your cluster is new, however, I'd be safe and just re-deploy the whole thing from scratch if it will be used in production. CNI is pretty critical in Kubernetes and you don't want leftover weirdness anywhere from an old misconfigured one.

[gaoya16]

I'm very sorry that i have not follow you, and i am not sure how to execute, I deploy flannel by rpm, not an
k8s app, should i just "kubectl create -f kube-flannel.yml" ?

[vllry]

/remove-triage unresolved

[pswica]

I see, honestly I haven't installed flannel by .rpm before only as a bunch of kubernetes resources via the yaml file. How did you set Kubernetes up (e.g. via kubeadm, completely by hand, using kops, etc)?

Also, how did you set your pod cidr network when you installed? Did you do it with etcdctl directly (e.g. something like echo '{"Network": "10.10.0.0/16", "SubnetLen": 24, "Backend": {"Type": "vxlan", "VNI": 1}}' | etcdctl set /atomic.io/network/config)?

[gaoya16]

thanks for reply,

I installed flannel just by copying binary file, and then run some scripts; And I set kubernetes up completely by hand.

I use etcdctl mk command to set the subnet, and i checked it:
root> etcdctl get /cmpk8s/network/config
{"Network":"172.18.0.0/16", "SubnetMin": "172.18.1.0", "SubnetMax": "172.18.254.0", "Backend": {"Type": "vxlan"}}

[pswica]

Is there a route to Node 2 from the other nodes? Does Node 2 have the appropriate routes to it's pods. Also, is Node 2 listening on the appropriate port?

For the first question, you can run route on all the nodes and compare. For the second, you can use something like netstat -tulpn | grep <port> from Node 2.

One other idea comes to my mind: Did you set all the kernel params on all three nodes? (e.g. with a tool like sysctl? Perhaps leaving this out during manual setup of the node prevents IP forwarding?

[gaoya16]

yes, i have checked that:
i have three nodes:
128-195-0-1 Ready 166d v1.10.0
128-195-0-2 Ready 165d v1.10.0
128-195-0-3 Ready 166d v1.10.0
and the pod run on 128-195-0-2, ip is 172.18.1.5;
and in pod, i curl 128.195.0.2:8888, it failed
but in another pod run on 128.195.0.1 or 128.195.0.3, i curl 128.195.0.2:8888, will be success;

And i have checked the iptables, and did not find some wrong

thanks a lot

[pswica]

This is a pretty tough issue then! Can you at least ping the node from the pod? Can you ping it by both IP and DNS name?

[gaoya16]

ping is ok， in pod, ping host or other pod, all is ok.
and in the pod on 128.195.0.2, can curl 128.195.0.2:9999(another service provided by another pod), so i mean that the pod cant curl the service itself provide by the host:port, and the host is the pod where it is running, but in the pod can curl pod_ip:port.

[freehan]

This is not a kubernetes issue. More like a flannel issue?

[gaoya16]

all right，thanks a lot for your assistance!