Cannot ping pods from different minions

[moix]

Hi, I've been for a couple of weeks trying to setup a kubernetes cluster in centos with etcd and flanneld following current getting started guidelines (https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/getting-started-guides/centos/centos_manual_config.md) and all the information I could find out in the repostory and other sources from google, so far still couldnt find how to make it work :(

Sorry for the length of the ticket but I want to detail every configuration:

My setup has 3 servers:

Server	Name
centos-master	kubernetes master node
centos-minion1	first minion node
centos-minion2	first minion node

host file is updated in all of them so they can ping each other.

The problem is when I try to ping to pod ips from different minions. Following all the relevant information:

configuration and tests in centos-master

packages installed

# rpm -qa | grep kube
kubernetes-0.17.1-3.el7.x86_64
cockpit-kubernetes-0.53-3.el7.x86_64
kubernetes-master-0.17.1-3.el7.x86_64
kubernetes-node-0.17.1-3.el7.x86_64

# rpm -qa | grep etcd
etcd-2.0.9-2.el7.x86_64

configuration files for kubernetes and etcd

- /etc/kubernetes/apiserver

###
# kubernetes system config
#
# The following values are used to configure the kube-apiserver
#

# The address on the local server to listen to.
KUBE_API_ADDRESS="--address=0.0.0.0"

# The port on the local server to listen on.
KUBE_API_PORT="--port=8080"

# How the replication controller and scheduler find the kube-apiserver
KUBE_MASTER="--master=http://centos-master:8080"

# Port minions listen on
KUBELET_PORT="--kubelet_port=10250"

# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd_servers=http://127.0.0.1:4001"

# Address range to use for services
# KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"
KUBE_SERVICE_ADDRESSES="--portal_net=10.254.0.0/16"

# default admission control policies
KUBE_ADMISSION_CONTROL="--admission_control=NamespaceAutoProvision,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"

# Add your own!
KUBE_API_ARGS=""

• /etc/kubernetes/controller-manager

###
# The following values are used to configure the kubernetes controller-manager

# defaults from config and apiserver should be adequate

# Comma separated list of minions
KUBELET_ADDRESSES="--machines=centos-minion1,centos-minion2"

# Add your own!
KUBE_CONTROLLER_MANAGER_ARGS=""

• /etc/etcd/etcd.conf

ETCD_NAME=default
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:4001"

processes running in centos-master

/usr/bin/kube-apiserver --logtostderr=true --v=0 --etcd_servers=http://127.0.0.1:4001 --address=0.0.0.0 --port=8080 --kubelet_port=10250 --allow_privileged=false --portal_net=10.254.0.0/16 --admission_control=NamespaceAutoProvision,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota
/usr/bin/kube-controller-manager --logtostderr=true --v=0 --machines=centos-minion1,centos-minion2
/usr/bin/kube-scheduler --logtostderr=true --v=0

/usr/bin/etcd

output from some kubectl commands

• configuration of etcd

# etcdctl get /coreos.com/network/config
{
    "Network": "18.16.0.0/16",
    "SubnetLen": 24,
    "Backend": {
        "Type": "vxlan",
        "VNI": 1
     }
}

• kubectl get nodes

NAME                                 LABELS                                                      STATUS
centos-minion1                       kubernetes.io/hostname=centos-minion1                       Ready
centos-minion2                       kubernetes.io/hostname=centos-minion2                       Ready

After creating a couple of simple rc:

• kubectl get pods

POD              IP            CONTAINER(S)   IMAGE(S)         HOST                                               LABELS          STATUS    CREATED      MESSAGE
busybox1-59sm9   18.16.98.21                                   centos-minion1/159.107.29.71                       name=busybox1   Running   6 minutes
                               busybox1       busybox:latest                                                                      Running   6 minutes
busybox2-zoisy   18.16.37.39                                   centos-minion2/159.107.29.89                       name=busybox2   Running   6 minutes
                               busybox2       busybox:latest                                                                      Running   6 minutes

configuration and tests in centos-minion1 (same for centos-minion2)

packages installed

# rpm -qa | grep kube
kubernetes-master-0.17.1-3.el7.x86_64
kubernetes-node-0.17.1-3.el7.x86_64
kubernetes-0.17.1-3.el7.x86_64

# rpm -qa | grep flannel
flannel-0.2.0-7.el7.x86_64

# rpm -qa | grep docker
docker-master-1.7.0-4.git56481a3.el7.x86_64

configuration files for kubernetes and flannel

- /etc/kubernetes/config

# Comma separated list of nodes in the etcd cluster
KUBE_ETCD_SERVERS="--etcd_servers=http://centos-master:4001"

# logging to stderr means we get it in the systemd journal
KUBE_LOGTOSTDERR="--logtostderr=true"

# journal message level, 0 is debug
KUBE_LOG_LEVEL="--v=0"

# Should this cluster be allowed to run privileged docker containers
KUBE_ALLOW_PRIV="--allow_privileged=false"

# How the controller-manager, scheduler, and proxy find the apiserver
KUBE_MASTER="--master=http://centos-master:8080"

• /etc/kubernetes/kubelet

###
# kubernetes kubelet (minion) config

# The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
KUBELET_ADDRESS="--address=0.0.0.0"

# The port for the info server to serve on
KUBELET_PORT="--port=10250"

# You may leave this blank to use the actual hostname
KUBELET_HOSTNAME="--hostname_override=centos-minion1"

# location of the api-server
KUBELET_API_SERVER="--api_servers=http://centos-master:8080"

# Add your own!
KUBELET_ARGS="--cluster-dns=10.254.100.10 --cluster-domain=kubdomain.local"

• /etc/sysconfig/flanneld

# Flanneld configuration options

# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD="http://centos-master:4001"

# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_KEY="/coreos.com/network"

# Any additional options that you want to pass
#FLANNEL_OPTIONS=""

• Content of flannel subnet.env in centos-minions is apparently ok, for example in centos-minion1

[root@centos-minion1 ~]# cat /run/flannel/subnet.env
FLANNEL_SUBNET=18.16.98.1/24
FLANNEL_MTU=1450

processes running in minions

/usr/bin/kube-proxy --logtostderr=true --v=0 --master=http://centos-master:8080
/usr/bin/kubelet --logtostderr=true --v=0 --api_servers=http://centos-master:8080 --address=0.0.0.0 --port=10250 --hostname_override=centos-minion1 --allow_privileged=false --cluster-dns=10.254.100.10 --cluster-domain=kubdomain.local

/usr/bin/flanneld -etcd-endpoints=http://centos-master:4001 -etcd-prefix=/coreos.com/network

/usr/bin/docker -d --selinux-enabled --bip=18.16.37.1/24 --mtu=1450

And now the problem: I can ping ips to other pods within the same minion but when I pinging from pods in different minions:

ping from busybox1-59sm9 in centos-minion1 (pod ip 18.16.98.21) to busybox2-zoisy in centos-minion2 (pod ip 18.16.37.39)

[root@centos-minion1]# docker exec -ti 04d53905fc4b ping 18.16.37.39
PING 18.16.37.39 (18.16.37.39): 56 data bytes
^C
--- 18.16.37.39 ping statistics ---
8 packets transmitted, 0 packets received, 100% packet loss

I'm not sure what else I can try, I'm a bit lost with all the different guidelines in the documentation but not sure which one is more accurate and what can be failing in my setup.

Any clue what can I check, thanks!

[moix]

Hi, already got the issue and fixed. somehow at some step the flannel0 was removed from the routing table in one of the minions, so adding it fixed the issue.

Thanks.

[bprashanth]

@moix thanks for the detailed bug report anyway

[suyogbarve]

@moix : I am facing the same issue, I am unable to ping the pods on a different node.
Even from master I am not able to ping any pods.
How did you fixed your problem?

[moix]

Hi @suyogbarve, in my case it was a problem in one node related to flannel interface was removed from routing table.
Please check ip a and route -n commands and ensure pods can route each other via flannel interface.
Is docker running with --bip option?

[thuey]

@moix I'm having the same problem. Docker is not running with --bip. Should it be?

[moix]

Hi @thuey, yes flannel should configure docker service in nodes with that option so each node uses a different network in the range configured in etcd (18.16.0.0/16 in this case)

[pgnaleen]

i am also face this problem please some one give me clear simple answer.