HDFS cluster can't work with kubernetes + flannel in coreos

[ztao1987]

Hi,

I deployed kubernetes with flanneld.service enabled in coreos. And then I started hdfs namenode and datanode via kubernetes replication-controller. I also created kubernetes service for namenode. The namenode service ip is 10.100.220.223, while the pod ip of namenode is 10.20.96.4. In my case, one namenode and one datanode happens to be on same host. And namenode pod and datanode pod can ping each other successfully.
However I encountered the following two problems when trying to start hdfs datanode:

1. If I used namenode service ip 10.100.220.223 as fs.defaultFS in core-site.xml for datanode. When datanode tried to register itself to namenode via rpc request, namenode got the wrong ip address for the datanode. Normally it should get the pod ip of the datanode, but in this case docker0 inet address of datanode host is reported to namenode.
2. in order to workaround this, I used namenode pod ip 10.20.96.4 in core-site.xml for datanode. This time datanode can't be started at all. The error info reports that "k8s_POD-2fdae8b2_namenode-controller-keptk_default_55b8147c-881f-11e5-abad-02d07c9f6649_e41f815f.bridge" is used as namenode host instead of the namenode pod ip.

I tried to search this issue over the network, but nothing helps me. Could you please help me out of this? Thanks.

[eyakubovich]

@ztao1987 I don't know much about HDFS but I think what is happening is this. Because namenode is accessed as a service, the connection goes through kube-proxy, so all connections are actually coming from it (and its connections bind to docker0 interface). I think it would be best to ask Kubernetes community for best practices for working around this problem.

[eyakubovich]

Also see https://groups.google.com/forum/#!search/hdfs$20flannel/google-containers/P4uh7y383oo/bPzIRaxhs5gJ

[ztao1987]

@eyakubovich Thanks for you reply. I already posted this question in Kubernetes community. I checked this google group thread, it seems he workarounds it by using namenode pod ip. But in my case, namenode pod ip is translated into somethings like bridge name, do you happend to known why?

[eyakubovich]

But in my case, namenode pod ip is translated into somethings like bridge name

Not sure what you mean

[ztao1987]

I mean if I used namenode actual pod ip to start datanode, the datanode regcognize it as "k8s_POD-2fdae8b2_namenode-controller-keptk" and failed to start. I got workaround for this from @luqman in the google group thread you provide. He mentioned that I need DNS to solve this problem.

[ztao1987]

Is this issue caused by flannel or kubernetes? I mean "if using namenode actual pod ip to start datanode, the datanode regcognize it as "k8s_POD-2fdae8b2_namenode-controller-keptk""

[ztao1987]

I got the answer from kubernetes community. use the latest kubernetes and pass the params --proxy-mode=iptables to kube-proxy start command, HDFS cluster works now

[philips]

@ztao1987 Nice! We plan to make github.com/coreos/coreos-kubernetes default to iptables soon.

[ksr1]

@ztao1987
Hi, I'm trying to setup a HDFS cluster in Kubernetes. Do you have any info you can share on setting up the cluster? Any set of instructions would be appreciated if have them documented. Thanks.

[ztao1987]

@ksr1 Assumed you already have a namenode/datanode docker image, and then deploy it using ReplicationController/deamonset in k8s. Rember to start kube-proxy using params --proxy-mode=iptables. If you want to do data persistent, you should use volumes to volumeMounts the data whatever you want. No special things for other parts.

[ksr1]

@ztao1987 Thanks for the reply. Will try setting the params in Kubernetes and try the setup.

[tomdee]

I think this is resolved. Please comment if not and I can reopen.