Wrong datanode IP in topology when deployed in kubernetes

[ilyakitaev]

Hello, I deployed ozone with helm chart. Everythin is ok and running. But when I try to put file with hdfs dfs -put I got error that connection refused. Client wants to connect to NODE ip, not cluster ip of a pod.
Hostnames inside the datanodes is ok, everything resolving as it should be. But ozone admin printTopology shows same kubernetes node IP for every datanode but different hostnames (datanode pods are running on different nodes)

The main question is how OM determines IP of datanode? If someone can help, thanks.

[ivandika3]

Hi @ilyakitaev , this might be related to the issues highlighted in discussion #7259. Currently, client seems to use only datanode IP address causing issues in K8s env. https://issues.apache.org/jira/browse/HDDS-13124 fixed this issue so that client can read the datanodes using hostname instead of IP. You can try to backport it first and try with the by setting the hdds.datanode.use.datanode.hostname to true both on the server-side (OMs and SCMs).

@sokui @lukmanulhakkeem @KochankovID might be more familiar with this issue.

[ilyakitaev]

Hi @ivandika3, for me problem was solved disabling NAT in trusted zone in firewalld. For calico if it is installed you have to add k8s internal network in trusted zone. Why only single node ip was resolved for all data nodes idk.

[kerneltime]

cc @Tejaskriya