StatefulSet Init container cannot resolve another Pod hostname

[mayask]

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:57:05Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"08e099554f3c31f6e6f07b448ab3ed78d0520507", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Cloud provider or hardware configuration: MacBook Pro 15, VirtualBox
• OS (e.g. from /etc/os-release): CentOS 7
• Kernel (e.g. uname -a): Linux localhost.localdomain 3.10.0-514.2.2.el7.x86_64 #1 SMP Tue Dec 6 23:06:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
• Others:

What happened:
When trying out Running a Replicated StatefulSet cluster example on kubernetes site, clone-mysql init container of mysql-1 Pod cannot resolve hostname of previous StatefulSet instance mysql-0.mysql and therefore fails.

[root@localhost docker-multinode]# kubectl get po
NAME                    READY     STATUS                  RESTARTS   AGE
mysql-0                 2/2       Running                 0          59m
mysql-1                 0/2       Init:CrashLoopBackOff   11         46m

[root@localhost docker-multinode]# kubectl logs -f -c clone-mysql mysql-1                                                                                                                            
+ [[ -d /var/lib/mysql/mysql ]]
++ hostname
+ [[ mysql-1 =~ -([0-9]+)$ ]]
+ ordinal=1
+ [[ 1 -eq 0 ]]
+ ncat --recv-only mysql-0.mysql 3307
+ xbstream -x -C /var/lib/mysql
Ncat: Connection timed out.
+ xtrabackup --prepare --target-dir=/var/lib/mysql
xtrabackup version 2.4.4 based on MySQL server 5.7.13 Linux (x86_64) (revision id: df58cf2)
xtrabackup: cd to /var/lib/mysql
xtrabackup: Error: cannot open ./xtrabackup_checkpoints
xtrabackup: error: xtrabackup_read_metadata()
xtrabackup: This target seems not to have correct metadata...
InnoDB: Number of pools: 1
InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find.
InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
  xtrabackup: Fatal error: cannot find ./xtrabackup_logfile.
xtrabackup: Error: xtrabackup_init_temp_log() failed.

This bring me to the question, is DNS propagated yet by the time Pod's init container is executing?

What you expected to happen:
clone-mysql container of mysql-1.mysql Pod successfully resolves mysql-0.mysql hostname.

How to reproduce it (as minimally and precisely as possible):
Try running Replicated StatefulSet cluster example.

Anything else we need to know:

[mayask]

The issue was because of Docker 1.13 iptables bug #40182.

[donghuarui]

so,how do you solve this problem?

[mayask]

Yeah, I threw this k8s crap out of the window