[openmpi]: Fix wait_mpi_ready() hangs when using OpenMPI 2.1.3

[everpeace]

What is the problem

We mainly use Open MPI 2.1.3 because OpenMPI 3.0.1 contains some bug on GPU Direct feature.

But unfortunately mpiexec in OpenMPI v2.1.3 doesn't exit properly when it couldn't resolve
hostnames of worker pods. Consequently wait_mpi_ready() loop in init.sh will hang forever.

How the PR fixes it

The only way I found to stop hanged mpiexec was to set MPIEXEC_TIMEOUT environ. So, I put this environ only in the loop of wait_mpi_ready(). I know it is a dirty hack though.

ref: https://www.open-mpi.org/doc/v2.1/man1/mpiexec.1.php#sect4

I would be happy if someone know other workarounds for it 🙇

How to Reproduce

Build docker image (Dockerfile)

This was published as everpeace/kubeflow-openmpi-21x-test:latest. So you don't need to build yourself. Please check what the image contains and proceed next step.

FROM ubuntu:16.04
ARG OPENMPI_VERSION="2.1.3"

# Install basic dependencies and locales
RUN apt-get update && apt-get install -yq --no-install-recommends \
  locales wget sudo ca-certificates ssh build-essential && \
  rm -rf /var/lib/apt/lists/* /var/cache/apt/archives/* && \
  echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && locale-gen

# Install OpenMPI
RUN cd /tmp && \
  wget -q https://www.open-mpi.org/software/ompi/v${OPENMPI_VERSION%\.*}/downloads/openmpi-$OPENMPI_VERSION.tar.bz2 && \
   tar -xjf openmpi-$OPENMPI_VERSION.tar.bz2 && \
   cd /tmp/openmpi-$OPENMPI_VERSION && \
   ./configure --prefix=/usr && make -j2 && make install && rm -r /tmp/openmpi-$OPENMPI_VERSION*  

Run

on 18b670dc3883ff733f840b25e3c65c6db6483637 that is current master branch head.

NAMESPACE=kubeflow
SECRET=openmpi-secret
yes | ssh-keygen -N "" -f .tmp/id_rsa
kubectl delete secret ${SECRET} -n ${NAMESPACE} || true
kubectl create secret generic ${SECRET} -n ${NAMESPACE} --from-file=id_rsa=.tmp/id_rsa --from-file=id_rsa.pub=.tmp/id_rsa.pub --from-file=authorized_keys=.tmp/id_rsa.pub
VERSION=18b670dc3883ff733f840b25e3c65c6db6483637
APP_NAME=openmpi-21x-test
ks init ${APP_NAME}
cd ${APP_NAME}
ks env set default --namespace ${NAMESPACE}
ks registry add kubeflow github.com/kubeflow/kubeflow/tree/${VERSION}/kubeflow
ks pkg install kubeflow/openmpi@${VERSION}

COMPONENT=this-will-hang
IMAGE=everpeace/kubeflow-openmpi-21x-test:latest
WORKERS=2
GPU=0
EXEC="mpiexec -n ${WORKERS} --hostfile /kubeflow/openmpi/assets/hostfile --allow-run-as-root --display-map -- echo hello"
ks generate openmpi ${COMPONENT} --image ${IMAGE} --secret ${SECRET} --workers ${WORKERS} --gpu ${GPU} --exec "${EXEC}"

ks apply default 

Once master pod reach Running, You can see mpiexec hanged as below.

$ kubectl logs -f this-will-hand-master
...
+ [ master = master ]
+ wait_mpi_ready 2 30
+ local workers=2
+ local max_retries=30
+ local retries=0
+ mpiexec -n 2 --hostfile /kubeflow/openmpi/assets/hostfile --allow-run-as-root -q sh -c echo $(hostname) is ready
ssh: Could not resolve hostname this-will-hang-worker-1.this-will-hang.kubeflow: Name or service not known
ssh: Could not resolve hostname this-will-hang-worker-0.this-will-hang.kubeflow: Name or service not known
# ... hang forever...

---

This change is 

[everpeace]

/assign @jiezhang @inc0
/cc @alsrgv @jlewi

[jiezhang]

/lgtm /approve

[jiezhang]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jiezhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• kubeflow/openmpi/OWNERS [jiezhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jiezhang]

/lgtm

[everpeace]

/test kubeflow-presubmit

[everpeace]

/test kubeflow-presubmit