openmpi: master/workers sync mechanism is replaced with k8s api from redis

[everpeace]

Currently, openmpi package uses redis to sync master/workers. But I think it is overkilled a bit. And we can watch master/workers status via k8s api.

In a pod, service account's token and kubernetes api server endpoint is embedded. So, we can see pods status just executing curl like this.

# I think --insecure is safe because this will be executed in a pod (inside of kubernetes cluster)
curl --insecure \
  --header "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT/api/v1/namespaces/$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)/pods

So, I replaced master/workers sync mechanism with k8s api from redis.

What do you think?? I'm very happy if I could have some feedbacks.

---

This change is 

[everpeace]

cc/ @jiezhang @pdmack would you mind reviewing this?

[pdmack]

/ok-to-test

[everpeace]

@pdmack I'm so sorry. I force pushed my branch because I had typo... Could you /ok-to-test again 🙇 ??

[pdmack]

It's fine. Test is running. Stay tuned.

[pdmack]

/approve
/lgtm

Thanks @everpeace !

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: pdmack

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [pdmack]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jiezhang]

@everpeace @pdmack The worker reaches "running" state once init.sh starts running, but at that point the worker is not ready yet - it still has a bunch of initialization to finish.

[jiezhang]

But I do agree that introducing redis for synchronization wasn't an optimal solution.

[everpeace]

@jiezhang

• it still has a bunch of initialization to finish.

ah, you're right!! Then, the master can wait until ssh to all workers succeed. What do you think?

I created PR about this issue. Could you review this??