Create & label P1 issues needed for an initial release of Horovod support

[jlewi]

We should create issues for everything that needs to happen to do an initial release of Horvod support. All such issues should be priority/p1.

We'd like to include Horvod in our 0.2 release. We need to figure out all the work that needs to happen to support that.

I created the label area/horvod

Anyone in the org should be able to label issues as follows

/area horvod
/priority p1

@jiezhang @everpeace Could you take a stab at coming up with a list of issues and labeling existing issues?

[jiezhang]

Thanks @jlewi.

What's the timeline for 0.2 release? I think we should rename the area as openmpi. The package was written in a way that it's not coupled with horovod. For example, @everpeace is using the package to run distributed machine learning tasks which has nothing to do with horovod.

cc @alsrgv

[jiezhang]

/area openmpi

[jiezhang]

@jlewi Looks like the area command is not working for me.

[jlewi]

The area/openmpi label doesn't exist. Can we use the area/horvod?

[jiezhang]

@jlewi The package is not limited to horovod. It can run any MPI job. That's why I propose to rename the label as openmpi.

[everpeace]

@jlewi Yeah, naming is a bit complicated. as jiezhang said, openmpi package is technically independent from Horovod. And, actually, I'm using this package with Chainer/ChainerMN.

Could you take a stab at coming up with a list of issues and labeling existing issues?

Sure. I'm happy to do it. Here are several new issues which I would like to put in 0.2. "new" means that they have not created yet. What do you think @jiezhang ??

• support securityContext(especially RunAs,RunAsGroup)
  • In our cluster, we use NFS. when mounting NFS, we prohibit users to run pods with root from security reason.
• support volume and volume mounts (relates to [openmpi] Upload trained models to persistent storage  #713 ?)
• pave e2e test (at least on basic sample??)
• (low priority) support initContainer
  • Imagine pulling user code via git-sync or so.

labeling existing issues?

@jlewi I opened three PR yesterday, How should I handle them? I think I would create stab isseues for them and put the label. Is that ok?? Or can I put labels to PR directly? cc/ @jiezhang

off topic here though, I'm now thinking I would make a proposal of chainer-operator by the way.

[jlewi]

I created the label area/openmpi

[pineking]

support securityContext(especially RunAs)
In our cluster, we use NFS. when mounting NFS, we prohibit users to run pods with root from security reason.

@everpeace securityContext(RunAs) does not support group id, how can you specify gid in container when your programs read/write data using NFS

[everpeace]

/area openmpi

[everpeace]

@pineking RunAsGroup is already on feature-gate since 1.10.

However, this RunAsGroup didn't work nicely with NFS VolumeSource. So we patched by ourselves (it is a small patch though). Of course, we plan to contribute it to the upstream.

To secure NFS data, you will need to stripSETUID capability in pod security policy too. Otherwise, users can build sudo-able conatainer.

// I updated my original comment.

[pineking]

@everpeace Great, Thanks for your information. I will try the RunAsGroup feature, and look forward to your PR of the patch.

[jiezhang]

Thanks @jlewi . We'll create/tag issues needed for initial release.

[pdmack]

's/horvod/horovod/g'

[jlewi]

@jiezhang @everpeace I see two issues labeled area/openmpi ? Is the list complete? Does it include items related to Horovod and Open MPI support?

[everpeace]

@jlewi I would like to support one more feature of volumes/volumeMounts support (#838) if I could, sorry it was added just now, once kubeflow supports ks 0.10.2(#727). I set p2 to it because I'm not sure #727 was fixed before kubeflow 0.2 🙇 This is the last issue from me for kubeflow 0.2.

[jbottum]

/area 0.4.0

[jlewi]

@everpeace What is the status openmpi support? Can we close this issue and open up appropriate issues tagged 0.4.0?

[everpeace]

yes, please 🙇 But as a personal opinion, encouraging users to migrate to MPIJob CRDs would be nicer.

[jlewi]

@everpeace Can you explain that last comment? Are you saying that in 0.4 we should remove
https://github.com/kubeflow/kubeflow/tree/master/kubeflow/openmpi

And tell people to use the
mpi job crd
https://github.com/kubeflow/kubeflow/tree/master/kubeflow/mpi-job

[everpeace]

@jlewi sorry for the confusion.

Are you saying that in 0.4 we should remove

Yes, I was. But, in this several days, I understood that there are some active users using openmpi package.

So, I think we don't need to remove this package at least in 0.4. However, I would like to inform users the fact that they have an option to use mpi crd instead of openmpi package. Because, openmpi package is based on bare pods so no fault tolerance (it can NOT retry when failure), but mpi-operator expands mpi crd to kind:Job and kind:StatefulSet so it has fault-tolerance (it CAN retry when failure).

[jlewi]

SGTM.

Any suggestion about how we should start pushing users to use the CRD?

[everpeace]

I would suggest putting some note on openmpi package README which recommends using MPIJob instead of openmpi package with an explicit description of MPIJob advantages (retry behavior etc.).

[chrisheecho]

Hello all, could someone give status update of this issue? would love to connect with the driver for this..is this you @everpeace and @jiezhang ?

[jlewi]

I filed kubeflow/website#272 to update the docs.

#1859 to remove the existing package.

[everpeace]

Thanks for creating issues 🙇