-
Notifications
You must be signed in to change notification settings - Fork 214
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Question] v2 versus v1alpha2 #443
Comments
https://github.com/kubeflow/mpi-operator/blob/master/proposals/scalable-robust-operator.md Perhaps it would be great to have an upgrading guide. /cc @alculquicondor |
Your images need some compatible SSH configuration. You can find a base image here: An upgrade guide would be great, I'm happy to review a PR to the README. |
Is there any plan to put out an upgrade guide? I attempted performing the upgrade myself and tried a few things but was unsuccessful. |
I unfortunately don't have time to do it. Does this help? https://github.com/kubeflow/mpi-operator/pull/416/files#diff-a7f96af68d899e069e558ca372a2a3976a62761ce47e1e90a4f2f281f47690eeL72 |
Greetings! I hope you had a wonderful holiday season :) So in lieu of an upgrade guide, I decided to just try and perform the upgrade and see what happens. I hope by doing this I can ask you more specific, smaller scoped questions as I fiddle my way through this. Currently hitting an error when I use helm to deploy the mpi operator. I am using this chart with some minor innocuous changes to get the docker images from my own private repo. For reference, I am using a build mpi-operator image based on the tag kubeflow/mpi-operator@v0.3.0 I was able to successfully get this CRD created in my env: https://github.com/kubeflow/mpi-operator/blob/a566d1d18046fe7f756b0ec7004693c898d589a0/deploy/v2beta1/mpi-operator.yaml I attempted to deploy with the following command: The mpi-operator comes up and is shown as running in kubernetes, but when I look at the logs, I see this:
In the design docs, it says the service account has been removed for v2, so I'm wondering if I missed a step or did something wrong. Please let me know if you need any other information from me to debug or if I'm doing something obviously incorrect. |
What was removed was the need for the controller to create service accounts. However, the mpi-operator itself still requires permissions that can be given through a service account. Perhaps it would help to have a look at the cluster role: https://github.com/kubeflow/mpi-operator/blob/master/manifests/base/cluster-role.yaml Also, there is an open PR for the helm update, but I haven't had a chance to review it yet: #447 |
Hello again! Quick thank you for all your help @alculquicondor, each interaction we've had has moved me forward so I appreciate your time! I've been able to stand up the V2 mpi-operator to a point where it is waiting for a job. However, When I attempt to start a simple "hello_world" job (which runs a simple 2 layer model using horovod to distribute work) I see the launchers fail over to the back off limit. mpi-operator logs seems pretty normal but when I dump logs from one of the failed launcher I see this:
I hypothesize a few things that could be wrong, but first I think it's important to state that I am currently running k8s version v1.19.7 with no possibility for an upgrade atm. I'm wondering if the lack of ability to use Indexed Jobs is causing problems. In the design doc, it states that "plain pods" could still be used. Is there some flag or config needed in order to make sure this path is taken? Is k8s version v1.19.7 compatible with v2 mpi-operator? Last piece of info I want to give you. In a recent PR, there was a bunch of ssh config stuff added to an example here: https://github.com/kubeflow/mpi-operator/pull/428/files |
I also wanted to add my MpiJobSpec for your reference in case it helps:
Not sure if there is some "new" way mpijobs need to be defined. |
The v2 controller doesn't use Indexed Jobs, although it's my desire that we do that in the future.
It is possible that the worker's sshd is not allowing the connection due to a misconfiguration. You can gather more information about this if you have a look at the worker's logs ( BUT! You are not actually running sshd in your workers, you are running I see you are trying to use |
So I actually have to run sshd from my mpijob yaml? Do I need to do all the mpiuser stuff? |
If you run as root, the mpi operator takes care of that. Unfortunately, if you run as non-root, the challenging part is providing the location of a PID file that the user has permissions to use https://github.com/kubeflow/mpi-operator/blob/master/examples/base/sshd_config |
Okay, so in my worker container, I am indeed running as root. |
Then you can probably remove the command and let mpi-operator set it for you |
Okay, so I tried a bunch of things.
It's strange to me the logs of the launcher say it cannot resolve the host, but inside the worker, the hostname is found, but prompts for a password. |
I think it's possible that this:
is a red-herring. Maybe initially the hostnames are not reachable, but openmpi retries and eventually it connects and a password is required. It's more useful if you take a look at the worker's logs to see why they are rejecting the connections. |
Hello! I had an issue similar to the one in #443 (comment). I migrated from v1alpha2 to v2beta1. When I connect to the launcher with I don't know whether it is because of mpi-operator or any underlying setup like Kubernetes. For reference, I tried this with kind with 2 worker nodes. |
Previous versions don't use SSH, so you wouldn't see connection errors. SSH errors are expected as networking takes some time to setup. But again, OpenMPI implements retries, so it should eventually connect. Unless you are missing some ssh/sshd configuration parameters. |
Thanks @alculquicondor . I think the problem was that I was using a base image that was different from the one in the example base image, and this image contained an old version of OpenMPI. It somehow didn't retried. I used an image that included a newer version. It solved my problem. |
That's interesting, can you share those errors and the parameters? Do you think it's possible that others might run into the same issues and require these parameters? |
I experimented a little bit more. It looks like I was just impatient and I deleted the job without letting it restart enough. My MPI job worked after 3 restarts without setting any MCA parameters. Sorry for misleading you @alculquicondor and others. |
You can use |
/close |
@alculquicondor: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Hello,
I am attempting to port some custom functionality from my previously used v1alpha2 code/fork into v2 as I notice v1alpha2 is being deprecated.
I have read through the code but am struggling to identify all the functional differences added to v2 except for worker statefulsets being converted to plain pods, and OpenSSH support.
Could someone explain the main architectual differences introduced in v2 and the associated implications? Or perhaps point me to issues or Pull Requests that describe the functionality? If I successfully port over my custom logic (for the sake of brevity let's just say it's a no-op), would I be able to point my cluster to the v2 version of mpi-operator and it will "just work"? Or are there configurations I need to take into account besides "pointing" to v2?
The text was updated successfully, but these errors were encountered: