Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updates to Kind with GPUs setup #21

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

rohanarora
Copy link
Contributor

Thomas (@tdoublep), As a result of this development, the use of kind clusters with GPUs has been streamlined. The use of the forked version of kind can now be avoided.

But it requires the prerequisite steps to be done and those need sudo privileges. Should we consider switching to this approach?

@tdoublep
Copy link
Collaborator

@rohanarora We did have a valid reason for wanting to be able to set it up without sudo, because we did not have sudo on a bunch of the machines @mal-zurich was using for her experiments.

that said, looking at the pre-requisite steps here, I think this stuff is probably already there on those machines and if not, we could ask it to be setup. and it is indeed better to not be using a fork of kind.

--set driver.enabled=false \
--set mig.strategy=none \
--version 23.3.1
--set driver.enabled=false
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure it is OK to remove:

--set mig.strategy=none \
--version 23.3.1

Missing these two options caused many problems for us in the past.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It has worked without those options for me in both MIG and non-MIG environments on different machines.
Ir-respective for environments which have GPUs with MIG enabled, the key seems to be to create GPU MIG instances prior to installing the operator other-wise the CUDA validator fails.

docs/SETUP.md Outdated Show resolved Hide resolved
docs/SETUP.md Outdated Show resolved Hide resolved
@rohanarora
Copy link
Contributor Author

Made the updates, Thomas (@tdoublep). Let me know how you wish to proceed on this one.

```

And verify the installation:
```shell
$ which kind
~/bin/kind
/usr/local/bin/kind
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert to:

~/bin/kind

Copy link
Collaborator

@tdoublep tdoublep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants