-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can I limit the distributed GPUs? #8
Comments
Hi @bermeitinger-b. If I understand it correctly then |
Thanks, let me clarify. I'm running experiments on a machine that has 16 GPUs. I'm running a lot of tasks and use Fixing which job runs on which GPU with |
I got it. I sometimes got complained about selecting GPUs randomly too. I guess limiting the visibility of GPUs to |
For now you can use |
Hi @bermeitinger-b. I prototyped this feature in this branch. Basically, you can set an env var like |
Thank you very much. I tried the branch and it seems it working exactly as intended. |
Does this new limit also the number of concurrent tasks? My current approach to use GPUs 0-8 with 16 concurrent jobs (so 2 per GPU) would be to
However, I'm not seeing 16 concurrent jobs but only 8. (The jobs are small enough that the 90% full-rate is not reached.) |
Hi @bermeitinger-b. This is intended. |
Hi @bermeitinger-b. You can pull to get the new features. To run multiple processes in a GPU, you can set the free memory threshold (in percentage) appropriately via |
I'm using your script on a machine with 16 GPUs. For my tasks, I want specific GPUs to not be used or rather select which GPUs are used.
For example, I want GPUs 0-8 to be available to
ts
but 9-15 be left alone. Is this something that can be done?The text was updated successfully, but these errors were encountered: