-
Notifications
You must be signed in to change notification settings - Fork 65
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue when discrepancy between available CUDA device at build time / runtime #39
Comments
Hi @timlacroix , when you set Can you try to call For instance: import torch
from pykeops.torch import LazyTensor
def test(data):
neigh_state = LazyTensor(data[None, :, :])
state = LazyTensor(data[:, None, :])
all_distances = ((neigh_state - state) ** 2).sum(dim=2)
return (- all_distances).logsumexp(dim=1)
tensor = torch.randn(10,128).to('cuda:0')
test(tensor) # should run on gpu 0
tensor1 = torch.randn(10,128).to('cuda:1')
print(torch.cuda.device_count())
test(tensor) # should run on gpu 1... without recompiling |
hi, I used CUDA_VISIBLE_DEVICES here to make the problem reproducible. My question is in a set-up where development (and thus compilation) happens on a machine with N gpus and test happens on a machine with M gpus, but sharing the same compilation cache. Couldn't the number of GPUs available at compile time be used in the compiled code hash ? This way, changing the number of GPUs would just force a rebuild, but wouldn't raise an error. |
Maybe @joanglaunes know that better than me, but I think it will not be possible to make the same shared lib working on 2 different system. Why don't you define 2 separated cache folder ? |
ok, a quick solution could be: include the number of gpu and their respected arch in the name of the cache folder. So when you call your code from different node, it will get the sharedlib from the right cache dir |
@bcharlier yes, if that's possible that would be great :) |
is the hostname unique in your case? I mean, is one of those output different on the various nodes : import platform
print(platform.node())
import socket
print(socket.gethostname()) |
both are different on various nodes. I don't know if including the hostname is a good idea. This means using a separate cache folder per machine which the user can do if necessary by just using a random cache at runtime. In my case I would be happy to re-use the same cache for various nodes of the cluster. |
Hello @timlacroix , |
Hey, first off, thanks for the library !
I have had some weird issues today when trying to use a kernel on 'cuda:1' when the kernel was built on a machine with only 2 gpus. I run into this because I use a shared home filesystem (and hence shared .cache folder) on a cluster where I have access to machines with various number of GPUS.
Here is how to reproduce, on a machine with 2 GPUs:
test.py :
run
CUDA_VISIBLE_DEVICES=0 python test.py
. This should build a kernel.then change 'cuda:0' to 'cuda:1' in test.py
run
python test.py
.This fails with error :
invalid Gpu device number. If the number of available Gpus is > 12, add required lines at the end of function SetGpuProps and recompile.
Recompiling is not a great option for me, as I might run different experiments using the same kernel but on machines with different number of available gpus.
The text was updated successfully, but these errors were encountered: