-
Notifications
You must be signed in to change notification settings - Fork 202
RuntimeError: cuda runtime error (11) : invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp:383 #7
Comments
I'm going to need more information to help with this. It seems to be a problem with multi-gpu. Can you share your setup? What is your version of pytorch. What is your computer configuration? What are you trying to achieve? |
Closing this due to inactivity |
@tano297 I also got this error message it looks like. I am running two GeForce RTX 2080 Ti on an SLI bridge, running Ubuntu 16.04. My version of pytorch is 1.1.0. I've double checked all my versions with your requirements doc and they all match. I'm just trying to run the data to get the training pipeline to work on my computer. I'll attach the output from running the program below. Thanks for sharing your work, this is everything on PRBonn is super cool! Input:
Output:
|
I think this is related to this issue in turing gpus. |
Sorry, had a bit of a saga. I think it is training now, but there's no tensorboard output. I'm going to let it run for a while. I had this going shortly after you suggested installing pytorch from source (using your advice), but wasn't getting the tensorboard output (it just says there's no dashboards for the current data set). Fiddling around with it I had one run where the log parameter in the ./train.py command was directed at the ~ directory. It deleted the .bashrc file and messed with several other things. Not sure what all happened, but getting all the drivers and libraries in the right place again took a little time. I still don't see any tensorboard output, but I'll let this run for a day. Using nvidia-smi there's a python3 process taking most of the memory usage on both cards, so I'm pretty sure it's going. Any guesstimate on how long the training will take on 2x2080 RTX Ti's (1 hour, 1 day, 1 week...)? Any idea why there's no Tensorboard output (the tb directory doesn't have any files in it)?
|
Met the same issue, but
will do the trick. |
Had the same issue on Ubuntu 18.04 with Nvidia GTX 1660 Ti and cuda 10.0 installed. Removing the version from torch and torchvision packages in the requirements.txt, solved the problem for me |
Train command:
./train.py -d ~/dev-env/dataset/auto_driver/semantic_kitti/data_odometry_labels/dataset/ -ac ./config/arch/darknet21.yaml -l log
The code can run!
Train command:
./train.py -d ~/dev-env/dataset/auto_driver/semantic_kitti/data_odometry_labels/dataset/ -ac ./config/arch/darknet53.yaml -l log
The code can not run!
Error message:
Traceback (most recent call last): File "./train.py", line 115, in <module> trainer.train() File "../../tasks/semantic/modules/trainer.py", line 237, in train show_scans=self.ARCH["train"]["show_scans"]) File "../../tasks/semantic/modules/trainer.py", line 318, in train_epoch output = model(in_vol, proj_mask) #[2, 5, 64, 2048]; [2 , 64, 2048] -> [2, 20, 64, 2048] File "/home/amax-syg/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__ result = self.forward(*input, **kwargs) File "/home/amax-syg/.local/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward outputs = self.parallel_apply(replicas, inputs, kwargs) File "/home/amax-syg/.local/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)]) File "/home/amax-syg/.local/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 83, in parallel_apply raise output File "/home/amax-syg/.local/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 59, in _worker output = module(*input, **kwargs) File "/home/amax-syg/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__ result = self.forward(*input, **kwargs) File "../../tasks/semantic/modules/segmentator.py", line 151, in forward y, skips = self.backbone(x) File "/home/amax-syg/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__ result = self.forward(*input, **kwargs) File "../..//backbones/darknet.py", line 167, in forward x, skips, os = self.run_layer(x, self.conv1, skips, os) File "../..//backbones/darknet.py", line 150, in run_layer y = layer(x) File "/home/amax-syg/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__ result = self.forward(*input, **kwargs) File "/home/amax-syg/.local/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 338, in forward self.padding, self.dilation, self.groups) RuntimeError: cuda runtime error (11) : invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp:383
The text was updated successfully, but these errors were encountered: