RuntimeError: CUDA error: no kernel image is available for execution on the device

[abhishek-peri]

Thanks for providing such a library!

I want to use the lietorch library in my project where I'm trying to optimize over the SE3 space. I've installed the required dependencies as stated. However, I'm receiving the below error when I try to run the ./run_test.sh script. So, I wanted to check if anyone can tell what could be the source of this error and how to rectify it?

Note: torch.cuda.get_device_name(0) returns correctly the GPU's name.

Testing lietorch forward pass (CPU) ...                                                                                                                                                                                    
- <class 'lietorch.groups.SO3'> Passed exp-log test                                                                                                                                                               
 - <class 'lietorch.groups.SO3'> Passed inv test                                                                                                                                                                   
 - <class 'lietorch.groups.SO3'> Passed adj test                                                                                                                                                                    
- <class 'lietorch.groups.SO3'> Passed act test                                                                                                                                                                   
 - <class 'lietorch.groups.RxSO3'> Passed exp-log test                                                                                                                                                            
  - <class 'lietorch.groups.RxSO3'> Passed inv test                                                                                                                                                                 
 - <class 'lietorch.groups.RxSO3'> Passed adj test                                                                                                                                                                 
 - <class 'lietorch.groups.RxSO3'> Passed act test                                                                                                                                                                  
- <class 'lietorch.groups.SE3'> Passed exp-log test                                                                                                                                                                
- <class 'lietorch.groups.SE3'> Passed inv test                                                                                                                                                                    
- <class 'lietorch.groups.SE3'> Passed adj test                                                                                                                                                                   
 - <class 'lietorch.groups.SE3'> Passed act test                                                                                                                                                                  
  - <class 'lietorch.groups.Sim3'> Passed exp-log test                                                                                                                                                              
 - <class 'lietorch.groups.Sim3'> Passed inv test                                                                                                                                                                   
- <class 'lietorch.groups.Sim3'> Passed adj test                                                                                                                                                                  
 - <class 'lietorch.groups.Sim3'> Passed act test                                                                                                                                                           
Testing lietorch backward pass (CPU)...                                                                                                                                                                                    
- <class 'lietorch.groups.SO3'> Passed eye-grad test                                                                                                                                                              
 - <class 'lietorch.groups.SO3'> Passed inv-grad test                                                                                                                                                             
  - <class 'lietorch.groups.SO3'> Passed adj-grad test                                                                                                                                                               
- <class 'lietorch.groups.SO3'> Passed adjT-grad test                                                                                                                                                            
  - <class 'lietorch.groups.SO3'> Passed act-grad test                                                                                                                                                              
 - <class 'lietorch.groups.RxSO3'> Passed eye-grad test                                                                                                                                                            
 - <class 'lietorch.groups.RxSO3'> Passed inv-grad test                                                                                                                                                             
- <class 'lietorch.groups.RxSO3'> Passed adj-grad test                                                                                                                                                            
 -
 <class 'lietorch.groups.RxSO3'> Passed adjT-grad test                                                                                                                                                            
- <class 'lietorch.groups.RxSO3'> Passed act-grad test                                                                                                                                                             
- <class 'lietorch.groups.SE3'> Passed eye-grad test                                                                                                                                                               
- <class 'lietorch.groups.SE3'> Passed inv-grad test                                                                                                                                                             
  - <class 'lietorch.groups.SE3'> Passed adj-grad test                                                                                                                                                               
- <class 'lietorch.groups.SE3'> Passed adjT-grad test                                                                                                                                                              
- <class 'lietorch.groups.SE3'> Passed act-grad test                                                                                                                                                               
- <class 'lietorch.groups.Sim3'> Passed eye-grad test                                                                                                                                                             
 - <class 'lietorch.groups.Sim3'> Passed inv-grad test                                                                                                                                                             
 - <class 'lietorch.groups.Sim3'> Passed adj-grad test                                                                                                                                                             
 - <class 'lietorch.groups.Sim3'> Passed adjT-grad test                                                                                                                                                            
 - <class 'lietorch.groups.Sim3'> Passed act-grad test                                                                                                                                                      
Testing lietorch forward pass (GPU) ...                                                                                                                                                                            
Traceback (most recent call last):                                                                                                                                                                                   
File "lietorch/run_tests.py", line 217, in <module>                                                                                                                                                                 
 test_exp_log(Group, device='cuda')                                                                                                                                                                               
File "lietorch/run_tests.py", line 19, in test_exp_log                                                                                                                                                              
 b = Group.exp(a).log()                                                                                                                                                                                           
File "/home/abhishek_peri/anaconda3/envs/nerfing/lib/python3.6/site-packages/lietorch-0.1-py3.6-linux-x86_64.egg/lietorch/groups.py", line 132, in log                                                              
 return self.apply_op(Log, self.data)                                                                                                                                                                             
File "/home/abhishek_peri/anaconda3/envs/nerfing/lib/python3.6/site-packages/lietorch-0.1-py3.6-linux-x86_64.egg/lietorch/groups.py", line 122, in apply_op                                                         
 data = op.apply(cls.group_id, *inputs)                                                                                                                                                                           
File "/home/abhishek_peri/anaconda3/envs/nerfing/lib/python3.6/site-packages/lietorch-0.1-py3.6-linux-x86_64.egg/lietorch/group_ops.py", line 12, in forward                                                        
 out = cls.forward_op(ctx.group_id, *inputs)                                                                                                                                                                   
 RuntimeError: CUDA error: no kernel image is available for execution on the device

[zachteed]

What GPU are you using? Its possible I may need to update the compiler args in setup.py

[abhishek-peri]

Hi,

I'm using 'GeForce GTX TITAN X' as my GPU

[zachteed]

Ok it looks like you will need to use a slightly modified setup.py, but this should work

from setuptools import setup
from torch.utils.cpp_extension import BuildExtension, CUDAExtension

import os.path as osp


ROOT = osp.dirname(osp.abspath(__file__))
print(ROOT)

setup(
    name='lietorch',
    version='0.1',
    description='Lie Groups for PyTorch',
    author='teedrz',
    packages=['lietorch'],
    ext_modules=[
        CUDAExtension('lietorch_backends', 
            include_dirs=[
                osp.join(ROOT, 'lietorch/include'), 
                osp.join(ROOT, 'eigen')],
            sources=[
                'lietorch/src/lietorch.cpp', 
                'lietorch/src/lietorch_gpu.cu',
                'lietorch/src/lietorch_cpu.cpp'],
            extra_compile_args={
                'cxx': ['-O2'], 
                'nvcc': ['-O2',
                    '-gencode=arch=compute_52,code=sm_52', 
                    '-gencode=arch=compute_60,code=sm_60', 
                    '-gencode=arch=compute_61,code=sm_61', 
                    '-gencode=arch=compute_70,code=sm_70', 
                    '-gencode=arch=compute_75,code=sm_75',
                    '-gencode=arch=compute_75,code=compute_75',
                    
                ]
            }),

        CUDAExtension('lietorch_extras', 
            sources=[
                'lietorch/extras/altcorr_kernel.cu',
                'lietorch/extras/corr_index_kernel.cu',
                'lietorch/extras/se3_builder.cu',
                'lietorch/extras/se3_inplace_builder.cu',
                'lietorch/extras/se3_solver.cu',
                'lietorch/extras/extras.cpp',
            ],
            extra_compile_args={
                'cxx': ['-O2'], 
                'nvcc': ['-O2',
                    '-gencode=arch=compute_52,code=sm_52', 
                    '-gencode=arch=compute_60,code=sm_60', 
                    '-gencode=arch=compute_61,code=sm_61', 
                    '-gencode=arch=compute_70,code=sm_70', 
                    '-gencode=arch=compute_75,code=sm_75',
                    '-gencode=arch=compute_75,code=compute_75',
                    
                ]
            }),
    ],
    cmdclass={ 'build_ext': BuildExtension }
)

[abhishek-peri]

Yes, this seems to solve my issue. Thanks for the quick response.

[abhishek-peri]

On a different note, I have seen in other issue that the backward operation for ToMatrix() is yet to be integrated. Is there any tentative timeline for its integration? Thanks.

[zachteed]

Just made an update so the .ToMatrix() function is now differentiable. I'm working on a differentiable FromMatrix function currently