Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED #19

Closed
AhsanBilal7 opened this issue Jul 17, 2023 · 1 comment
Closed

RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED #19

AhsanBilal7 opened this issue Jul 17, 2023 · 1 comment

Comments

@AhsanBilal7
Copy link

I have cuDNN installed but still I am getting this error

File "D:\FYP\GaitGraph2-main\GaitGraph2-main\GaitGraph\gaitgraph_casia_b.py", line 300, in
cli_main()
File "D:\FYP\GaitGraph2-main\GaitGraph2-main\GaitGraph\gaitgraph_casia_b.py", line 290, in cli_main
LightningCLI(
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\cli.py", line 353, in init
self._run_subcommand(self.subcommand)
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\cli.py", line 642, in _run_subcommand
fn(**fn_kwargs)
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\trainer\trainer.py", line 529, in fit
call._call_and_handle_interrupt(
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\trainer\call.py", line 42, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\trainer\trainer.py", line 568, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\trainer\trainer.py", line 973, in _run
results = self._run_stage()
^^^^^^^^^^^^^^^^^
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1014, in _run_stage
self._run_sanity_check()
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1043, in _run_sanity_check
val_loop.run()
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\loops\utilities.py", line 177, in _decorator
return loop_run(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\loops\evaluation_loop.py", line 115, in run
self._evaluation_step(batch, batch_idx, dataloader_idx)
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\loops\evaluation_loop.py", line 375, in _evaluation_step
output = call._call_strategy_hook(trainer, hook_name, *step_kwargs.values())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\trainer\call.py", line 291, in _call_strategy_hook
output = fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\pytorch_lightning\strategies\strategy.py", line 379, in validation_step
return self.model.validation_step(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\FYP\GaitGraph2-main\GaitGraph2-main\GaitGraph\gaitgraph_casia_b.py", line 92, in validation_step
y_hat = self(x)
^^^^^^^
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\FYP\GaitGraph2-main\GaitGraph2-main\GaitGraph\gaitgraph_casia_b.py", line 79, in forward
return self.backbone(x)[0]
^^^^^^^^^^^^^^^^
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\FYP\GaitGraph2-main\GaitGraph2-main\GaitGraph\models\ResGCNv1\nets.py", line 65, in forward
x_cat.append(branch(x[:, i]))
^^^^^^^^^^^^^^^
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\FYP\GaitGraph2-main\GaitGraph2-main\GaitGraph\models\ResGCNv1\nets.py", line 24, in forward
x = self.bn(x)
^^^^^^^^^^
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\modules\batchnorm.py", line 171, in forward
return F.batch_norm(
^^^^^^^^^^^^^
File "C:\Users\Pc\AppData\Local\Programs\Python\Python311\Lib\site-packages\torch\nn\functional.py", line 2450, in batch_norm
return torch.batch_norm(
^^^^^^^^^^^^^^^^^
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

@AhsanBilal7
Copy link
Author

Solved !!!!

def force_cudnn_initialization():
    s = 32
    dev = torch.device('cuda')
    torch.nn.functional.conv2d(torch.zeros(s, s, s, s, device=dev), torch.zeros(s, s, s, s, device=dev))

call the function in the start of main function

if __name__ == "__main__":
    force_cudnn_initialization()
    ---- rest of code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant