Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RuntimeError: No such operator aten::cudnn_convolution_backward_weight #97

Open
fancyshirt opened this issue Apr 15, 2022 · 1 comment

Comments

@fancyshirt
Copy link

fancyshirt commented Apr 15, 2022

I'm trying to create a new model by myself by using the following command:

python train.py --outdir=training-runs --data=https://data-efficient-gans.mit.edu/datasets/100-shot-obama.zip --gpus=1

And I ran into the Error:

return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] Traceback (most recent call last): File "F:\USB Backup\Clemson University\Course\Artificial Intellengent\Project\data-efficient-gans-master\data-efficient-gans-master\DiffAugment-stylegan2-pytorch\train.py", line 554, in main() # pylint: disable=no-value-for-parameter File "C:\Users\Dinner.conda\envs\pytorch\lib\site-packages\click\core.py", line 1130, in call return self.main(*args, **kwargs) File "C:\Users\Dinner.conda\envs\pytorch\lib\site-packages\click\core.py", line 1055, in main rv = self.invoke(ctx) File "C:\Users\Dinner.conda\envs\pytorch\lib\site-packages\click\core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "C:\Users\Dinner.conda\envs\pytorch\lib\site-packages\click\core.py", line 760, in invoke return _callback(*args, **kwargs) File "C:\Users\Dinner.conda\envs\pytorch\lib\site-packages\click\decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "F:\USB Backup\Clemson University\Course\Artificial Intellengent\Project\data-efficient-gans-master\data-efficient-gans-master\DiffAugment-stylegan2-pytorch\train.py", line 547, in main subprocess_fn(rank=0, args=args, temp_dir=temp_dir) File "F:\USB Backup\Clemson University\Course\Artificial Intellengent\Project\data-efficient-gans-master\data-efficient-gans-master\DiffAugment-stylegan2-pytorch\train.py", line 398, in subprocess_fn training_loop.training_loop(rank=rank, **args) File "F:\USB Backup\Clemson University\Course\Artificial Intellengent\Project\data-efficient-gans-master\data-efficient-gans-master\DiffAugment-stylegan2-pytorch\training\training_loop.py", line 284, in training_loop loss.accumulate_gradients(phase=phase.name, real_img=real_img, real_c=real_c, gen_z=gen_z, gen_c=gen_c, sync=sync, gain=gain) File "F:\USB Backup\Clemson University\Course\Artificial Intellengent\Project\data-efficient-gans-master\data-efficient-gans-master\DiffAugment-stylegan2-pytorch\training\loss.py", line 79, in accumulate_gradients loss_Gmain.mean().mul(gain).backward() File "C:\Users\Dinner.conda\envs\pytorch\lib\site-packages\torch_tensor.py", line 363, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "C:\Users\Dinner.conda\envs\pytorch\lib\site-packages\torch\autograd_init.py", line 173, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass File "C:\Users\Dinner.conda\envs\pytorch\lib\site-packages\torch\autograd\function.py", line 253, in apply return user_fn(self, *args) File "F:\USB Backup\Clemson University\Course\Artificial Intellengent\Project\data-efficient-gans-master\data-efficient-gans-master\DiffAugment-stylegan2-pytorch\torch_utils\ops\conv2d_gradfix.py", line 133, in backward grad_weight = Conv2dGradWeight.apply(grad_output, input) File "F:\USB Backup\Clemson University\Course\Artificial Intellengent\Project\data-efficient-gans-master\data-efficient-gans-master\DiffAugment-stylegan2-pytorch\torch_utils\ops\conv2d_gradfix.py", line 145, in forward op = torch._C._jit_get_operation('aten::cudnn_convolution_backward_weight' if not transpose else 'aten::cudnn_convolution_transpose_backward_weight') RuntimeError: No such operator aten::cudnn_convolution_backward_weight

My Cuda version:
nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2021 NVIDIA Corporation Built on Sun_Mar_21_19:24:09_Pacific_Daylight_Time_2021 Cuda compilation tools, release 11.3, V11.3.58 Build cuda_11.3.r11.3/compiler.29745058_0

Pytorch version:
1.11.0

How should I fix it?

@zsyzzsoft
Copy link
Collaborator

Maybe you need to use an earlier PyTorch version like 1.4.0...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants