RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [68]] is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True). #19853
Labels
Bug description
Traceback (most recent call last):
File "main_train.py", line 61, in
main(cfg)
File "main_train.py", line 50, in main
trainer.fit()
File "/mnt/inais/data1/syp/wgan/fabric/decalib/trainer.py", line 373, in fit
self.fabric.backward(all_loss)
File "/root/miniconda3/envs/DECA_2/lib/python3.7/site-packages/lightning/fabric/fabric.py", line 359, in backward
self._precision.backward(tensor, module, *args, **kwargs)
File "/root/miniconda3/envs/DECA_2/lib/python3.7/site-packages/lightning/fabric/plugins/precision/precision.py", line 73, in backward
tensor.backward(*args, **kwargs)
File "/root/miniconda3/envs/DECA_2/lib/python3.7/site-packages/torch/_tensor.py", line 396, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/root/miniconda3/envs/DECA_2/lib/python3.7/site-packages/torch/autograd/init.py", line 175, in backward
allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [68]] is at version 3; expected version 2 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
I encountered this problem during operation, how should I solve it?
What version are you seeing the problem on?
master
How to reproduce the bug
No response
Error messages and logs
Environment
Current environment
More info
No response
The text was updated successfully, but these errors were encountered: