RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

[andy-96]

Hi all,

when training the model, I get the following error when Pytorch's anomaly detection is on

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [4, 64, 10, 25]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

The traceback is the following:

[W python_anomaly_mode.cpp:104] Warning: Error detected in ReluBackward0. Traceback of forward call that caused the error:
  File "tools/train.py", line 159, in <module>
    main()
  File "tools/train.py", line 155, in main
    meta=meta)
  File "/home/<user>/Code/conditional-lane-detection/mmdet/apis/train.py", line 167, in train_detector
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
  File "/home/<user>/.conda/envs/conditional-lane-detection/lib/python3.7/site-packages/mmcv/runner/runner.py", line 383, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/<user>/.conda/envs/conditional-lane-detection/lib/python3.7/site-packages/mmcv/runner/runner.py", line 282, in train
    self.model, data_batch, train_mode=True, **kwargs)
  File "/home/<user>/Code/conditional-lane-detection/mmdet/apis/train.py", line 74, in batch_processor
    losses = model(**data)
  File "/home/<user>/.conda/envs/conditional-lane-detection/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/<user>/.conda/envs/conditional-lane-detection/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 166, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/<user>/.conda/envs/conditional-lane-detection/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/<user>/Code/conditional-lane-detection/mmdet/models/detectors/condlanenet.py", line 327, in forward
    return self.forward_train(img, img_metas, **kwargs)
  File "/home/<user>/Code/conditional-lane-detection/mmdet/models/detectors/condlanenet.py", line 344, in forward_train
    output, memory = self.neck(output)
  File "/home/<user>/.conda/envs/conditional-lane-detection/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/<user>/Code/conditional-lane-detection/mmdet/core/fp16/decorators.py", line 49, in new_func
    return old_func(*args, **kwargs)
  File "/home/<user>/Code/conditional-lane-detection/mmdet/models/necks/trans_fpn.py", line 257, in forward
    trans_feat = self.trans_head(src[self.trans_idx])
  File "/home/<user>/.conda/envs/conditional-lane-detection/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/<user>/Code/conditional-lane-detection/mmdet/models/necks/trans_fpn.py", line 153, in forward
    src = layer(src, pos.to(src.device))
  File "/home/<user>/.conda/envs/conditional-lane-detection/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/<user>/Code/conditional-lane-detection/mmdet/models/necks/trans_fpn.py", line 105, in forward
    x = self.pre_conv(x)
  File "/home/<user>/.conda/envs/conditional-lane-detection/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/<user>/.conda/envs/conditional-lane-detection/lib/python3.7/site-packages/mmcv/cnn/bricks/conv_module.py", line 181, in forward
    x = self.activate(x)
  File "/home/<user>/.conda/envs/conditional-lane-detection/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/<user>/.conda/envs/conditional-lane-detection/lib/python3.7/site-packages/torch/nn/modules/activation.py", line 98, in forward
    return F.relu(input, inplace=self.inplace)
  File "/home/<user>/.conda/envs/conditional-lane-detection/lib/python3.7/site-packages/torch/nn/functional.py", line 1299, in relu
    result = torch.relu(input)
 (function _print_stack)

Anybody knows how to fix this?

[andy-96]

For everybody, who is facing the same problem. This was the error for me:

mmdet/models/necks/trans_fpn.py (Line 108)

        if pos is not None:
            x = x + pos # instead of x += pos
        proj_query = self.query_conv(x).view(m_batchsize, -1,
                                             width * height).permute(0, 2, 1)

[haerrel]

You saved my day!