Problem of the _calculate_2nd_gradient

[alibaba-yiwuyao]

When I run the cream nas with pytorch==1.7.1, problem appears:

File "/app/nni_lib/trainer.py", line 294, in _calculate_2nd_gradient
grad_outputs=grad_student_val,
File "/opt/conda/lib/python3.7/site-packages/torch/autograd/init.py", line 192, in grad
inputs, allow_unused)
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [3136, 1]], which is output 0 of TBackward, is at version 5; expected version 4 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

The second grad calculation may have some problem?

grad_teacher = torch.autograd.grad(
students_weight[0],
self.model.rand_parameters(teacher_cand, self.pick_method=='meta'),
grad_outputs=grad_student_val,
)

[Z7zuqer]

Hi,

This error is the result of using different pytorch version. If you install pytorch=1.2 and apex, the code could run without error.

Best,
Hao.

[alibaba-yiwuyao]

Pytorch 1.2 is OK for this issue, thanks:)