the setting about torch.autograd.grad function

[lynlynlyn]

Thank you for your code.
if i want to run 2-order MAML，should I set create_graph=True?

[dragen1860]

Generally speaking, you should set create_graph for 2nd order gradient.
However, the implementaton in this repo. is the 2nd order version.

[lynlynlyn]

I use pytorch1.0.The default setting create_graph is False.
And I checked the grad on Meta.py line 86 and line116.
grad = torch.autograd.grad(loss, fast_weights)
The require_grads about grad is False.
I am not sure the implementaton in this repo. is the 2nd order version.

[dragen1860]

@lynlynlyn IT's 2nd order implementation.
I think you should digest the paper to understand how to derive 2nd order gradient.
It's not simply set create_graph=True.
Sorry.

[lynlynlyn]

if the inner loop gradients's require_grads=False,where is the 2nd order gradient?

[eugval]

Hello!

Is there a consensus on this? Unless I am missing something, I also think you might need create_graph = True in order to propagate your gradients correctly through the inner gradient.

[Deng-Y]

I also think it needs create_graph = True.
grad = torch.autograd.grad(loss, self.net.parameters())
My understanding is that the line above will give you grad.requires_grad = False. Thus, there is no higher order gradients available. It's a first order MAML.
grad = torch.autograd.grad(loss, self.net.parameters(), create_graph = True)
The line above will give you grad.requires_grad = True.

[dragen1860]

@Deng-Y well, I have not worked on this project a long time. sorry for misleading information. Have you tried setting create_grape=True?

[Deng-Y]

@dragen1860 I only try it on the other project. Not sure if it will improve the performance on this one.