can't change requires_grad for non leaf variables

[glample]

In my code I have a target y which is computed using some Linear and Conv2d modules, and I can't do y.requires_grad = False before I pass it to a loss module.

Problem comes from:

pytorch/torch/autograd/variable.py

Lines 39 to 41 in 93d02e4

	if self.creator is not None:
	raise RuntimeError("you can only change requires_grad flags of "
	"leaf variables")

Is there any particular reason why this restriction was implemented?

Also I was wondering what would happen if we have something like:
x -> var1 -> var2 -> var3 -> var_loss

and we call loss.backward() when every variable requires a gradient, apart from var2, for instance? Would it be like if var1.requires_grad is also set to False?

[apaszke]

Yeah, so the reason why I locked changing requires_grad for non-leaf variables is that it's quite tricky to solve properly. There are two ways I though of, and both have problems:

1. If you want to disable the gradient flow in the backward, it's not enough to do this. You also have to modify the requires_grad flag of the function that created the variable. In your example it would effectively block var1 and x from receiving gradients, which seems to be an error. If you didn't care about their gradients in the first place, you could have marked them so.
2. If you only want to mark that you don't care about the gradient of a variable (a single graph branch), without changing the flag of the creator function, it will likely throw an error in the backward phase, as the following function will see that its input didn't require gradient and will simply return None.

We might be able to get around 2. by converting Nones to correctly sized zero-filled gradient tensors in the backward.

[glample]

But having zero-filled gradient will create unnecessary computations, no? What about just making a copy of the variable we don't want to backpropagate through, so that it becomes a leaf that does not require gradient? If we do this for var2 in the example, then var1 won't receive gradients, although it will be flagged as requires_grad = True, but I guess this is fine, isn't it?

[apaszke]

I guess it depends on what do you call an unnecessary computation. Of course, if you're sure you don't want to detach the Variable it will work, and will prevent computation of its predecessors, however it won't encode the data dependencies properly, which might carry a performance penalty when using lazy evaluation.

As I said, it's only a problem if the creator had multiple outputs, and you still want to backprop through some of them. We should probably have some function that allows you to "detach" the variable without accessing the .data, while still retaining the data dependencies. We'll discuss it today.

[apaszke]

Fixed in #177. You can now get a non-requiring gradient variable by calling var.no_grad().

[Waki7]

Yeah, so the reason why I locked changing requires_grad for non-leaf variables is that it's quite tricky to solve properly. There are two ways I though of, and both have problems:

1. If you want to disable the gradient flow in the backward, it's not enough to do this. You also have to modify the requires_grad flag of the function that created the variable. In your example it would effectively block var1 and x from receiving gradients, which seems to be an error. If you didn't care about their gradients in the first place, you could have marked them so.
2. If you only want to mark that you don't care about the gradient of a variable (a single graph branch), without changing the flag of the creator function, it will likely throw an error in the backward phase, as the following function will see that its input didn't require gradient and will simply return None.

We might be able to get around 2. by converting Nones to correctly sized zero-filled gradient tensors in the backward.

I apologize for bringing up this old issue, but what if you did want to stop the gradient flow. Say you have an RNN with a constant window of overlapping inputs. Essentially where as you do a new forward pass, you want to cut off the the tail of the back prop off.

[dorcoh]

Hey @apaszke , about the case of freezing single weights (i.e., elements of torch.tensor contained in torch.nn.parameter.Parameter), is that possible following your merge? Thanks

[MohammadJavadD]

I'm getting a similar error when I use PyTorch to train an HF model Here

[mostafaelhoushi]

In case someone is looking into it, this might solve the problem:

y = y.detach()