Grad is None after using view

[sdaulton]

🐛 Bug

After initializing a tensor with requires_grad=True, applying a view, summing, and calling backward, the gradient is None. This is not the case if the tensor is initialized using the dimensions specified in the view.

To Reproduce

import torch

# test without view
X = torch.tensor([[[0.25]],[[ 0.75]]],requires_grad=True,)
print(f"X.shape: {X.shape}")
X.sum().backward()
print(f"X.grad: {X.grad}")

# test with view
X_view = torch.tensor([0.25, 0.75], requires_grad=True,).view(2, 1, 1)
print(f"X_view.shape: {X_view.shape}")
X_view.sum().backward()
print(f"X_view.grad: {X_view.grad}")
print(f"X_view.grad is None: {X_view.grad is None}")

Output

X.shape: torch.Size([2, 1, 1])
X.grad: tensor([[[1.]],

        [[1.]]])
X_view.shape: torch.Size([2, 1, 1])
X_view.grad: None
X_view.grad is None: True

Expected behavior

Grad is not None and X.grad is the same as X_view.grad

Environment

Collecting environment information...
PyTorch version: 1.0.0a0
Is debug build: No
CUDA used to build PyTorch: 9.2.88

OS: CentOS Linux 7 (Core)
GCC version: (GCC) 4.8.5 20150623 (Red Hat 4.8.5-28)
CMake version: Could not collect

Python version: 3.6
Is CUDA available: No
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA

Versions of relevant libraries:
[pip] Could not collect
[conda] Could not collect

[sdaulton]

So this isn't a bug per se, but it is definitely a source of confusion. The issue with the above code is that the gradient information is attached to the initial tensor before the view, but not the viewed tensor. Performing the initialization and view operation before assigning the tensor to the variable results in losing the access to the gradient information. Splitting out the view works fine. It would be useful to call this out in the docs (maybe I missed this).

X0 = torch.tensor([0.25, 0.75], requires_grad=True,)
X_view = X0.view(2, 1, 1)
print(f"X_view.shape: {X_view.shape}")
X_view.sum().backward()
print(f"X_view.grad: {X_view.grad}")
print(f"X_view.grad is None: {X_view.grad is None}")
print(f"X0.grad: {X0.grad}")

Output:

X_view.shape: torch.Size([2, 1, 1])
X_view.grad: None
X_view.grad is None: True
X0.grad: tensor([1., 1.])

[fmassa]

You analysis is correct: we only retain gradients of leaf variables. Once you do tensor.view(-1), this returns a new tensor which is not a leaf variable anymore (and has a grad_fn).

Where do you think we could improve the documentation to explain this? Maybe in the https://pytorch.org/docs/stable/notes/autograd.html or in https://pytorch.org/docs/stable/notes/faq.html

Also, could you send a PR improving the documentation?

[kit1980]

I think this "only retain gradients of leaf variables" should be in the FAQ.
Just saw a StackOverflow question about this: https://stackoverflow.com/questions/55942423/pytorch-backpropagating-from-sum-of-matrix-elements-to-leaf-variable

[albanD]

We added a warning for this special case. Please re-open if you need more.