nn.init.eye_ sets requires_grad to False on tensors

[zou3519]

repro:

import torch.nn as nn
L = nn.Linear(5,5)
nn.init.eye_(L.weight)
print(L.weight.requires_grad)  # False

Initially reported from https://discuss.pytorch.org/t/since-0-4-using-nn-init-eye-disables-gradient/19999/2

[bhavesh2109]

Hey,
All the initialisation methods set the requires_grad to False. All the initialisers return the tensors with the torch.no_grad method. Is this intended?
https://github.com/pytorch/pytorch/blob/dc0faab18dd5296b66dfb00776253062def87fdf/torch/nn/init.py

[soumith]

All the initialisation methods set the requires_grad to False

It's not all, afaik.

[bhavesh2109]

It's not all, afaik.

Hey Soumith, all the initialisers in the link(https://github.com/pytorch/pytorch/blob/dc0faab18dd5296b66dfb00776253062def87fdf/torch/nn/init.py) seem to use the torch.no_grad method before returning. Please correct me if I'm wrong. Thank you!

[vishwakftw]

Operations done in context of no_grad are not recorded in the autograd history for computing the gradients. This will not affect the initialization.

>>> import torch
>>> a = torch.randn(3, 4)
>>> torch.nn.init.normal_(a)
tensor([[ 1.6385,  1.0863, -1.3725, -0.9510],
        [-1.5241, -0.8865, -0.0878, -0.9455],
        [-0.3817, -0.1652, -1.2502, -0.1927]])
>>> a = torch.randn(3, 4, requires_grad=True)
>>> torch.nn.init.normal_(a)
tensor([[ 0.9640,  1.4952, -0.1739, -0.4024],
        [ 0.7667,  1.1724,  0.2663, -0.7854],
        [ 0.7183,  0.1480,  1.0115,  0.1128]], requires_grad=True)

[bhavesh2109]

@vishwakftw : Thanks for the explanation!