nn.linear module weight initialization fix

[interesaaat]

This PR fixes the module weight initialization in nn.Linear by matching what state in the module documentation.

Fixes #19376.

[ezyang]

@pytorchbot rebase this please

[ezyang]

Thanks for the PR. Can we please get a test for this?

[interesaaat]

Thanks for the PR. Can we please get a test for this?

Let me look into this. (Any hint on where the test should go?)

[soumith]

@interesaaat i took a closer look. Mathematically, init.kaiming_uniform_(self.weight, a=math.sqrt(5)) is doing the same as a uniform with 1.0 / std::sqrt(weight.size(1)), from what I can tell.

If a is math.sqrt(5), then kaiming init's bound becomes: sqrt(6 / (6 * weight.size(1)) which is 1.0 / sqrt(weight.size(1)). They are both exactly doing the same thing.

[interesaaat]

I still believe that this is sort of confusing (using kaiming_uniform for initializing the weight in the linear module, but not the bias, nor the weight in the bilinear module).