How to do weight normalization at last layer

[dwy927]

❓ Questions and Help

Please note that this issue tracker is not a help form and this issue will be closed.

We have a set of listed resources available on the website. Our primary means of support is our discussion forum:

• Discussion Forum
  https://discuss.pytorch.org/t/how-to-do-weight-normalization-in-classification-layer/35193?u=wenying_dai

Many loss functions in face recognition will norm feature or norm weight before calculate softmax loss, such as normface(https://arxiv.org/abs/1704.06369), L2-softmax(https://arxiv.org/abs/1703.09507) and so on.

I’d like to know how to norm weight in the last classification layer.

self.feature =  torch.nn.Linear(7*7*64, 2) # Feature extract layer
self.pred = torch.nn.Linear(2, 10, bias=False) # Classification layer

I want to replace the weight parameter in self.pred module with a normalized one.
In another word, I want to replace weight in-place, like this:

self.pred.weight = self.pred.weight / torch.norm(self.pred.weight, dim=1, keepdim=True)

When I trying to do this, there is something wrong:

TypeError: cannot assign 'torch.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)

I am new comer to pytorch, I don’t know what is the standard way to handle this. Thanks a lot!

[vadimkantorov]

I think, any operation on torch.nn.Parameter type will return a torch.Tensor, so you need to manually wrap it again in torch.nn.Parameter before re-assigning. Maybe try:

self.pred.weight = torch.nn.Parameter(self.pred.weight / torch.norm(self.pred.weight, dim=1, keepdim=True))

[dwy927]

@vadimkantorov Thanks a lot. Actually I want the norm of weight matrix equals one for every class. So this way will help, am I right?

Besides, how I can get the weight matrix value before the normalization and after the normalization?

A more detailed question links here: https://discuss.pytorch.org/t/how-to-do-weight-normalization-in-last-classification-layer/35193

[yf225]

@dwy927 @vadimkantorov we should move the discussion to https://discuss.pytorch.org/t/how-to-do-weight-normalization-in-last-classification-layer/35193/6 because the issue tracker is mainly for bug reports and feature requests

[jaideep11061982]

I think, any operation on torch.nn.Parameter type will return a torch.Tensor, so you need to manually wrap it again in torch.nn.Parameter before re-assigning. Maybe try:

self.pred.weight = torch.nn.Parameter(self.pred.weight / torch.norm(self.pred.weight, dim=1, keepdim=True))

how about torch.nn.Parameter(F.normalize(self.pred.weight) )