F.normalize NaN gradient

[elbaro]

0.2.0+dc6510f

F.normalize(p=1) gives NaN gradients.
Manually dividing by the sum works.

Code

import torch
from torch.autograd.variable import Variable
import torch.nn.functional as F

a = Variable(torch.FloatTensor([[0,0,0,0,0.1996]]),requires_grad=True)
# b = F.normalize(a, p=1, dim=1)  # nan nan nan nan 0
# b = a / a.sum().clamp(min=1e-12)  # 0 0 0 0 0
b.sum().backward()

print(a.grad.data)

Output

nan nan nan nan 0

or

0 0 0 0 0

[albanD]

Hi,
The problem is that you use the norm 1, not the sum in this case (there is an absolute value difference).
The problem is that the current implementation of the norms with p < 1 has a nan gradient at 0 (I guess you were expecting it to return the subgradient of 0).
@soumith should we give this a similar treatment as #2421 ?

[soumith]

@albanD yes we should

[albanD]

@colesbury should I do this now or do you prefer to include this in the refactor for #3275 ?