x.grad should be 0 but get NaN after x/0

[DIYer22]

x.grad should be 0 but get NaN after x/0

Reproduction BUG code

import torch
from torch.autograd import Variable

x = Variable(torch.FloatTensor([1.,1]), requires_grad=True)
div = Variable(torch.FloatTensor([0.,1]))

y = x/div # => y is [inf, 1]

zero_mask = (div==0) # => zero_mask is [1, 0]
y[zero_mask] = 0  # => y is [0, 1]

loss = y.sum()
loss.backward()
print(x.grad) # grad is [nan, 1], but expected [0, 1]

Computational graph of loss not include x[0]
So, gradient of x[0] should be 0, but get NaN

more simple reproduction

x = Variable(torch.FloatTensor([1.,1]), requires_grad=True)
div = Variable(torch.FloatTensor([0.,1]))

y = x/div # => y is [inf, 1]

mask = (div!=0) # => mask is [0, 1]
loss = y[mask]

loss.backward()
print(x.grad) # grad is [nan, 1], but expected [0, 1]

Versions:

• Python: 2.7
• pyTorch: 0.3.0.post4

cc @svekars @holly1238 @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @lezcano @Varal7 @brianjo @mruberry @gchanan @bdhirsh @jbschlosser @anjali411 @jlin27

[zou3519]

I think the reason why this is happening is that the backwards pass for indexing (y[mask]) returns a tensor that has 0's in it for the masked-out indices. 0 * float('inf') gives nan, which is why the nan is showing up instead of 0.

[DIYer22]

@zou3519 I argee that reason,
But, is it a BUG? because Index should do select rather than multiplication a mask, and x[0] should not be in computational graph

[DIYer22]

Key to avoid nan

Your code should't generat any inf in forward, which often produce by torch.log(0) and x/[0, ]
That means 0 should be filtered before do torch.log(x) and x/div

My example

Variables

└─ /: 4
    ├── gtind: torch.Size([1, 2, 300, 400]) torch.cuda.FloatTensor
    ├── edge: torch.Size([1, 300, 400]) torch.cuda.ByteTensor
    ├── probnb: torch.Size([1, 8, 2, 300, 400]) torch.cuda.FloatTensor
    └── gtdf: torch.Size([1, 8, 300, 400]) torch.cuda.FloatTensor
th = torch
tots = lambda x:x.data

code(before)

    otherSideEdgeLossMap = -th.log(((probnb*gtind).sum(-3)*gtdf).sum(-3)/gtdf.sum(-3))
    otherSideEdgeLossMap[~tots(edge)] = 0

code(after)

    numerator = ((probnb*gtind).sum(-3)*gtdf).sum(-3)
    numerator[tots(edge)] /= gtdf.sum(-3)[tots(edge)]
    numerator[tots(edge)] = -th.log(numerator[tots(edge)])
    otherSideEdgeLossMap = (numerator)
    otherSideEdgeLossMap[~tots(edge)] = 0

Both code(before) and code(after) has total same otherSideEdgeLossMap.
After otherSideEdgeLossMap[edge].mean().backward().
the simple one code(before)'s grad has lots of nan

After many trys, code(after) could get the right grad !!

In my opinion,It's a BUG, because index operation should totally isolate the gradient of tensors which are not be indexed!

[pranerd]

@DIYer22 Did you release your complete code about this project on github?

[DIYer22]

@pranerd I didn't, The project is dirty code right now

[pranerd]

@DIYer22 will you release it after you finish it?

[yawudede]

my solution beyond the owner's advice

import torch
from torch.autograd import Variable
x = Variable(torch.FloatTensor([1.,1]), requires_grad=True)
div = Variable(torch.FloatTensor([0.,1]))
numerator=Variable(torch.FloatTensor([0.,1]))
one_mask = (div!=0) # => one_mask is [0, 1]
numerator[one_mask]=x[one_mask] /div[one_mask] 
y=(numerator)
zero_mask = (div==0) # => zero_mask is [1, 0]
y[zero_mask] = 0  # => y is [0, 1]
print(y.data)
loss = y.sum()
loss.backward()
print(x.grad) # grad is [0, 1], as expected [0, 1] #打印梯度 dy/dx`

[Rpersie]

If you add a small value in the zero items, this will happen? such as 1e-30

[zou3519]

I think this theme (getting NaNs) comes up a lot. Maybe we should have some documentation around NaN best practices, if we don't have that already.

[Rpersie]

I think this theme (getting NaNs) comes up a lot. Maybe we should have some documentation around NaN best practices, if we don't have that already.

if I add a eps=1e-15, this problem nan caused by x/0 will be avoided ?? Thank you !

[chengxuanying]

This bug is actually not a bug, 1/0=inf, the easy solution is to use eps instead of 0

[saluto]

It is a bug. The fact that in pytorch 1/0=inf (which is, by the way, mathematically wrong) has nothing to do with the fact, that y is independent of x[0], thus y's gradient w.r.t. x[0] should be 0. Masking does not solve the bug, just avoids its occurrence if done right. It is a pain, inefficient and should not be necessary. Please, dear pytorch team, fix this. Thank you!

---

Another example:

import torch


a = torch.tensor([1.], requires_grad=True)
b = a ** 2
b.backward()
# `a.grad` is `2` as expected
print(a.grad)

a = torch.tensor([1.], requires_grad=True)
b = a ** 2
b = b.clone()
b[0] = 0
b.backward()
# `a.grad` is `0` as expected
print(a.grad)


a = torch.tensor([0.], requires_grad=True)
b = a.sqrt()
b.backward()
# `a.grad` is `inf` as expected
print(a.grad)

a = torch.tensor([0.], requires_grad=True)
b = a.sqrt()
b = b.clone()
b[0] = 0
b.backward()
# `a.grad` is `nan` but expected `0`
print(a.grad)

[ezyang]

This still reproduces on master

[ppwwyyxx]

The repro at the top looks the same as #15506 and #12986. Would be difficult to solve properly unless there is a way for autograd to distinguish "zero gradient" and "no gradient" on a per-element basis.

[zou3519]

At the triage review meeting, we decided: this is expected behavior (for now at least). It is a legitimate request to distinguish "zero gradient" and "no gradient". However, we may need a first-class concept of masking in PyTorch to accomplish this, so this won't happen anytime soon.

[george-qi]

Hi all, just wanted to surface some attention for MaskedTensor, a tensor extension library that we just launched the prototype for along PyTorch 1.11. It should help solve many existing problems related to masked semantics, distinguishing between 0 and undefined gradients, etc.

e.g. for this example:

x = torch.tensor([1., 1.], requires_grad=True)
div = torch.tensor([0., 1.])
y = x/div # => y is [inf, 1]

mask = (div != 0) # => mask is [0, 1]
loss = as_masked_tensor(y, mask)
loss = loss.sum()
loss.backward()

x.grad
masked_tensor(
  [      --,   1.0000]
)

For more details, you can check out the website or Github. We're actively looking for new users, so if you're interested in using it or have any suggestions on things we should build out, please reach out to me or @cpuhrsch. Thanks!!