Tensor.add_() returns different values on different runs using some inputs (using CPU).

[sakhar]

🐛 Bug

When using the CPU only (no CUDA), the in-place add function torch.Tensor.add_() returns different values on different calls (with trivial differences) using some inputs. However, torch.Tensor.add() does not.

To Reproduce

1. use this numpy array as an example https://raw.githubusercontent.com/sakhar/pytorch_add_bug/master/values-np.p
   or values-np.p.zip (unzip).

2. try it with add_():

for i in range(10):
    values = pickle.load(open('values-np.p','rb'))
    x = torch.from_numpy(values[0])
    y = torch.from_numpy(values[1])
    print(x.add_(-0.7,y).data.detach().numpy()[-4])

results:

using add_():
-0.09407266
-0.094072655
-0.09407266
-0.094072655
-0.09407266
-0.09407266
-0.094072655
-0.094072655
-0.09407266
-0.09407266

2. with add():

for i in range(10):
    values = pickle.load(open('values-np.p','rb'))
    x = torch.from_numpy(values[0])
    y = torch.from_numpy(values[1])
    print(x.add(-0.7,y).data.detach().numpy()[-4])

results:

using add():
-0.094072655
-0.094072655
-0.094072655
-0.094072655
-0.094072655
-0.094072655
-0.094072655
-0.094072655
-0.094072655
-0.094072655

Environment

 - PyTorch Version: 0.4.1
 - OS: Mac OSX 10.12.6
 - How you installed PyTorch: pip
 - Python version: 2.7
 - CUDA/cuDNN version: No CUDA
 - GPU models and configuration: No CUDA

- Versions of relevant libraries:
[pip] numpy (1.14.5)
[pip] torch (0.4.1)
[pip] torchvision (0.2.1)

cc @ezyang @gchanan @zou3519

[vishwakftw]

The difference observed is in the 7th digit after the decimal point, and is perhaps a floating point issue rather than a functional issue.

[apaszke]

While the difference really is very small it is quite suspicious behaviour, because it shouldn't matter if you read the values in place or not. It's not like floating point math is non-deterministic.

[sakhar]

While the difference really is very small it is quite suspicious behaviour, because it shouldn't matter if you read the values in place or not. It's not like floating point math is non-deterministic.

If fact, I discovered this after observing strange behavior of a NN model. The final result of the model (i.e. acc. and f-1) are really different on different runs and the results are always one of two values (after fixing all seeds)! So even though the difference is tiny, the final effect is not. I also found that in optim.sgd.py, even though I have the same gradient and the values of all parameters (among two different runs), the updated values of the parameters (after adding the gradient) are not the same after some iterations. So again, the final effect of this issue is not trivial.

[sakhar]

I just realized that this behavior does not only depend on a particular element of a tensor, but it depends on the other values in the tensor. Here is an example where I use only the element that its final value (after adding using add_() changes on different runs:

for i in range(10):
    values = pickle.load(open('values-np.p','rb'))
    x = torch.from_numpy(values[0][-4:-3])
    y = torch.from_numpy(values[1][-4:-3])
    print(x.add_(-0.7,y).data.detach().numpy())

[fmassa]

Can you convert the torch tensor to a list before printing, via the .tolist()? Just to factor out that this is not a printing issue

[sakhar]

Can you convert the torch tensor to a list before printing, via the .tolist()? Just to factor out that this is not a printing issue

The same pattern with different precision.

for i in range(10):
    values = pickle.load(open('values-np.p','rb'))
    x = torch.from_numpy(values[0])
    y = torch.from_numpy(values[1])
    x.add_(-0.7,y)
    print 'to_list():', x.tolist()[-4]
    print 'numpy:', x.data.detach().numpy()[-4]
    print

result:

to_list(): -0.0940726622939
numpy: -0.09407266

to_list(): -0.0940726622939
numpy: -0.09407266

to_list(): -0.0940726548433
numpy: -0.094072655

to_list(): -0.0940726622939
numpy: -0.09407266

to_list(): -0.0940726548433
numpy: -0.094072655

to_list(): -0.0940726622939
numpy: -0.09407266

to_list(): -0.0940726548433
numpy: -0.094072655

to_list(): -0.0940726622939
numpy: -0.09407266

to_list(): -0.0940726622939
numpy: -0.09407266

to_list(): -0.0940726622939
numpy: -0.09407266

[sakhar]

Also, the numbers returned by torch are always inaccurate even when I use add(), it gives: -0.0940726548433
While the correct result (by numpy) is x-0.7y = -0.09407265968620777 where:
x = -0.0723949819803
y = 0.0309681110084

Numpy returns the correct (or more accurate) result.

You can use an external calculator to check that the result returned by numpy is the correct one.

[albanD]

Hi,

With the floating point semantic, both results are correct. There isn't one that is more accurate than the other.

Also I cannot reproduce this issue on any of the Linux machines I have, but I can reproduce on a colleague's Mac. Unfortunately I cannot use that machine for further investigation.
The next debugging step I think would be to print the bit representation of both inputs and outputs and see what is the difference.

[cpuhrsch]

cc @colesbury

[cpuhrsch]

@sakhar - is it at all possible for you to update to a more recent version of pytorch?

[colesbury]

In 0.4.1, depending on the CPU architecture, we used FMA instructions for add(). These are more precise than a multiply followed by an add because it does only one rounding instead of two.

The old implementation used FMA instructions for first N elements where N is a multiple of 8. The remaining elements used a C loop which may not be compiled with FMAs:

pytorch/aten/src/TH/vector/AVX2.cpp

Lines 26 to 28 in a24163a

	for (; i<(n); i++) {
	z[i] = x[i] + y[i] * c;
	}

[apaszke]

@colesbury all that makes sense. The problem is that the different results are not at the end of the tensor, but are scattered across it.

[colesbury]

@apaszke, you're right, that doesn't explain the problem. We were actually using THBlas(axpy) for in-place contiguous addition:

https://github.com/pytorch/pytorch/blob/v0.4.1/aten/src/TH/generic/THTensorMath.cpp#L1301-L1303

My guess is that the BLAS implementation on Mac depends on the alignment of the data. For 32-byte aligned data it probably uses FMA and for non-aligned data it probably uses separate multiply and addition instructions. The alignment is non-deterministic and every loop iteration re-allocates the NumPy array.

Anyways, the code path is pretty different in master and I don't think this problem exists there.

[jacobaustin123]

Has any work been done on this? This was actually a significant issue while grading homework using PyTorch (in the Columbia University machine learning class). Non-determinism in vanilla gradient descent on Linux (Colab) vs. Mac (most students) machines made grading extremely hard. I feel like basic functionality like addition and gradient descent ought to be deterministic on all machines.

[albanD]

@ja3067 This is a different issue. Here we try to get reproducible results for a given machine with a fixed software.

Across different machines, this is absolutely not possible: Addition on different CPUs can already give you different results due to multithreading for large enough Tensors.
Similarly, for different GPUs, you will get different results as they have different properties.