`tf.range` have accumulate floating point error on CPU

[fuzzyswan]

Click to expand!

Issue Type

Bug

Source

binary

Tensorflow Version

tf 2.10.0

Custom Code

Yes

OS Platform and Distribution

Linux Ubuntu 20.04

Mobile device

No response

Python version

No response

Bazel version

No response

GCC/Compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current Behaviour?

tf.range have accumulate greater and greater floating point error on CPU. In the example below, range(- 10, 10, 0.01) gives [... 9.980267 9.990267] when the correct result should be [... 9.98 9.99], and this deviation is too much in my opinion.

Standalone code to reproduce the issue

import tensorflow as tf
with tf.device('cpu'):
    tensor1 = tf.range(- 10, 10, 0.01) # [-10., -9.99 ... 9.980267 9.990267]
    print(tensor1)
with tf.device('gpu'):
    tensor2 = tf.range(- 10, 10, 0.01) # [-10., -9.99 ... 9.98 9.99]
    print(tensor2)
assert np.allclose(tensor1, tensor2) # AssertionError

Relevant log output

tf.Tensor([-10.        -9.99      -9.98     ...   9.970266   9.980267   9.990267], shape=(2000,), dtype=float32)
tf.Tensor([-10.        -9.99      -9.98     ...   9.969999   9.98       9.99    ], shape=(2000,), dtype=float32)
AssertionError

[tilakrayal]

@fuzzyswan,
I was able to execute the given code without any issue/error. Kindly find the gist of it here. Thank you!

[fuzzyswan]

Hi @tilakrayal , I run on TensorFlow 2.10 and turn on GPU on colab, and was able to reproduce the issue/error. Please find the gist here, thanks!

[bhack]

@fuzzyswan Same as #58131 (comment) probably they are confused that the soft placement is enabled by default. @tilakrayal You are running on a CPU colab (@tilakrayal test your gist with tf.config.set_soft_device_placement(False))

[bhack]

@fuzzyswan Python floats are represented as 64-bit double-precision values:

Can you try to reproduce the same error using dtype=tf.float64 in both tf.range calls?

[fuzzyswan]

@bhack The assertion indeed passes with dtype=tf.float64 and both CPU and GPU produce accurate results.

Although this issue can be avoided in this way (by using a higher precision data type), I just wonder if the implementation for tf.range could be improved because now the accumulated error seems very large on CPU, while on GPU it can produce relatively accurate results even with float32.

[bhack]

It is in the natural c++ float precision limit. What we could do is to align CPU and GPU impl.

See #58259

[bhack]

You can check yourself in pure c++ replicating your example with the CPU version in master vs the GPU aligned version.

#include <iostream>

using namespace std;

int main()
{
   // TF master
    float val = -10.f;
    for (int64_t i=0; i<2000; ++i) {
        val+=0.01f;
        cout << "A " << val << endl;
    }
    
    // PR #58259 
     for (int64_t i=0; i<2000; ++i) {
        val=-10.f + static_cast<float>(i) * 0.01f;
        cout << "B " << val << endl;
    }
}