AttentionWrapperTest results failing on nightlies

[seanpmorgan]

Describe the bug
Nightly tests and CI is breaking for testLuongMonotonicNotNormalized and testLuongMonotonicScaled.

Quick look I couldn't find a commit on TF core that explains the result differing from before. Possbily related to #736 ?

======================================================================
FAIL: testLuongMonotonicNotNormalized (__main__.AttentionWrapperTest)
testLuongMonotonicNotNormalized (__main__.AttentionWrapperTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/absl/third_party/unittest3_backport/case.py", line 37, in testPartExecutor
    yield
  File "/usr/local/lib/python2.7/dist-packages/absl/third_party/unittest3_backport/case.py", line 162, in run
    testMethod()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/framework/test_util.py", line 1104, in decorated
    f(self, *args, **kwargs)
  File "/root/.cache/bazel/_bazel_root/e051f2f195071208ea7f081f88730f4f/execroot/tensorflow_addons/bazel-out/k8-opt/bin/tensorflow_addons/seq2seq/attention_wrapper_test.runfiles/tensorflow_addons/tensorflow_addons/seq2seq/attention_wrapper_test.py", line 865, in testLuongMonotonicNotNormalized
    expected_final_alignment_history=expected_final_alignment_history)
  File "/root/.cache/bazel/_bazel_root/e051f2f195071208ea7f081f88730f4f/execroot/tensorflow_addons/bazel-out/k8-opt/bin/tensorflow_addons/seq2seq/attention_wrapper_test.runfiles/tensorflow_addons/tensorflow_addons/seq2seq/attention_wrapper_test.py", line 349, in _testWithAttention
    create_attention_kwargs=create_attention_kwargs)
  File "/root/.cache/bazel/_bazel_root/e051f2f195071208ea7f081f88730f4f/execroot/tensorflow_addons/bazel-out/k8-opt/bin/tensorflow_addons/seq2seq/attention_wrapper_test.runfiles/tensorflow_addons/tensorflow_addons/seq2seq/attention_wrapper_test.py", line 523, in _testWithMaybeMultiAttention
    expected_final_state, final_state_info)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/util/nest.py", line 568, in map_structure
    structure[0], [func(*x) for x in entries],
  File "/root/.cache/bazel/_bazel_root/e051f2f195071208ea7f081f88730f4f/execroot/tensorflow_addons/bazel-out/k8-opt/bin/tensorflow_addons/seq2seq/attention_wrapper_test.runfiles/tensorflow_addons/tensorflow_addons/seq2seq/attention_wrapper_test.py", line 279, in assertAllCloseOrEqual
    x, y, atol=1e-3, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/framework/test_util.py", line 1146, in decorated
    return f(*args, **kwds)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/framework/test_util.py", line 2492, in assertAllClose
    self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/framework/test_util.py", line 2459, in _assertAllCloseRecursive
    (path_str, path_str, msg)))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/framework/test_util.py", line 2394, in _assertArrayLikeAllClose
    a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
  File "/usr/local/lib/python2.7/dist-packages/numpy/testing/_private/utils.py", line 1503, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/usr/local/lib/python2.7/dist-packages/numpy/testing/_private/utils.py", line 829, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-06, atol=0.001
Mismatched value: a is different from b.
not close lhs = 0.059128221
not close rhs = 0.0618268698454
not close dif = 0.00269864884539
not close tol = 0.00100006182687
dtype = float64, shape = ()
Mismatch: 100%
Max absolute difference: 0.00269865
Max relative difference: 0.04364848
 x: array(0.059128)
 y: array(0.061827, dtype=float32)

======================================================================
FAIL: testLuongMonotonicScaled (__main__.AttentionWrapperTest)
testLuongMonotonicScaled (__main__.AttentionWrapperTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/absl/third_party/unittest3_backport/case.py", line 37, in testPartExecutor
    yield
  File "/usr/local/lib/python2.7/dist-packages/absl/third_party/unittest3_backport/case.py", line 162, in run
    testMethod()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/framework/test_util.py", line 1104, in decorated
    f(self, *args, **kwargs)
  File "/root/.cache/bazel/_bazel_root/e051f2f195071208ea7f081f88730f4f/execroot/tensorflow_addons/bazel-out/k8-opt/bin/tensorflow_addons/seq2seq/attention_wrapper_test.runfiles/tensorflow_addons/tensorflow_addons/seq2seq/attention_wrapper_test.py", line 901, in testLuongMonotonicScaled
    create_attention_kwargs=create_attention_kwargs)
  File "/root/.cache/bazel/_bazel_root/e051f2f195071208ea7f081f88730f4f/execroot/tensorflow_addons/bazel-out/k8-opt/bin/tensorflow_addons/seq2seq/attention_wrapper_test.runfiles/tensorflow_addons/tensorflow_addons/seq2seq/attention_wrapper_test.py", line 349, in _testWithAttention
    create_attention_kwargs=create_attention_kwargs)
  File "/root/.cache/bazel/_bazel_root/e051f2f195071208ea7f081f88730f4f/execroot/tensorflow_addons/bazel-out/k8-opt/bin/tensorflow_addons/seq2seq/attention_wrapper_test.runfiles/tensorflow_addons/tensorflow_addons/seq2seq/attention_wrapper_test.py", line 523, in _testWithMaybeMultiAttention
    expected_final_state, final_state_info)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/util/nest.py", line 568, in map_structure
    structure[0], [func(*x) for x in entries],
  File "/root/.cache/bazel/_bazel_root/e051f2f195071208ea7f081f88730f4f/execroot/tensorflow_addons/bazel-out/k8-opt/bin/tensorflow_addons/seq2seq/attention_wrapper_test.runfiles/tensorflow_addons/tensorflow_addons/seq2seq/attention_wrapper_test.py", line 279, in assertAllCloseOrEqual
    x, y, atol=1e-3, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/framework/test_util.py", line 1146, in decorated
    return f(*args, **kwds)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/framework/test_util.py", line 2492, in assertAllClose
    self._assertAllCloseRecursive(a, b, rtol=rtol, atol=atol, msg=msg)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/framework/test_util.py", line 2459, in _assertAllCloseRecursive
    (path_str, path_str, msg)))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow_core/python/framework/test_util.py", line 2394, in _assertArrayLikeAllClose
    a, b, rtol=rtol, atol=atol, err_msg="\n".join(msgs), equal_nan=True)
  File "/usr/local/lib/python2.7/dist-packages/numpy/testing/_private/utils.py", line 1503, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/usr/local/lib/python2.7/dist-packages/numpy/testing/_private/utils.py", line 829, in assert_array_compare
    raise AssertionError(msg)
AssertionError:
Not equal to tolerance rtol=1e-06, atol=0.001
Mismatched value: a is different from b.
not close lhs = 0.059128221
not close rhs = 0.0618268698454
not close dif = 0.00269864884539
not close tol = 0.00100006182687
dtype = float64, shape = ()
Mismatch: 100%
Max absolute difference: 0.00269865
Max relative difference: 0.04364848
 x: array(0.059128)
 y: array(0.061827, dtype=float32)

[seanpmorgan]

cc @guillaumekln @qlzh727

[guillaumekln]

It seems there were some precision changes/fixes impacting tf.sigmoid between tf_nightly-2.1.0.dev20191203 and tf_nightly-2.1.0.dev20191219. The later returns values with higher precision, for example:

In 2.1.0.dev20191203:

>>> tf.sigmoid(-20.0)
<tf.Tensor: shape=(), dtype=float32, numpy=0.0>

In 2.1.0.dev20191219:

>>> tf.sigmoid(-20.0)
<tf.Tensor: shape=(), dtype=float32, numpy=2.0611537e-09>

I did not find the exact commit but find this one which update the documentation in consequence: tensorflow/tensorflow@365ca18

[guillaumekln]

But there are some strange things happening. For example going from 3 to 4 elements changes the result on GPU...

>>> tf.sigmoid([34.0, 0.0, 0.0])
<tf.Tensor: shape=(3,), dtype=float32, numpy=array([1. , 0.5, 0.5], dtype=float32)>

>>> tf.sigmoid([34.0, 0.0, 0.0, 0.0])
<tf.Tensor: shape=(4,), dtype=float32, numpy=array([0.99999994, 0.5       , 0.5       , 0.5       ], dtype=float32)

I will raise this to the TensorFlow team to better know what are the expectations.

EDIT: tensorflow/tensorflow#35309

[guillaumekln]

I tested with TensorFlow 2.1 and the latest nightly and it seems the sigmoid issue is now resolved. I will send a new PR that enables the tests again.