Broken test_sparse_operator.test_sparse_mathematical_core with scipy 1.1.0

[marcoabreu]

Check out the latest master, build it with ci/build.py --platform ubuntu_cpu /work/runtime_functions.sh build_ubuntu_cpu_openblas.
Run with ci/build.py --platform ubuntu_cpu /work/runtime_functions.sh unittest_ubuntu_python2_cpu.
Happens on all configurations, no matter whether CPU or GPU etc.

build.py: 2018-05-11 08:49:35,496 Executing: docker run --rm -t --shm-size=500m -v /home/ubuntu/mxnet-apache:/work/mxnet -v /home/ubuntu/mxnet-apache/build:/work/build -u 0:0 mxnet/build.ubuntu_cpu /work/runtime_functions.sh unittest_ubuntu_python2_cpu
+ set +x
+ export PYTHONPATH=./python/
+ PYTHONPATH=./python/
+ export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
+ MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
+ nosetests-2.7 --verbose tests/python/unittest/test_sparse_operator.py:test_sparse_mathematical_core
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1188828163 to reproduce.
test_sparse_operator.test_sparse_mathematical_core ... [INFO] Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1220244381 to reproduce.
FAIL

======================================================================
FAIL: test_sparse_operator.test_sparse_mathematical_core
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/work/mxnet/tests/python/unittest/common.py", line 157, in test_new
    orig_test(*args, **kwargs)
  File "/work/mxnet/tests/python/unittest/test_sparse_operator.py", line 1084, in test_sparse_mathematical_core
    density=density, ograd_density=ograd_density)
  File "/work/mxnet/tests/python/unittest/test_sparse_operator.py", line 1056, in check_mathematical_core
    density=density, ograd_density=ograd_density)
  File "/work/mxnet/tests/python/unittest/test_sparse_operator.py", line 698, in check_sparse_mathematical_core
    assert_almost_equal(arr_grad, input_grad, equal_nan=True)
  File "/work/mxnet/python/mxnet/test_utils.py", line 493, in assert_almost_equal
    raise AssertionError(msg)
AssertionError:
Items are not equal:
Error nan exceeds tolerance rtol=0.000010, atol=0.000000.  Location of maximum error:(0, 0), a=inf, b=-inf
 a: array([[inf, inf, inf, inf, inf],
       [inf, inf, inf, inf, inf],
       [inf, inf, inf, inf, inf],...
 b: array([[-inf, -inf, -inf, -inf, -inf],
       [-inf, -inf, -inf, -inf, -inf],
       [-inf, -inf, -inf, -inf, -inf],...
-------------------- >> begin captured stdout << ---------------------
('pass', 0)
0.0, 0.0, False

--------------------- >> end captured stdout << ----------------------
-------------------- >> begin captured logging << --------------------
common: INFO: Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1188828163 to reproduce.
common: INFO: Setting test np/mx/python random seeds, use MXNET_TEST_SEED=1220244381 to reproduce.
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 1 test in 0.031s

FAILED (failures=1)

[marcoabreu]

This seems to be related to the scipy version. This test breaks with scipy 1.1.0, but works with 1.0.1.

[marcoabreu]

+ set +x
+ export PYTHONPATH=./python/
+ PYTHONPATH=./python/
+ export MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
+ MXNET_STORAGE_FALLBACK_LOG_VERBOSE=0
+ pip2 install scipy==1.0.1
Collecting scipy==1.0.1
  Downloading https://files.pythonhosted.org/packages/9c/0b/5deb712a9ea5bb0a1de837d04ef7625c5f3ee44efe7ed0765ceda681d7f1/scipy-1.0.1-cp27-cp27mu-manylinux1_x86_64.whl (46.7MB)
    100% |################################| 46.7MB 974kB/s
Requirement already satisfied: numpy>=1.8.2 in /usr/local/lib/python2.7/dist-packages (from scipy==1.0.1) (1.14.3)
matplotlib 2.2.2 has requirement python-dateutil>=2.1, but you'll have python-dateutil 1.5 which is incompatible.
Installing collected packages: scipy
  Found existing installation: scipy 1.1.0
    Uninstalling scipy-1.1.0:
      Successfully uninstalled scipy-1.1.0
Successfully installed scipy-1.0.1
+ pip3 install scipy==1.0.1
Collecting scipy==1.0.1
  Cache entry deserialization failed, entry ignored
  Downloading https://files.pythonhosted.org/packages/51/3d/494e1a81121c12233cb2f511e31b0dae3944008c81bbfa0218ec2d0038a8/scipy-1.0.1-cp35-cp35m-manylinux1_x86_64.whl (49.6MB)
    100% |################################| 49.7MB 1.4MB/s
Requirement already satisfied: numpy>=1.8.2 in /usr/local/lib/python3.5/dist-packages (from scipy==1.0.1) (1.14.3)
Installing collected packages: scipy
  Found existing installation: scipy 1.1.0
    Uninstalling scipy-1.1.0:
      Successfully uninstalled scipy-1.1.0
Successfully installed scipy-1.0.1
+ nosetests-2.7 --verbose tests/python/unittest/test_sparse_operator.py:test_sparse_mathematical_core
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=1319374570 to reproduce.
test_sparse_operator.test_sparse_mathematical_core ... ok

----------------------------------------------------------------------
Ran 1 test in 5.101s

OK

[larroy]

Do we know the cause of this bug? @eric-haibin-lin

[ptrendx]

We saw that in our internal testing and @DickJC123 made some investigation yesterday. @DickJC123 - could you share your findings?

[DickJC123]

#10896

[DickJC123]

I also filed an issue with scipy: scipy/scipy#8819

[eric-haibin-lin]

Fixed by #10961