ANCOM unit test failures with scipy 0.17.0

[jairideout]

The following ANCOM unit tests fail with scipy 0.17.0 on Python 2 and 3:

$ python skbio/stats/tests/test_composition.py
.F.....................F................
======================================================================
FAIL: test_ancom_alternative_test (__main__.AncomTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "skbio/stats/tests/test_composition.py", line 642, in test_ancom_alternative_test
    assert_data_frame_almost_equal(result, exp)
  File "/Users/jairideout/dev/scikit-bio/skbio/util/_testing.py", line 1205, in assert_data_frame_almost_equal
    check_exact=False)
  File "/Users/jairideout/miniconda/envs/scikit-bio/lib/python3.5/site-packages/pandas/util/testing.py", line 1049, in assert_frame_equal
    obj='DataFrame.iloc[:, {0}]'.format(i))
  File "/Users/jairideout/miniconda/envs/scikit-bio/lib/python3.5/site-packages/pandas/util/testing.py", line 946, in assert_series_equal
    check_less_precise, obj='{0}'.format(obj))
  File "pandas/src/testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas/src/testing.c:3809)
  File "pandas/src/testing.pyx", line 147, in pandas._testing.assert_almost_equal (pandas/src/testing.c:2685)
  File "/Users/jairideout/miniconda/envs/scikit-bio/lib/python3.5/site-packages/pandas/util/testing.py", line 819, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: DataFrame.iloc[:, 0] are different

DataFrame.iloc[:, 0] values are different (100.0 %)
[left]:  [0, 0, 0, 0, 0, 0, 0]
[right]: [6, 6, 2, 2, 2, 2, 2]

======================================================================
FAIL: test_ancom_normal_data (__main__.AncomTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "skbio/stats/tests/test_composition.py", line 655, in test_ancom_normal_data
    assert_data_frame_almost_equal(result, exp)
  File "/Users/jairideout/dev/scikit-bio/skbio/util/_testing.py", line 1205, in assert_data_frame_almost_equal
    check_exact=False)
  File "/Users/jairideout/miniconda/envs/scikit-bio/lib/python3.5/site-packages/pandas/util/testing.py", line 1049, in assert_frame_equal
    obj='DataFrame.iloc[:, {0}]'.format(i))
  File "/Users/jairideout/miniconda/envs/scikit-bio/lib/python3.5/site-packages/pandas/util/testing.py", line 946, in assert_series_equal
    check_less_precise, obj='{0}'.format(obj))
  File "pandas/src/testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas/src/testing.c:3809)
  File "pandas/src/testing.pyx", line 147, in pandas._testing.assert_almost_equal (pandas/src/testing.c:2685)
  File "/Users/jairideout/miniconda/envs/scikit-bio/lib/python3.5/site-packages/pandas/util/testing.py", line 819, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: DataFrame.iloc[:, 0] are different

DataFrame.iloc[:, 0] values are different (22.22222 %)
[left]:  [8, 7, 3, 3, 7, 3, 3, 3, 3]
[right]: [8, 8, 3, 3, 8, 3, 3, 3, 3]

----------------------------------------------------------------------
Ran 40 tests in 0.435s

FAILED (failures=2)

The tests pass using the previous scipy release (0.16.1, available via conda and pip). Note that scipy 0.17.0 is currently only available via pip.

The tests are failing because something in scipy.stats.mannwhitneyu changed between scipy 0.16.1 and 0.17.0. Changing test_ancom_alternative_test to pass alternative='less' fixes the test. However, making the same change to test_ancom_normal_data doesn't fix the failure.

See #1258 for original discussion.

[mortonjt]

Thanks for investigating this.

What do you recommend to fix this? Can we just update the tests and require the development version of scikit-bio to depend on scipy 0.17.0?

[jairideout]

I'm not sure what the fix is -- we'd want to know why the expected values are different between scipy versions before updating the unit tests to use new expected values (the observed vs. expected values are very different).

[mortonjt]

Of course, let me add some more context

Looking at the test_ancom_normal_data test case

If you look at the original p-value matrix generated by the mann-whitney U-test with scipy==0.16, you get the following.

[[  0.00e+00   5.87e-15   5.48e-15   6.06e-15   5.70e-15   6.10e-15    5.67e-15   4.74e-15   4.08e-15]
 [  5.87e-15   0.00e+00   6.59e-15   6.54e-15   2.40e-01   6.10e-15    6.38e-15   6.28e-15   5.97e-15]
 [  5.48e-15   6.59e-15   0.00e+00   4.46e-01   6.57e-15   1.29e-01    2.06e-01   4.79e-01   2.63e-01]
 [  6.06e-15   6.54e-15   4.46e-01   0.00e+00   6.25e-15   1.99e-01    2.59e-01   3.10e-01   3.65e-01]
 [  5.70e-15   2.40e-01   6.57e-15   6.25e-15   0.00e+00   6.26e-15    6.01e-15   6.15e-15   5.82e-15]
 [  6.10e-15   6.10e-15   1.29e-01   1.99e-01   6.26e-15   0.00e+00    4.44e-01   5.62e-02   3.08e-01]
 [  5.67e-15   6.38e-15   2.06e-01   2.59e-01   6.01e-15   4.44e-01    0.00e+00   1.10e-01   2.91e-01]
 [  4.74e-15   6.28e-15   4.79e-01   3.10e-01   6.15e-15   5.62e-02    1.10e-01   0.00e+00   1.47e-01]
 [  4.08e-15   5.97e-15   2.63e-01   3.65e-01   5.82e-15   3.08e-01    2.91e-01   1.47e-01   0.00e+00]]

Now, if you look at the same p-value matrix with scipy==0.17, you'll get different results

[[  0.00e+00   1.34e-14   1.24e-14   1.28e-14   1.31e-14   1.23e-14    1.30e-14   1.24e-14   1.32e-14]
 [  1.34e-14   0.00e+00   1.34e-14   1.35e-14   6.79e-01   1.32e-14    1.30e-14   1.27e-14   1.35e-14]
 [  1.24e-14   1.34e-14   0.00e+00   6.34e-01   1.32e-14   6.75e-01    2.77e-01   4.24e-01   7.19e-01]
 [  1.28e-14   1.35e-14   6.34e-01   0.00e+00   1.34e-14   5.44e-01    4.49e-01   9.06e-01   9.96e-01]
 [  1.31e-14   6.79e-01   1.32e-14   1.34e-14   0.00e+00   1.28e-14    1.32e-14   1.32e-14   1.33e-14]
 [  1.23e-14   1.32e-14   6.75e-01   5.44e-01   1.28e-14   0.00e+00    3.36e-01   9.84e-01   7.74e-01]
 [  1.30e-14   1.30e-14   2.77e-01   4.49e-01   1.32e-14   3.36e-01    0.00e+00   3.95e-01   2.87e-01]
 [  1.24e-14   1.27e-14   4.24e-01   9.06e-01   1.32e-14   9.84e-01    3.95e-01   0.00e+00   6.47e-01]
 [  1.32e-14   1.35e-14   7.19e-01   9.96e-01   1.33e-14   7.74e-01    2.87e-01   6.47e-01   0.00e+00]]

And this is still different when alternative='less'

[[  0.00e+00   1.00e+00   6.32e-15   5.91e-15   1.00e+00   6.07e-15   6.13e-15   6.17e-15   6.53e-15]
 [  1.00e+00   0.00e+00   6.38e-15   6.28e-15   4.06e-01   6.61e-15    6.33e-15   6.63e-15   6.28e-15]
 [  6.32e-15   6.38e-15   0.00e+00   2.91e-01   1.00e+00   1.01e-01    4.74e-01   2.10e-01   5.35e-01]
 [  5.91e-15   6.28e-15   2.91e-01   0.00e+00   1.00e+00   3.38e-01    4.02e-01   3.78e-01   5.84e-01]
 [  1.00e+00   4.06e-01   1.00e+00   1.00e+00   0.00e+00   6.55e-15  6.10e-15   6.56e-15   6.80e-15]
 [  6.07e-15   6.61e-15   1.01e-01   3.38e-01   6.55e-15   0.00e+00    8.44e-01   5.16e-01   9.03e-01]
 [  6.13e-15   6.33e-15   4.74e-01   4.02e-01   6.10e-15   8.44e-01     0.00e+00   2.38e-01   7.16e-01]
 [  6.17e-15   6.63e-15   2.10e-01   3.78e-01   6.56e-15   5.16e-01     2.38e-01   0.00e+00   8.21e-01]
 [  6.53e-15   6.28e-15   5.35e-01   5.84e-01   6.80e-15   9.03e-01     7.16e-01   8.21e-01   0.00e+00]]

So, looks like the mann-whitney U-test has completely changed since they incorporated the alternative keyword. Not sure what the best way to proceed with this.

[jairideout]

Thanks for the details! Can these two tests be updated to use a different scipy stat function that is better behaved across scipy versions?

[jairideout]

Fixed in #1260.

[josef-pkt]

Hi,

The following two explain what happened with scipy 1.7.0
scipy/scipy#6062
scipy/scipy#6100
The current plan is to revert to the old behavior for 1.7.1, see PR

[jairideout]

Thanks @josef-pkt!