Arrays are not equal in 2.0.0b4 testsuite on Fedora rawhide/aarch64 (ARM v8 64bit)

[rathann]

To help us understand and resolve your issue please check that you have provided
the information below.

• Matplotlib version, Python version and Platform (Windows, OSX, Linux ...)

  matplotlib-2.0.0b4, python-2.7.12-6.fc26, python3-3.5.2-4.fc26, Fedora rawhide/aarch64

• How did you install Matplotlib and Python (pip, anaconda, from source ...)

  matplotlib from source, python from system packages

Log excerpt:

======================================================================
FAIL: test_psd_csd_equal (matplotlib.tests.test_mlab.spectral_testcase_Fs4_complex_onesided)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILDROOT/python-matplotlib-2.0.0-0.1.b4.fc26.aarch64/usr/lib64/python2.7/site-packages/matplotlib/tests/test_mlab.py", line 2096, in test_psd_csd_equal
    assert_array_equal(Pxx, Pxy)
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 813, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 739, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal
(mismatch 96.9230769231%)
 x: array([  6.851587e-11,   1.131382e-10,   1.330045e-10,   2.446886e-10,
         2.588422e-10,   9.790277e-11,   9.632650e-11,   4.858538e-10,
         7.779196e-10,   4.126505e-10,   2.595295e-11,   8.736602e-10,...
 y: array([  6.851587e-11 +0.000000e+00j,   1.131382e-10 +5.414530e-31j,
         1.330045e-10 -2.252263e-28j,   2.446886e-10 -1.512307e-28j,
         2.588422e-10 -5.607124e-29j,   9.790277e-11 -2.958476e-28j,...
======================================================================
FAIL: test_psd_csd_equal (matplotlib.tests.test_mlab.spectral_testcase_Fs4_complex_defaultsided)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILDROOT/python-matplotlib-2.0.0-0.1.b4.fc26.aarch64/usr/lib64/python2.7/site-packages/matplotlib/tests/test_mlab.py", line 2096, in test_psd_csd_equal
    assert_array_equal(Pxx, Pxy)
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 813, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 739, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal
(mismatch 98.4375%)
 x: array([  6.851587e-11,   5.656909e-11,   6.650227e-11,   1.223443e-10,
         1.294211e-10,   4.895139e-11,   4.816325e-11,   2.429269e-10,
         3.889598e-10,   2.063253e-10,   1.297648e-11,   4.368301e-10,...
 y: array([  6.851587e-11 +0.000000e+00j,   5.656909e-11 -7.521779e-30j,
         6.650227e-11 +3.815069e-29j,   1.223443e-10 +8.121089e-30j,
         1.294211e-10 -5.664577e-29j,   4.895139e-11 +1.171260e-28j,...
======================================================================
FAIL: test_psd_csd_equal (matplotlib.tests.test_mlab.spectral_testcase_Fs4_complex_twosided)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILDROOT/python-matplotlib-2.0.0-0.1.b4.fc26.aarch64/usr/lib64/python2.7/site-packages/matplotlib/tests/test_mlab.py", line 2096, in test_psd_csd_equal
    assert_array_equal(Pxx, Pxy)
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 813, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 739, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal
(mismatch 98.4375%)
 x: array([  6.851587e-11,   5.656909e-11,   6.650227e-11,   1.223443e-10,
         1.294211e-10,   4.895139e-11,   4.816325e-11,   2.429269e-10,
         3.889598e-10,   2.063253e-10,   1.297648e-11,   4.368301e-10,...
 y: array([  6.851587e-11 +0.000000e+00j,   5.656909e-11 -7.521779e-30j,
         6.650227e-11 +3.815069e-29j,   1.223443e-10 +8.121089e-30j,
         1.294211e-10 -5.664577e-29j,   4.895139e-11 +1.171260e-28j,...
======================================================================
FAIL: test_psd_csd_equal (matplotlib.tests.test_mlab.spectral_testcase_Fs4_real_defaultsided)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILDROOT/python-matplotlib-2.0.0-0.1.b4.fc26.aarch64/usr/lib64/python2.7/site-packages/matplotlib/tests/test_mlab.py", line 2096, in test_psd_csd_equal
    assert_array_equal(Pxx, Pxy)
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 813, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 739, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal
(mismatch 96.9230769231%)
 x: array([  6.851587e-11,   1.131382e-10,   1.330045e-10,   2.446886e-10,
         2.588422e-10,   9.790277e-11,   9.632650e-11,   4.858538e-10,
         7.779196e-10,   4.126505e-10,   2.595295e-11,   8.736602e-10,...
 y: array([  6.851587e-11 +0.000000e+00j,   1.131382e-10 +5.414530e-31j,
         1.330045e-10 -2.252263e-28j,   2.446886e-10 -1.512307e-28j,
         2.588422e-10 -5.607124e-29j,   9.790277e-11 -2.958476e-28j,...
======================================================================
FAIL: test_psd_csd_equal (matplotlib.tests.test_mlab.spectral_testcase_Fs4_real_onesided)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILDROOT/python-matplotlib-2.0.0-0.1.b4.fc26.aarch64/usr/lib64/python2.7/site-packages/matplotlib/tests/test_mlab.py", line 2096, in test_psd_csd_equal
    assert_array_equal(Pxx, Pxy)
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 813, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 739, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal
(mismatch 96.9230769231%)
 x: array([  6.851587e-11,   1.131382e-10,   1.330045e-10,   2.446886e-10,
         2.588422e-10,   9.790277e-11,   9.632650e-11,   4.858538e-10,
         7.779196e-10,   4.126505e-10,   2.595295e-11,   8.736602e-10,...
 y: array([  6.851587e-11 +0.000000e+00j,   1.131382e-10 +5.414530e-31j,
         1.330045e-10 -2.252263e-28j,   2.446886e-10 -1.512307e-28j,
         2.588422e-10 -5.607124e-29j,   9.790277e-11 -2.958476e-28j,...
======================================================================
FAIL: test_psd_csd_equal (matplotlib.tests.test_mlab.spectral_testcase_Fs4_real_twosided)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILDROOT/python-matplotlib-2.0.0-0.1.b4.fc26.aarch64/usr/lib64/python2.7/site-packages/matplotlib/tests/test_mlab.py", line 2096, in test_psd_csd_equal
    assert_array_equal(Pxx, Pxy)
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 813, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 739, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal
(mismatch 98.4375%)
 x: array([  6.851587e-11,   5.656909e-11,   6.650227e-11,   1.223443e-10,
         1.294211e-10,   4.895139e-11,   4.816325e-11,   2.429269e-10,
         3.889598e-10,   2.063253e-10,   1.297648e-11,   4.368301e-10,...
 y: array([  6.851587e-11 +0.000000e+00j,   5.656909e-11 -7.521779e-30j,
         6.650227e-11 +3.815069e-29j,   1.223443e-10 +8.121089e-30j,
         1.294211e-10 -5.664577e-29j,   4.895139e-11 +1.171260e-28j,...
======================================================================
FAIL: test_psd_csd_equal (matplotlib.tests.test_mlab.spectral_testcase_FsAll_complex_defaultsided)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILDROOT/python-matplotlib-2.0.0-0.1.b4.fc26.aarch64/usr/lib64/python2.7/site-packages/matplotlib/tests/test_mlab.py", line 2096, in test_psd_csd_equal
    assert_array_equal(Pxx, Pxy)
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 813, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 739, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal
(mismatch 98.4375%)
 x: array([  1.084763e-07,   5.947154e-08,   3.894422e-09,   5.098036e-08,
         1.243062e-07,   9.022498e-08,   8.124808e-09,   4.643260e-08,
         1.732248e-07,   1.748687e-07,   3.730256e-08,   3.213152e-08,...
 y: array([  1.084763e-07 +0.000000e+00j,   5.947154e-08 -2.846132e-25j,
         3.894422e-09 -3.678538e-27j,   5.098036e-08 -2.565235e-25j,
         1.243062e-07 -7.234946e-26j,   9.022498e-08 -1.238325e-25j,...
======================================================================
FAIL: test_psd_csd_equal (matplotlib.tests.test_mlab.spectral_testcase_FsAll_complex_onesided)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILDROOT/python-matplotlib-2.0.0-0.1.b4.fc26.aarch64/usr/lib64/python2.7/site-packages/matplotlib/tests/test_mlab.py", line 2096, in test_psd_csd_equal
    assert_array_equal(Pxx, Pxy)
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 813, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 739, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal
(mismatch 96.9230769231%)
 x: array([  7.203696e-06,   4.624615e-04,   1.539987e-03,   2.218494e-03,
         1.157601e-03,   1.667240e-04,   1.000658e-02,   4.707813e-02,
         9.254057e-02,   2.373157e-02,   1.300382e+00,   1.744418e+02,...
 y: array([  7.203696e-06 +0.000000e+00j,   4.624615e-04 -1.080357e-21j,
         1.539987e-03 +7.529828e-21j,   2.218494e-03 +2.647608e-21j,
         1.157601e-03 +3.916541e-21j,   1.667240e-04 +4.591193e-22j,...
======================================================================
FAIL: test_psd_csd_equal (matplotlib.tests.test_mlab.spectral_testcase_FsAll_real_defaultsided)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILDROOT/python-matplotlib-2.0.0-0.1.b4.fc26.aarch64/usr/lib64/python2.7/site-packages/matplotlib/tests/test_mlab.py", line 2096, in test_psd_csd_equal
    assert_array_equal(Pxx, Pxy)
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 813, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 739, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal
(mismatch 96.9230769231%)
 x: array([  7.203696e-06,   4.624615e-04,   1.539987e-03,   2.218494e-03,
         1.157601e-03,   1.667240e-04,   1.000658e-02,   4.707813e-02,
         9.254057e-02,   2.373157e-02,   1.300382e+00,   1.744418e+02,...
 y: array([  7.203696e-06 +0.000000e+00j,   4.624615e-04 -1.080357e-21j,
         1.539987e-03 +7.529828e-21j,   2.218494e-03 +2.647608e-21j,
         1.157601e-03 +3.916541e-21j,   1.667240e-04 +4.591193e-22j,...
======================================================================
FAIL: test_psd_csd_equal (matplotlib.tests.test_mlab.spectral_testcase_FsAll_complex_twosided)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILDROOT/python-matplotlib-2.0.0-0.1.b4.fc26.aarch64/usr/lib64/python2.7/site-packages/matplotlib/tests/test_mlab.py", line 2096, in test_psd_csd_equal
    assert_array_equal(Pxx, Pxy)
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 813, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 739, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal
(mismatch 98.4375%)
 x: array([  1.084763e-07,   5.947154e-08,   3.894422e-09,   5.098036e-08,
         1.243062e-07,   9.022498e-08,   8.124808e-09,   4.643260e-08,
         1.732248e-07,   1.748687e-07,   3.730256e-08,   3.213152e-08,...
 y: array([  1.084763e-07 +0.000000e+00j,   5.947154e-08 -2.846132e-25j,
         3.894422e-09 -3.678538e-27j,   5.098036e-08 -2.565235e-25j,
         1.243062e-07 -7.234946e-26j,   9.022498e-08 -1.238325e-25j,...
======================================================================
FAIL: test_psd_csd_equal (matplotlib.tests.test_mlab.spectral_testcase_FsAll_real_onesided)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILDROOT/python-matplotlib-2.0.0-0.1.b4.fc26.aarch64/usr/lib64/python2.7/site-packages/matplotlib/tests/test_mlab.py", line 2096, in test_psd_csd_equal
    assert_array_equal(Pxx, Pxy)
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 813, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 739, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal
(mismatch 96.9230769231%)
 x: array([  7.203696e-06,   4.624615e-04,   1.539987e-03,   2.218494e-03,
         1.157601e-03,   1.667240e-04,   1.000658e-02,   4.707813e-02,
         9.254057e-02,   2.373157e-02,   1.300382e+00,   1.744418e+02,...
 y: array([  7.203696e-06 +0.000000e+00j,   4.624615e-04 -1.080357e-21j,
         1.539987e-03 +7.529828e-21j,   2.218494e-03 +2.647608e-21j,
         1.157601e-03 +3.916541e-21j,   1.667240e-04 +4.591193e-22j,...
======================================================================
FAIL: test_psd_csd_equal (matplotlib.tests.test_mlab.spectral_testcase_FsAll_real_twosided)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builddir/build/BUILDROOT/python-matplotlib-2.0.0-0.1.b4.fc26.aarch64/usr/lib64/python2.7/site-packages/matplotlib/tests/test_mlab.py", line 2096, in test_psd_csd_equal
    assert_array_equal(Pxx, Pxy)
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 813, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/usr/lib64/python2.7/site-packages/numpy/testing/utils.py", line 739, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal
(mismatch 98.4375%)
 x: array([  1.084763e-07,   5.947154e-08,   3.894422e-09,   5.098036e-08,
         1.243062e-07,   9.022498e-08,   8.124808e-09,   4.643260e-08,
         1.732248e-07,   1.748687e-07,   3.730256e-08,   3.213152e-08,...
 y: array([  1.084763e-07 +0.000000e+00j,   5.947154e-08 -2.846132e-25j,
         3.894422e-09 -3.678538e-27j,   5.098036e-08 -2.565235e-25j,
         1.243062e-07 -7.234946e-26j,   9.022498e-08 -1.238325e-25j,...

[rathann]

It looks like all spectral_testcase_Fs4* and spectral_testcase_FsAll* test cases are failing here.

[tacaswell]

I am not surprised. These also all fail in 32bit architectures due to floating point differences between architectures (and I am not out of my depth at understanding why or what to do about it).

Does anyone know how numpy deals with this sort of thing?

@toddrjen you wrote these tests, any insight into how to deal with this?

[rathann]

FWIW, this occurs only on aarch64 in Fedora rawhide. x86_64, i686 and armv7hl builds are fine. I haven't tested ppc64/ppc64le or s390x yet.

[Kojoley]

@rathann is there an image for qemu so we can investigate this problem without arm boards?

[efiring]

In every case, the "x" is real and the "y" is complex with the imaginary part being essentially zero. The real parts match. Given that the problem arises only with one architecture, it seems almost certain that it is a problem with an upstream library (numpy, or something it links to), not matplotlib. It seems like it has to be in the fft routines and/or basic handling of complex numbers. Does numpy have full test coverage for all of this?

[rathann]

@Kojoley I only found this bit of information so far: https://fedoraproject.org/wiki/Architectures/AArch64/Install_with_QEMU . I'll keep searching and let you know when I find a prebuilt image.

[AdamWill]

So ppc64 has the same problem, as I reported at #7798 (also, I confirm it still happens with 2.0.0rc2). Not sure if the same happens on ppc64le (which is the little-endian ppc64 platform), might be interesting to see.

[AdamWill]

Yes, this does also affect ppc64le.

[AdamWill]

btw, it may be obvious, but just in case anyone hasn't noticed it yet, psd() does this:

Pxx, freqs = csd(x=x, y=None, NFFT=NFFT, Fs=Fs, detrend=detrend,
                 window=window, noverlap=noverlap, pad_to=pad_to,
                 sides=sides, scale_by_freq=scale_by_freq)
return Pxx.real, freqs

that is, it explicitly returns only the real component of the array generated by csd(), which is why the test fails. I suspect if you edited that line to just say return Pxx, freqs the test would pass, but the array values would still differ between the arches.

[dopplershift]

I don't think assert_array_equal is appropriate here (I'm amazed it worked before). Should probably be assert_array_almost_equal since you really do want to allow some epsilon, due to combination of floating point math and different calculation procedures. That would allow for making 1e-25 to be considered 0.