multiple test failures on s390x in fedora 23 #6338

ngothan · 2016-06-30T15:36:53Z

scipy-0.16 does not build on s390x, it just crash in the tests by running PYTHONPATH=$RPM_BUILD_ROOT/usr/lib64/python3.5/site-packages /usr/bin/python3 -c "import scipy; scipy.test('full', verbose=2)"

ERROR: Check linalg works with non-aligned memory
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/builddir/build/BUILDROOT/scipy-0.16.1-4.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/linalg/tests/test_decomp.py", line 2009, in test_aligned_mem
    eig(z.T, overwrite_a=True)
  File "/builddir/build/BUILDROOT/scipy-0.16.1-4.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/linalg/decomp.py", line 179, in eig
    overwrite_a=overwrite_a)
ValueError: On entry to DGEBAK parameter number 4 had an illegal value
======================================================================
FAIL: test_qhull.TestUtilities.test_degenerate_barycentric_transforms
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/builddir/build/BUILDROOT/scipy-0.16.1-4.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/spatial/tests/test_qhull.py", line 294, in test_degenerate_barycentric_transforms
    assert_(bad_count < 20, bad_count)
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 53, in assert_
    raise AssertionError(smsg)
AssertionError: 20
======================================================================
FAIL: test_qhull.TestUtilities.test_more_barycentric_transforms
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/builddir/build/BUILDROOT/scipy-0.16.1-4.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/spatial/tests/test_qhull.py", line 353, in test_more_barycentric_transforms
    unit_cube_tol=1e7*eps)
  File "/builddir/build/BUILDROOT/scipy-0.16.1-4.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/spatial/tests/test_qhull.py", line 280, in _check_barycentric_transforms
    assert_(ok.all(), "%s %s" % (err_msg, np.where(~ok)))
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 53, in assert_
    raise AssertionError(smsg)
AssertionError: ndim=4 (array([ 8071, 10512, 10513]),)
======================================================================
FAIL: test_data.test_boost(<Data for legendre_p_via_assoc_: legendre_p_large_ipp-legendre_p_large>,)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/builddir/build/BUILDROOT/scipy-0.16.1-4.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/special/tests/test_data.py", line 481, in _test_factory
    test.check(dtype=dtype)
  File "/builddir/build/BUILDROOT/scipy-0.16.1-4.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/special/_testutils.py", line 292, in check
    assert_(False, "\n".join(msg))
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 53, in assert_
    raise AssertionError(smsg)
AssertionError: 
Max |adiff|: 2.498e-16
Max |rdiff|: 9.56922e-14
Bad results (1 out of 160) for the following points (in output 0):
                         101.0               0.629447340965271 =>         -0.0007727131004340403 !=         -0.0007727131004341142  (rdiff          9.569216326469691e-14)
======================================================================
FAIL: test_data.test_boost(<Data for legendre_p_via_lpmn: legendre_p_large_ipp-legendre_p_large>,)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/builddir/build/BUILDROOT/scipy-0.16.1-4.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/special/tests/test_data.py", line 481, in _test_factory
    test.check(dtype=dtype)
  File "/builddir/build/BUILDROOT/scipy-0.16.1-4.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/special/_testutils.py", line 292, in check
    assert_(False, "\n".join(msg))
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 53, in assert_
    raise AssertionError(smsg)
AssertionError: 
Max |adiff|: 2.498e-16
Max |rdiff|: 9.56922e-14
Bad results (1 out of 160) for the following points (in output 0):
                         101.0               0.629447340965271 =>         -0.0007727131004340403 !=         -0.0007727131004341142  (rdiff          9.569216326469691e-14)
======================================================================
FAIL: test_orthogonal.test_j_roots
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/builddir/build/BUILDROOT/scipy-0.16.1-4.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/special/tests/test_orthogonal.py", line 323, in test_j_roots
    vgq(rf(18.24, 27.3), ef(18.24, 27.3), wf(18.24, 27.3), -1., 1., 25)
  File "/builddir/build/BUILDROOT/scipy-0.16.1-4.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/special/tests/test_orthogonal.py", line 286, in verify_gauss_quad
    assert_allclose(vv, np.eye(N), rtol, atol)
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 1358, in assert_allclose
    verbose=verbose, header=header)
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 713, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-15, atol=1e-14
(mismatch 0.3199999999999932%)
 x: array([[  1.000000e+00,  -9.107374e-16,  -2.974619e-16,  -1.468511e-16,
          1.018156e-16,   3.736521e-16,   7.011539e-16,  -1.840656e-16,
         -1.269248e-15,  -1.069845e-16,   1.338407e-15,   2.493098e-16,...
 y: array([[ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,
         0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
       [ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,...
======================================================================
FAIL: test_orthogonal.test_la_roots
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/builddir/build/BUILDROOT/scipy-0.16.1-4.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/special/tests/test_orthogonal.py", line 710, in test_la_roots
    vgq(rootf(0.1), evalf(0.1), weightf(0.1), 0., np.inf, 100, atol=1e-13)
  File "/builddir/build/BUILDROOT/scipy-0.16.1-4.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/special/tests/test_orthogonal.py", line 286, in verify_gauss_quad
    assert_allclose(vv, np.eye(N), rtol, atol)
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 1358, in assert_allclose
    verbose=verbose, header=header)
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 713, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-15, atol=1e-13
(mismatch 0.28000000000000114%)
 x: array([[  1.000000e+00,  -9.168102e-15,  -5.951562e-15, ...,
         -9.701575e-17,  -6.021814e-17,  -2.107937e-17],
       [ -9.223791e-15,   1.000000e+00,  -6.566734e-15, ...,...
 y: array([[ 1.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  1.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  1., ...,  0.,  0.,  0.],...
----------------------------------------------------------------------
Ran 19419 tests in 527.276s
FAILED (KNOWNFAIL=129, SKIP=1319, errors=1, failures=6)
Running unit tests for scipy
NumPy version 1.10.2rc1
NumPy relaxed strides checking option: True
NumPy is installed in /usr/lib64/python3.5/site-packages/numpy
SciPy version 0.16.1
SciPy is installed in /builddir/build/BUILDROOT/scipy-0.16.1-4.fc24.s390x/usr/lib64/python3.5/site-packages/scipy
Python version 3.5.0 (default, Dec 13 2015, 21:57:07) [GCC 5.2.1 20151104 (Red Hat 5.2.1-5)]
nose version 1.3.7
*** Error in `/usr/bin/python3': double free or corruption (!prev): 0x000002aa279ac130 ***

The text was updated successfully, but these errors were encountered:

ev-br · 2016-06-30T15:40:28Z

Scipy 0.16 is outdated and is unlikely to receive any updates.
The current stable version is 0.17.1. Scipy 0.18 is now in the release candidate phase. Please try either 0.18.rc1 or master.

ngothan · 2016-06-30T15:41:48Z

the issue still happens in latest scipy-0.18.0rc1

ev-br · 2016-06-30T15:42:41Z

Thanks for testing it!
Which ones, specifically?
See also #6286 for a report concerning Fedora 23.

pv · 2016-06-30T15:45:34Z

Note that the platform is s390x, which I think very few work with.

ngothan · 2016-06-30T16:02:58Z

the following tests failed on platform s390x (tested with scipi-0.18.0rc1)

======================================================================
ERROR: Check linalg works with non-aligned memory
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/than/rpmbuild/BUILDROOT/scipy-0.18.0rc1-1.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/linalg/tests/test_decomp.py", line 2234, in test_aligned_mem
    eig(z.T, overwrite_a=True)
  File "/home/than/rpmbuild/BUILDROOT/scipy-0.18.0rc1-1.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/linalg/decomp.py", line 175, in eig
    overwrite_a=overwrite_a)
ValueError: On entry to DGEBAK parameter number 4 had an illegal value

======================================================================
FAIL: test_nodata (test_arffread.NoDataTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/than/rpmbuild/BUILDROOT/scipy-0.18.0rc1-1.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/io/arff/tests/test_arffread.py", line 97, in test_nodata
    assert_equal(data.dtype, expected_dtype)
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 375, in assert_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
 ACTUAL: dtype([('sepallength', '>f8'), ('sepalwidth', '>f8'), ('petallength', '>f8'), ('petalwidth', '>f8'), ('class', 'S15')])
 DESIRED: dtype([('sepallength', '<f8'), ('sepalwidth', '<f8'), ('petallength', '<f8'), ('petalwidth', '<f8'), ('class', 'S15')])

======================================================================
FAIL: test_highpass (test_filter_design.TestCheb2ord)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 670, in chk_same_position
    assert_array_equal(x_id, y_id)
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 807, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 733, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal

(mismatch 0.9708737864077648%)
 x: array([ True, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,
       False, False, False, False, False, False, False, False, False,...
 y: array(False, dtype=bool)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/than/rpmbuild/BUILDROOT/scipy-0.18.0rc1-1.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/signal/tests/test_filter_design.py", line 755, in test_highpass
    assert_array_less(dB(h[w <= ws]), -rs + 0.1)
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 984, in assert_array_less
    header='Arrays are not less-ordered')
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 700, in assert_array_compare
    chk_same_position(x == -inf, y == -inf, hasval='-inf')
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 676, in chk_same_position
    raise AssertionError(msg)
AssertionError: 
Arrays are not less-ordered

x and y -inf location mismatch:
 x: array([      -inf, -91.742285, -85.750423, -82.276594, -79.845227,
       -77.994061, -76.51738 , -75.30565 , -74.293708, -73.439756,
       -72.715507, -72.101054, -71.581996, -71.147714, -70.790307,...
 y: array(-69.9)

======================================================================
FAIL: test_data.test_boost(<Data for legendre_p_via_assoc_: legendre_p_large_ipp-legendre_p_large>,)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/than/rpmbuild/BUILDROOT/scipy-0.18.0rc1-1.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/special/tests/test_data.py", line 515, in _test_factory
    test.check(dtype=dtype)
  File "/home/than/rpmbuild/BUILDROOT/scipy-0.18.0rc1-1.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/special/_testutils.py", line 344, in check
    assert_(False, "\n".join(msg))
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 71, in assert_
    raise AssertionError(smsg)
AssertionError: 
Max |adiff|: 2.498e-16
Max |rdiff|: 9.56922e-14
Bad results (1 out of 160) for the following points (in output 0):
                         101.0               0.629447340965271 =>         -0.0007727131004340403 !=         -0.0007727131004341142  (rdiff          9.569216326469691e-14)

======================================================================
FAIL: test_data.test_boost(<Data for legendre_p_via_lpmn: legendre_p_large_ipp-legendre_p_large>,)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/than/rpmbuild/BUILDROOT/scipy-0.18.0rc1-1.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/special/tests/test_data.py", line 515, in _test_factory
    test.check(dtype=dtype)
  File "/home/than/rpmbuild/BUILDROOT/scipy-0.18.0rc1-1.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/special/_testutils.py", line 344, in check
    assert_(False, "\n".join(msg))
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 71, in assert_
    raise AssertionError(smsg)
AssertionError: 
Max |adiff|: 2.498e-16
Max |rdiff|: 9.56922e-14
Bad results (1 out of 160) for the following points (in output 0):
                         101.0               0.629447340965271 =>         -0.0007727131004340403 !=         -0.0007727131004341142  (rdiff          9.569216326469691e-14)

======================================================================
FAIL: test_orthogonal.test_la_roots
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.5/site-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/than/rpmbuild/BUILDROOT/scipy-0.18.0rc1-1.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/special/tests/test_orthogonal.py", line 710, in test_la_roots
    vgq(rootf(0.1), evalf(0.1), weightf(0.1), 0., np.inf, 100, atol=1e-13)
  File "/home/than/rpmbuild/BUILDROOT/scipy-0.18.0rc1-1.fc24.s390x/usr/lib64/python3.5/site-packages/scipy/special/tests/test_orthogonal.py", line 286, in verify_gauss_quad
    assert_allclose(vv, np.eye(N), rtol, atol)
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 1391, in assert_allclose
    verbose=verbose, header=header)
  File "/usr/lib64/python3.5/site-packages/numpy/testing/utils.py", line 733, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-15, atol=1e-13

(mismatch 0.29999999999999716%)
 x: array([[  1.000000e+00,  -7.418044e-15,  -5.787505e-15, ...,
         -1.476561e-16,  -8.163852e-17,  -1.531694e-17],
       [ -7.418044e-15,   1.000000e+00,  -4.461875e-15, ...,...
 y: array([[ 1.,  0.,  0., ...,  0.,  0.,  0.],
       [ 0.,  1.,  0., ...,  0.,  0.,  0.],
       [ 0.,  0.,  1., ...,  0.,  0.,  0.],...

----------------------------------------------------------------------
Ran 24385 tests in 731.711s

FAILED (KNOWNFAIL=97, SKIP=1982, errors=1, failures=5)
*** Error in `/usr/bin/python3': double free or corruption (!prev): 0x000002aa30d72fa0 ***

rgommers · 2016-06-30T18:44:55Z

The first two and the last one are the only failures I don't recognize. The test_boost ones pop up on 32-bit Linux on x86 as well (but are very annoying to debug, unclear which point fails), and the test_highpass one is OpenBLAS. The qhull one is also long-standing.

person142 · 2016-06-30T19:26:59Z

FWIW the test_boost failure point appears to be here:

https://github.com/scipy/scipy/blob/master/scipy/special/tests/data/boost/legendre_p_large_ipp/legendre_p_large.txt#L96

so most likely the failures are these tests:

https://github.com/scipy/scipy/blob/master/scipy/special/tests/test_data.py#L221
https://github.com/scipy/scipy/blob/master/scipy/special/tests/test_data.py#L223

(I think this is what @rgommers was mentioning? Sorry if I read that wrong.)

rgommers · 2016-06-30T19:49:23Z

@person142 indeed. Finding the data files is the first pain, but worse is then to find the 1 or 2 out of 160 points that is failing and figuring out why that's platform-dependent and/or intermittent.

pv · 2016-06-30T20:06:18Z

The failure message prints the exact points that fail, what value was produced, what was expected, and by how much it is off. I'm not sure if there's something that can be improved here.

ev-br · 2016-06-30T20:33:42Z

It's almost magical that there are only a couple of reasonably mild failures on a platform we hardly support and definitely do not test on. I'm sure we won't object to patches which fix failures specific to s390x and do not break more common architectures.

ev-br · 2016-06-30T20:42:41Z

Of course, if someone feels like setting up an s390x buildbot or periodically run tests on this arch, that would be appreciated :-).

rgommers · 2016-06-30T21:01:36Z

The failure message prints the exact points that fail, what value was produced, what was expected, and by how much it is off. I'm not sure if there's something that can be improved here.

Right, sorry I described that quite wrong. Looking at it in more detail again, the trouble I had was after location the failure working back to a standalone example to reproduce it. All the check and Arg() stuff doesn't make that too easy. Last time I tried was for gh-3551, but I gave up.

pv · 2016-06-30T21:28:42Z

@rgommers: yes, what i'm saying is that the error message contains the numbers that are needed to produce the failure standalone. IOW, it is an allclose() check.

ngothan · 2016-07-01T11:45:03Z

@ev-br: i'm willing to test scipy for s390x and another platform like powerpc. Is there any idea how to fix these issues?
Thanks

ev-br · 2016-07-01T12:49:13Z

@ngothan Great that you're volunteering! To be clear: I don't think we'll be holding the release for s390x specific failures when more common platforms (Windows, OS X and Linux on x86) are clean.

Now, like Ralf said, some of the failures you report are seen on other platforms. See e.g. open gh issues with 0.18.0 tag and PRs with backport-candidate tag (both open and closed).

The second failure looks to be some endianness issue, <f8 vs >f8. The first one needs some debugging and some knowledge of the platform (which I don't have, sorry).

ev-br closed this as completed Jun 30, 2016

rgommers added the defect A clear bug or issue that prevents SciPy from being installed or used as expected label Jun 30, 2016

ev-br reopened this Jun 30, 2016

ocaisa mentioned this issue Jun 11, 2019

Catch errors or failures in numpy/scipy testsuites easybuilders/easybuild-easyblocks#1744

Closed

rgommers changed the title ~~scipy-0.16 - the tests crash on s390x in fedora 23~~ multiple test failures on s390x in fedora 23 Aug 11, 2019

rgommers mentioned this issue Jan 2, 2020

test_aligned_mem for eig crashing on ppcle64 #11305

Open

rgommers mentioned this issue Jul 10, 2020

Failing tests for ppc64le (and aarch64???) #10329

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

multiple test failures on s390x in fedora 23 #6338

multiple test failures on s390x in fedora 23 #6338

ngothan commented Jun 30, 2016 •

edited by pv

ev-br commented Jun 30, 2016

ngothan commented Jun 30, 2016

ev-br commented Jun 30, 2016

pv commented Jun 30, 2016 via email

ngothan commented Jun 30, 2016 •

edited by pv

rgommers commented Jun 30, 2016 •

edited

person142 commented Jun 30, 2016

rgommers commented Jun 30, 2016

pv commented Jun 30, 2016 •

edited

ev-br commented Jun 30, 2016

ev-br commented Jun 30, 2016

rgommers commented Jun 30, 2016

pv commented Jun 30, 2016 •

edited

ngothan commented Jul 1, 2016

ev-br commented Jul 1, 2016

multiple test failures on s390x in fedora 23 #6338

multiple test failures on s390x in fedora 23 #6338

Comments

ngothan commented Jun 30, 2016 • edited by pv

ev-br commented Jun 30, 2016

ngothan commented Jun 30, 2016

ev-br commented Jun 30, 2016

pv commented Jun 30, 2016 via email

ngothan commented Jun 30, 2016 • edited by pv

rgommers commented Jun 30, 2016 • edited

person142 commented Jun 30, 2016

rgommers commented Jun 30, 2016

pv commented Jun 30, 2016 • edited

ev-br commented Jun 30, 2016

ev-br commented Jun 30, 2016

rgommers commented Jun 30, 2016

pv commented Jun 30, 2016 • edited

ngothan commented Jul 1, 2016

ev-br commented Jul 1, 2016

ngothan commented Jun 30, 2016 •

edited by pv

ngothan commented Jun 30, 2016 •

edited by pv

rgommers commented Jun 30, 2016 •

edited

pv commented Jun 30, 2016 •

edited

pv commented Jun 30, 2016 •

edited