Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

numpy test failed with openblas #1673

Closed
beew opened this issue Jul 7, 2018 · 18 comments

Comments

Projects
None yet
5 participants
@beew
Copy link

commented Jul 7, 2018

updated to openblas0.3.1 and numpy appears to have some problems.

OS is Ubuntu 16.04.4 64 bits. I made sure that I don't have any bit of Ubuntu's packaged openblas in my system. I compiled openblas with

make PREFIX=/opt/openblas-mpi NO_AFFINITY=1 DYNAMIC_ARCH=1 USE_OPENMP=1 -j8

I use update alternatives to switch between intel-mkl and openblas.
tested numpy in python3 with numpy.test() (see outputs below for python and numpy versions)

intel mkl : All tests passed
openblas0.3.0: All test passed
openblas0.3.1 :7 tests failed

Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.test()
Running unit tests for numpy
NumPy version 1.14.5
NumPy relaxed strides checking option: True
NumPy is installed in /home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy
Python version 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609]
nose version 1.3.7
...
======================================================================
ERROR: numpy.linalg.tests.test_linalg.TestCholesky.test_basic_property
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 1528, in test_basic_property
    c = np.linalg.cholesky(a)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/linalg.py", line 614, in cholesky
    r = gufunc(a, signature=signature, extobj=extobj)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/linalg.py", line 92, in _raise_linalgerror_nonposdef
    raise LinAlgError("Matrix is not positive definite")
numpy.linalg.linalg.LinAlgError: Matrix is not positive definite

======================================================================
FAIL: numpy.core.tests.test_multiarray.TestDot.test_accelerate_framework_sgemv_fix
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/core/tests/test_multiarray.py", line 5350, in test_accelerate_framework_sgemv_fix
    assert_dot_close(A_f, X_f, desired)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/core/tests/test_multiarray.py", line 5335, in assert_dot_close
    assert_allclose(np.dot(A, X), desired, rtol=1e-5, atol=1e-7)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 1396, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 779, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

(mismatch 64.96000000000001%)
 x: array([54.477673, 48.264233, 48.50962 , ..., 51.572353, 47.31817 ,
       49.40976 ], dtype=float32)
 y: array([54.477672, 48.264233, 48.50962 , ..., 51.572354, 47.318168,
       49.40976 ])

======================================================================
FAIL: numpy.core.tests.test_multiarray.TestDot.test_dot_3args
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/core/tests/test_multiarray.py", line 5260, in test_dot_3args
    assert_array_equal(r2, r)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 855, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 779, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal

(mismatch 54.00390625%)
 x: array([[3.837556, 3.884982, 2.996465, ..., 4.124658, 3.355041, 4.719577],
       [5.508586, 5.532815, 4.610427, ..., 5.428518, 5.545739, 7.896924],
       [2.991256, 3.362753, 2.29233 , ..., 3.622751, 2.871046, 4.24088 ],...
 y: array([[3.837556, 3.884982, 2.996465, ..., 4.124658, 3.355041, 4.719577],
       [5.508586, 5.532815, 4.610427, ..., 5.428518, 5.545739, 7.896924],
       [2.991256, 3.362753, 2.29233 , ..., 3.622751, 2.871046, 4.24088 ],...

======================================================================
FAIL: numpy.lib.tests.test_polynomial.TestDocs.test_polyfit
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/lib/tests/test_polynomial.py", line 151, in test_polyfit
    assert_almost_equal(val0, cov, decimal=4)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 565, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 963, in assert_array_almost_equal
    precision=decimal)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 779, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 4 decimals

(mismatch 66.66666666666666%)
 x: array([[ 2.9388, -5.8776,  1.6327],
       [-5.8776, 12.7347, -4.2449],
       [ 1.6327, -4.2449,  2.322 ]])
 y: array([[ 2.9388, -3.2025, -1.5824],
       [-5.8776,  6.9386,  2.7211],
       [ 1.6327, -2.3129,  0.    ]])

======================================================================
FAIL: numpy.linalg.tests.test_linalg.TestCond2.test_sq_cases
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 357, in _check_cases
    case.check(func)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 85, in check
    do(self.a, self.b, tags=self.tags)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 702, in do
    single_decimal=5, double_decimal=11)
AssertionError: 
Arrays are not almost equal to 11 decimals
 ACTUAL: 109.14438169870651
 DESIRED: 2929.7593546253224

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 367, in test_sq_cases
    _check_cases(self.do, require={'square'}, exclude={'generalized', 'size-0'})
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 361, in _check_cases
    raise AssertionError(msg)
AssertionError: In test case: <LinalgCase: 8x8>

Traceback (most recent call last):
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 357, in _check_cases
    case.check(func)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 85, in check
    do(self.a, self.b, tags=self.tags)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 702, in do
    single_decimal=5, double_decimal=11)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 42, in assert_almost_equal
    old_assert_almost_equal(a, b, decimal=decimal, **kw)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 581, in assert_almost_equal
    raise AssertionError(_build_err_msg())
AssertionError: 
Arrays are not almost equal to 11 decimals
 ACTUAL: 109.14438169870651
 DESIRED: 2929.7593546253224


======================================================================
FAIL: numpy.linalg.tests.test_linalg.TestCond2.test_stacked_arrays_explicitly
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 706, in test_stacked_arrays_explicitly
    assert_equal(linalg.cond(A, 2), linalg.cond(A[None, ...], 2)[0])
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 411, in assert_equal
    raise AssertionError(msg)
AssertionError: 
Items are not equal:
 ACTUAL: 21.251679753309496
 DESIRED: 25.088240607326345

======================================================================
FAIL: numpy.linalg.tests.test_linalg.TestInv.test_sq_cases
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 357, in _check_cases
    case.check(func)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 85, in check
    do(self.a, self.b, tags=self.tags)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 528, in do
    identity_like_generalized(a))
AssertionError: 
Arrays are not almost equal to 12 decimals

(mismatch 50.0%)
 x: array([[ 1.000000000000e+00,  3.510493027442e+00,  3.531723631973e-16,
         3.362551919964e+00, -2.721455546200e-01,  7.802513871393e-01,
         7.582314584308e-01,  6.884173128842e-01],...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0.],...

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 367, in test_sq_cases
    _check_cases(self.do, require={'square'}, exclude={'generalized', 'size-0'})
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 361, in _check_cases
    raise AssertionError(msg)
AssertionError: In test case: <LinalgCase: 8x8>

Traceback (most recent call last):
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 357, in _check_cases
    case.check(func)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 85, in check
    do(self.a, self.b, tags=self.tags)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 528, in do
    identity_like_generalized(a))
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 42, in assert_almost_equal
    old_assert_almost_equal(a, b, decimal=decimal, **kw)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 565, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 963, in assert_array_almost_equal
    precision=decimal)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 779, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 12 decimals

(mismatch 50.0%)
 x: array([[ 1.000000000000e+00,  3.510493027442e+00,  3.531723631973e-16,
         3.362551919964e+00, -2.721455546200e-01,  7.802513871393e-01,
         7.582314584308e-01,  6.884173128842e-01],...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0.],...


======================================================================
FAIL: numpy.linalg.tests.test_linalg.TestSolve.test_sq_cases
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 357, in _check_cases
    case.check(func)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 85, in check
    do(self.a, self.b, tags=self.tags)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 461, in do
    assert_almost_equal(b, dot_generalized(a, x))
AssertionError: 
Arrays are not almost equal to 12 decimals

(mismatch 33.33333333333333%)
 x: array([[2., 1., 4.],
       [3., 4., 6.]])
 y: array([[2.            , 1.            , 5.666666666667],
       [3.            , 4.            , 2.            ]])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 367, in test_sq_cases
    _check_cases(self.do, require={'square'}, exclude={'generalized', 'size-0'})
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 361, in _check_cases
    raise AssertionError(msg)
AssertionError: In test case: <LinalgCase: double_2_stride_+1_-4_stride_-4_+1>

Traceback (most recent call last):
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 357, in _check_cases
    case.check(func)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 85, in check
    do(self.a, self.b, tags=self.tags)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 461, in do
    assert_almost_equal(b, dot_generalized(a, x))
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 42, in assert_almost_equal
    old_assert_almost_equal(a, b, decimal=decimal, **kw)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 565, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 963, in assert_array_almost_equal
    precision=decimal)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 779, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 12 decimals

(mismatch 33.33333333333333%)
 x: array([[2., 1., 4.],
       [3., 4., 6.]])
 y: array([[2.            , 1.            , 5.666666666667],
       [3.            , 4.            , 2.            ]])


----------------------------------------------------------------------
Ran 6821 tests in 21.165s

FAILED (KNOWNFAIL=14, SKIP=15, errors=1, failures=7)
<nose.result.TextTestResult run=6821 errors=1 failures=7>
>>> 
@martin-frbg

This comment has been minimized.

Copy link
Collaborator

commented Jul 7, 2018

Could you re-test with current "develop" branch please ? This could be another instance of #1666

@beew

This comment has been minimized.

Copy link
Author

commented Jul 7, 2018

Hi,
Just tested the dev branch
Still failed, but only 5 tests this time so that is an improvement. :)

python3
Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.test()
Running unit tests for numpy
NumPy version 1.14.5
NumPy relaxed strides checking option: True
NumPy is installed in /home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy
Python version 3.5.2 (default, Nov 23 2017, 16:37:01) [GCC 5.4.0 20160609]
nose version 1.3.7
...
======================================================================
ERROR: numpy.linalg.tests.test_linalg.TestCholesky.test_basic_property
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 1528, in test_basic_property
    c = np.linalg.cholesky(a)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/linalg.py", line 614, in cholesky
    r = gufunc(a, signature=signature, extobj=extobj)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/linalg.py", line 92, in _raise_linalgerror_nonposdef
    raise LinAlgError("Matrix is not positive definite")
numpy.linalg.linalg.LinAlgError: Matrix is not positive definite

======================================================================
FAIL: numpy.core.tests.test_multiarray.TestDot.test_accelerate_framework_sgemv_fix
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/core/tests/test_multiarray.py", line 5350, in test_accelerate_framework_sgemv_fix
    assert_dot_close(A_f, X_f, desired)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/core/tests/test_multiarray.py", line 5335, in assert_dot_close
    assert_allclose(np.dot(A, X), desired, rtol=1e-5, atol=1e-7)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 1396, in assert_allclose
    verbose=verbose, header=header, equal_nan=equal_nan)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 779, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Not equal to tolerance rtol=1e-05, atol=1e-07

(mismatch 35.56%)
 x: array([54.477673, 48.264233, 48.50962 , ..., 51.572353, 47.31817 ,
       49.40976 ], dtype=float32)
 y: array([54.477672, 48.264233, 48.50962 , ..., 51.572354, 47.318168,
       49.40976 ])

======================================================================
FAIL: numpy.core.tests.test_multiarray.TestDot.test_dot_3args
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/core/tests/test_multiarray.py", line 5260, in test_dot_3args
    assert_array_equal(r2, r)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 855, in assert_array_equal
    verbose=verbose, header='Arrays are not equal')
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 779, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not equal

(mismatch 72.4609375%)
 x: array([[3.837556, 3.884982, 2.996465, ..., 4.124658, 3.355041, 4.719577],
       [5.508586, 5.532815, 4.610427, ..., 5.428518, 5.545739, 7.896924],
       [2.991256, 3.362753, 2.29233 , ..., 3.622751, 2.871046, 4.24088 ],...
 y: array([[3.837556, 3.884982, 2.996465, ..., 4.124658, 3.355041, 4.719577],
       [5.508586, 5.532815, 4.610427, ..., 5.428518, 5.545739, 7.896924],
       [2.991256, 3.362753, 2.29233 , ..., 3.622751, 2.871046, 4.24088 ],...

======================================================================
FAIL: numpy.linalg.tests.test_linalg.TestCond2.test_sq_cases
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 357, in _check_cases
    case.check(func)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 85, in check
    do(self.a, self.b, tags=self.tags)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 702, in do
    single_decimal=5, double_decimal=11)
AssertionError: 
Arrays are not almost equal to 11 decimals
 ACTUAL: 109.14438169870651
 DESIRED: 956.1978190642191

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 367, in test_sq_cases
    _check_cases(self.do, require={'square'}, exclude={'generalized', 'size-0'})
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 361, in _check_cases
    raise AssertionError(msg)
AssertionError: In test case: <LinalgCase: 8x8>

Traceback (most recent call last):
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 357, in _check_cases
    case.check(func)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 85, in check
    do(self.a, self.b, tags=self.tags)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 702, in do
    single_decimal=5, double_decimal=11)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 42, in assert_almost_equal
    old_assert_almost_equal(a, b, decimal=decimal, **kw)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 581, in assert_almost_equal
    raise AssertionError(_build_err_msg())
AssertionError: 
Arrays are not almost equal to 11 decimals
 ACTUAL: 109.14438169870651
 DESIRED: 956.1978190642191


======================================================================
FAIL: numpy.linalg.tests.test_linalg.TestInv.test_sq_cases
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 357, in _check_cases
    case.check(func)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 85, in check
    do(self.a, self.b, tags=self.tags)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 528, in do
    identity_like_generalized(a))
AssertionError: 
Arrays are not almost equal to 12 decimals

(mismatch 37.5%)
 x: array([[ 1.000000000000e+00,  1.727493726144e-17,  3.763971890697e+00,
         3.362551919964e+00, -9.605628675042e-01,  7.802513871393e-01,
         1.142712647582e-15, -4.938916002863e-16],...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0.],...

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 367, in test_sq_cases
    _check_cases(self.do, require={'square'}, exclude={'generalized', 'size-0'})
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 361, in _check_cases
    raise AssertionError(msg)
AssertionError: In test case: <LinalgCase: 8x8>

Traceback (most recent call last):
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 357, in _check_cases
    case.check(func)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 85, in check
    do(self.a, self.b, tags=self.tags)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 528, in do
    identity_like_generalized(a))
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 42, in assert_almost_equal
    old_assert_almost_equal(a, b, decimal=decimal, **kw)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 565, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 963, in assert_array_almost_equal
    precision=decimal)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 779, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 12 decimals

(mismatch 37.5%)
 x: array([[ 1.000000000000e+00,  1.727493726144e-17,  3.763971890697e+00,
         3.362551919964e+00, -9.605628675042e-01,  7.802513871393e-01,
         1.142712647582e-15, -4.938916002863e-16],...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0.],...


======================================================================
FAIL: numpy.linalg.tests.test_linalg.TestSolve.test_sq_cases
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 357, in _check_cases
    case.check(func)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 85, in check
    do(self.a, self.b, tags=self.tags)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 461, in do
    assert_almost_equal(b, dot_generalized(a, x))
AssertionError: 
Arrays are not almost equal to 12 decimals

(mismatch 33.33333333333333%)
 x: array([[2., 1., 4.],
       [3., 4., 6.]])
 y: array([[2.            , 1.277777777778, 4.            ],
       [3.            , 1.333333333333, 6.            ]])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 367, in test_sq_cases
    _check_cases(self.do, require={'square'}, exclude={'generalized', 'size-0'})
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 361, in _check_cases
    raise AssertionError(msg)
AssertionError: In test case: <LinalgCase: double_2_stride_+3_-4_stride_+1_+1>

Traceback (most recent call last):
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 357, in _check_cases
    case.check(func)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 85, in check
    do(self.a, self.b, tags=self.tags)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 461, in do
    assert_almost_equal(b, dot_generalized(a, x))
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/linalg/tests/test_linalg.py", line 42, in assert_almost_equal
    old_assert_almost_equal(a, b, decimal=decimal, **kw)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 565, in assert_almost_equal
    return assert_array_almost_equal(actual, desired, decimal, err_msg)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 963, in assert_array_almost_equal
    precision=decimal)
  File "/home/bernard/.local/lib/python3.5/site-packages/numpy-1.14.5-py3.5-linux-x86_64.egg/numpy/testing/nose_tools/utils.py", line 779, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 12 decimals

(mismatch 33.33333333333333%)
 x: array([[2., 1., 4.],
       [3., 4., 6.]])
 y: array([[2.            , 1.277777777778, 4.            ],
       [3.            , 1.333333333333, 6.            ]])


----------------------------------------------------------------------
Ran 6821 tests in 20.711s

FAILED (KNOWNFAIL=14, SKIP=15, errors=1, failures=5)
<nose.result.TextTestResult run=6821 errors=1 failures=5>
@martin-frbg

This comment has been minimized.

Copy link
Collaborator

commented Jul 7, 2018

Would be a pity if we had to revert all that thread initialization speedup we gained with 0.3.1. BTW what is your cpu type (not that much changed in terms of cpu-specific code lately) ?

@sandwichmaker

This comment has been minimized.

Copy link

commented Jul 7, 2018

@beew would it be possible for you to bisect on one of these remaining failures? one more thing to try would be to see if any of these failures occur with a single threaded version. I recommend trying two cases:

  1. Compile OpenBLAS with USE_OPENMP=0
  2. Compiled with USE_OPENMP=1 but set OMP_NUM_THREADS=1 when running the tests.
@brada4

This comment has been minimized.

Copy link
Contributor

commented Jul 7, 2018

numpy.core.tests.test_multiarray.TestDot.test_accelerate_framework_sgemv_fix looks like accuracy deviation, not a failure.
On the CPU type note - can you try to setting OPENBLAS_CORETYPE= to older architectures than your CPU (some hints here: https://github.com/xianyi/OpenBLAS/tree/develop/kernel/x86_64 )
Indeed slight accuracy variation is expected from different processing sequence in accelerated BLAS and reference BLAS.

@martin-frbg

This comment has been minimized.

Copy link
Collaborator

commented Jul 7, 2018

Seeing that DYNAMIC_ARCH is involved, there is a (small) chance that this may be from running on one of the older cpus that got moved to the DYNAMIC_OLDER list in 0.3.1 (e.g. if the numpy test was run on an ATOM or OPTERON target), and fallback to the ancient Prescott kernels from libgoto may have exposed an old bug.

@beew

This comment has been minimized.

Copy link
Author

commented Jul 8, 2018

@martin-frbg
This is a 7th generation intel core i 7 cpu (Kabylake) so it is very new. The DYNAMIC_ARCH is necessary because otherwise I couldn't start the terminal and a few other apps. I forgot the error message since this happened long time ago and setting DYNAMIC_ARCH=1 fixed it (something about undefined gotoblas iirc)

This is because I have export LD_PRELOAD= /usr/lib/libblas.so in my .profile.

As mentioned before I switch between different blases and dynamic switching apparently doesn't work for octave, numpy and R unless blas is pre loaded. Merely update-alternatives doesn't work, many online tutorials are wrong, for example ldd or in numpy's case numpy.show_config() merely shows which blas the program is compiled against, not which one is being used at runtime, but actually monitoring the process with lsof you can see which blas is actually used.

There is no difference if not set LD_PRELOAD at runtime even with different alternatives (same blas is used). I see different blas is loaded along with performance differences only if I do LD_PRELOAD AND update-alterrantives (for my tests intel-mkl is slightly faster than openblas, both beat a custom built atlas by big margin) so instead of having to manually LD_PRELOAD each time I just put it in .profile.

@beew

This comment has been minimized.

Copy link
Author

commented Jul 8, 2018

@sandwichmaker
openblas-dev + USE_OMPENMP=0 tests segfault and python3 crashed

OpenBLAS : Program will terminate because you tried to start too many threads.
Segmentation fault

openblas-dev + USE_OPENMP=1+OMP_NUM_THREADS=1 all tests passed.

@beew

This comment has been minimized.

Copy link
Author

commented Jul 8, 2018

For some additional info openblas0.3.1 also segfaulted a lot of opencv tests while openblas0.3.0 passed (compiled with make PREFIX=/opt/openblas-mpi NO_AFFINITY=1 DYNAMIC_ARCH=1 USE_OPENMP=1 -j8)

Since the opencv tests are somewhat of a mess anyway I don't bother to report that. It seems that the same underlying issues are responsible.

@martin-frbg

This comment has been minimized.

Copy link
Collaborator

commented Jul 8, 2018

Not reproducible with current develop on my Kaby Lake system (though this has python 3.4.5/numpy 1.9.3). Note 0.3.1 has a known problem with thread allocation that should be fixed in develop.

@martin-frbg

This comment has been minimized.

Copy link
Collaborator

commented Jul 10, 2018

Problem now reproduced with a source install of current numpy, and bisected to a399d00 which is PR #1625 "Further improvements to memory.c".
@oon3m0oo @sandwichmaker as the problem appears to be specific to OPENMP builds, it seems the assumption of there only ever being one allocating thread in that case may be not entirely correct...

@martin-frbg martin-frbg added this to the 0.3.2 milestone Jul 10, 2018

@martin-frbg

This comment has been minimized.

Copy link
Collaborator

commented Jul 10, 2018

Indeed removing all the "!USE_OPENMP" special cases from the new thread memory handling fixes the problem (performance impact not studied).

@oon3m0oo

This comment has been minimized.

Copy link
Contributor

commented Jul 10, 2018

@beew

This comment has been minimized.

Copy link
Author

commented Jul 11, 2018

@martin-frbg

Problem now reproduced with a source install of current numpy

Yes, numpy has to be installed from source. Sorry for forgetting to mention that.

Otherwise I think it links against its own blas if update alternatives is set to openblas. I tried to reproduce this on another machine with numpy installed with pip3 (OS is Ubuntu 18.04, openblas compiled from source and set up update-alternatives to switch between mkl and openblas in same manner) Doing numpy.test() gives this curious error

FAIL: numpy.linalg.tests.test_build.TestF77Mismatch.test_lapack
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/nose/case.py", line 198, in runTest
    self.test(*self.arg)
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/nose_tools/decorators.py", line 147, in skipper_func
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/numpy/linalg/tests/test_build.py", line 54, in test_lapack
    information.""")
  File "/usr/local/lib/python3.6/dist-packages/numpy/testing/nose_tools/utils.py", line 93, in assert_
    raise AssertionError(smsg)
AssertionError: Both g77 and gfortran runtimes linked in lapack_lite ! This is likely to
cause random crashes and wrong results. See numpy INSTALL.txt for more
information.

----------------------------------------------------------------------
Ran 6821 tests in 55.320s

FAILED (KNOWNFAIL=14, SKIP=15, failures=1)

It seems that it is because libfortran got linked twice, once from system openblas once from numpy's(?)

ldd /usr/local/lib/python3.6/dist-packages/numpy/linalg/lapack_lite.cpython-36m-x86_64-linux-gnu.so
	...
	libgfortran.so.4 => /usr/lib/x86_64-linux-gnu/libgfortran.so.4 (0x00007f221baa9000)
	...
	libgfortran-ed201abd.so.3.0.0 => /usr/local/lib/python3.6/dist-packages/numpy/linalg/../.libs/libgfortran-ed201abd.so.3.0.0 (0x00007f221b57f000)

This doesn't happen if update-alternatives is set to intel mkl

@martin-frbg

This comment has been minimized.

Copy link
Collaborator

commented Jul 11, 2018

Looks like it. And it complains specifically because one of the two - numpy I assume - was built with the old fortran77-only version of the GNU compiler.
Quickest temporary fix for the original problem would be to change all occurences of USE_OPENMP in driver/others/memory.c to something that is not actually defined, like USE_OPENMPX

@beew

This comment has been minimized.

Copy link
Author

commented Jul 11, 2018

Is there any benchmark test that shows openblas 0.31 is actually faster than 0.30? I just did the basic random matrix multiplications and I don't see any difference.

@martin-frbg

This comment has been minimized.

Copy link
Collaborator

commented Jul 11, 2018

Some numbers were in #1618, #1625 (the data from #1624 is less useful as it combines both the memory.c rewrite and the introduction of AVX512 SGEMM). I expect some of the gain may already have been lost due to required corrections, also the main effect should be on pthread builds without OPENMP.

@oon3m0oo

This comment has been minimized.

Copy link
Contributor

commented Jul 11, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.