Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TEST: 1.10.x + blas variants #224

Closed
wants to merge 5 commits into from

Conversation

h-vetinari
Copy link
Member

Following the same scheme as #223, #199, #172, #153 & #130. Should not be merged for the same reasons as #153.

@conda-forge-webservices
Copy link

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

@h-vetinari
Copy link
Member Author

h-vetinari commented Jan 27, 2023

Scipy 1.10.0

TL;DR: Very similar to 1.9.3, but slight improvements

From 32 failures out of 96 runs for 1.9.3, we're now at 28 failures out of 96 CI runs

Notable

The good news:

  • the osx + accelerate failures are all very minor tolerance violations
  • linux + aarch64 failure gone

The bad news

Restrictions

  • aarch64/ppc64le only run tests with label='fast', not the full test suite
  • the following test skips:
# these multithreading tests occasionally hang in emulation
{% set tests_to_skip = tests_to_skip + " or test_immediate_updating" %}         # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or test_mixed_threads_processes" %}    # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or test_multiprocess[fft]" %}          # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test__differential_evolution and test_parallel)" %}  # [aarch64 or ppc64le]
# on the slowest agents, these tests take more than 20min in emulation
{% set tests_to_skip = tests_to_skip + " or (test_lsq_linear and test_large_rank_deficient)" %}   # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_iterative and test_precond_inverse[case1])" %}  # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_optimize and TestBrute and test_workers)" %}    # [aarch64 or ppc64le]
# tests that run into timeouts (set in scipy test suite) in emulation
{% set tests_to_skip = tests_to_skip + " or (test_propack and test_examples)" %}                  # [aarch64 or ppc64le]

Details

lib for 1.9.3 now updated
version
updated
build
scipy 1.9.3 1.10.0 X
numpy 1.24.1 1.24.1
libblas 3.9.0-16 3.9.0-16
blis 0.9.0-0 0.9.0-0
openblas 0.3.21-pthreads-3 0.3.21-pthreads-3
mkl 2022.1.0-915 2022.1.0-915
netlib 3.9.0-5 3.9.0-5
pypy not used not used
pythran 0.12.1-0 0.12.1-0
qemu-user-static 7.2.0-1 7.2.0-1

variant 1.9.3 1.10.0
linux + x64 + mkl + avx512 1 test failure;
scipy/scipy#15533
remains
linux + x64 + openblas + non-avx512 1 test failure
scipy/scipy#16927
remains
linux + aarch64 + openblas 1 test failure passes 🥳
linux + ppc64le test suite disabled due to emulation problems looks like this might
finally get solved 🥳
osx + x64 + accelerate 7 test failures 6 failures
osx + x64 + blis 1 test failure
scipy/scipy#16927
remains
win + mkl + avx512 1 test failure;
scipy/scipy#15533
remains
win + openblas 7 test failures
scipy/scipy#17869
same failures remain

variant accelerate blis mkl
w/ avx512
mkl
w/o avx512
netlib openblas
w/ avx512
openblas
w/o avx512
sum**
linux / x64 ✔️ ✔️ ✔️ ✔️ 8F
linux / aarch64 ✔️ ✔️ -
linux / ppc64le -
osx / arm -
osx / x64 ✔️ ✔️ ✔️ 8F
win / x64 ✔️ ✔️ ✔️ 12F
sum** 4F 4F 8F - - 4F 8F 28F

** sum of Segfaults (S), Failures (F), resp. Timeouts (T); out of a total of 96 CI combinations being tested

Build logs:
Azure

linux + x64 + mkl + avx512: 1 failure
=========================== short test summary info ============================
FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab] - AssertionError: 
Items are not equal:
 ACTUAL: -11
 DESIRED: 0
= 1 failed, 50141 passed, 2295 skipped, 255 xfailed, 9 xpassed, 268 warnings in 960.42s (0:16:00) =
linux + x64 + openblas + non-avx512: 1 failure
=========================== short test summary info ============================
FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
The list of emitted warnings is: [].
= 1 failed, 50140 passed, 2296 skipped, 253 xfailed, 11 xpassed, 270 warnings in 1133.54s (0:18:53) =
osx + x64 + accelerate: 6 failures
=========================== short test summary info ============================
FAILED interpolate/tests/test_rbfinterp.py::TestRBFInterpolatorNeighbors20::test_pickleable - AssertionError: 
Arrays are not equal

Mismatched elements: 1 / 50 (2%)
Max absolute difference: 5.55111512e-17
Max relative difference: 2.02452314e-16
 x: array([-0.238461,  0.257024, -0.811244,  0.295313,  0.132623,  0.2038  ,
       -0.99615 ,  0.29755 , -0.019889,  0.230791, -0.516801,  0.279867,
        0.227767,  0.178071, -0.172756,  0.251327, -0.406818,  0.271919,...
 y: array([-0.238461,  0.257024, -0.811244,  0.295313,  0.132623,  0.2038  ,
       -0.99615 ,  0.29755 , -0.019889,  0.230791, -0.516801,  0.279867,
        0.227767,  0.178071, -0.172756,  0.251327, -0.406818,  0.271919,...

FAILED linalg/tests/test_cython_blas.py::TestWfuncPointers::test_float_args

FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_v0 - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 86 / 100 (86%)
Max absolute difference: 2.22044605e-16
Max relative difference: 2.02120457e-15
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...

FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 25 / 100 (25%)
Max absolute difference: 5.55111512e-17
Max relative difference: 5.62192122e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...

FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[0] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 38 / 100 (38%)
Max absolute difference: 5.55111512e-17
Max relative difference: 5.62192122e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...

FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[1] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 15 / 100 (15%)
Max absolute difference: 2.77555756e-17
Max relative difference: 2.84357857e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
= 6 failed, 50122 passed, 2307 skipped, 255 xfailed, 11 xpassed, 269 warnings in 908.74s (0:15:08) =
osx + x64 + blis: 1 failure
=========================== short test summary info ============================
FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
The list of emitted warnings is: [].
= 1 failed, 50127 passed, 2307 skipped, 255 xfailed, 11 xpassed, 268 warnings in 907.17s (0:15:07) =
windows + mkl + avx512: 1 failure

=========================== short test summary info ===========================
FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab]
= 1 failed, 50112 passed, 2324 skipped, 255 xfailed, 9 xpassed, 269 warnings in 1256.40s (0:20:56) =
windows + openblas: 7 failures
=========================== short test summary info ===========================
FAILED linalg/tests/test_blas.py::TestFBLAS2Simple::test_syr_her - AssertionE...
FAILED linalg/tests/test_decomp.py::TestEig::test_bad_geneig - AssertionError: 
FAILED linalg/tests/test_decomp_ldl.py::test_simple - AssertionError: 
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex64]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex128]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex64]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex128]
= 7 failed, 50105 passed, 2325 skipped, 255 xfailed, 9 xpassed, 271 warnings in 1028.16s (0:17:08) =

@h-vetinari
Copy link
Member Author

Paused the runs here until we figure out #232.

@h-vetinari
Copy link
Member Author

h-vetinari commented Apr 25, 2023

Scipy 1.10.1

TL;DR: Now with PyPy & PPC, import error on linux+netlib, new failures

From 28 failures out of 96 runs for 1.10.0, we're now at 66 failures out of 138 CI runs

Notable

The good news:

  • [as for 1.10.0] for osx + accelerate, 6/7 failures are very minor tolerance violations

The bad news

  • An import error due to a missing symbol for netlib on all linux arches; this has a big blas(t) radius because it causes an import error!
  • [as for 1.10.0] persistent errors on win+openblas, unix+x64 and mkl+avx512

Restrictions

  • test on aarch64/ppc64le resp. for PyPy only run tests with label='fast', not the full test suite
  • the following test skips:
# skip a test that fails with MKL + AVX512 (non-AVX512 passes), scipy/scipy#15533
{% set tests_to_skip = tests_to_skip + " or test_x0_equals_Mb[bicgstab]" %}
# scipy/scipy#16927
{% set tests_to_skip = tests_to_skip + " or test_failure_to_run_iterations" %}  # [linux]
# on the slowest agents, these tests take more than 20min in emulation
{% set tests_to_skip = tests_to_skip + " or (test_lsq_linear and test_large_rank_deficient)" %}   # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_iterative and test_precond_inverse[case1])" %}  # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_optimize and TestBrute and test_workers)" %}    # [aarch64 or ppc64le]
# tests that run into timeouts (set in scipy test suite) in emulation
{% set tests_to_skip = tests_to_skip + " or (test_propack and test_examples)" %}                  # [aarch64 or ppc64le]
# occasionally runs out of memory on windows + pypy
{% set tests_to_skip = tests_to_skip + " or test_bootstrap_against_" %}                           # [(python_impl == "pypy") and win]
# very slow tests for pypy in emulation
[several]

Details

lib for 1.10.0 now updated
version
updated
build
scipy 1.10.0 1.10.1 X
numpy 1.24.1 1.24.3 X
libblas 3.9.0-16 3.9.0-16
blis 0.9.0-0 0.9.0-0
openblas 0.3.21-pthreads-3 0.3.21-pthreads-3
mkl 2022.1.0-915 2022.2.1-16997 (linux)
2022.2.1-16952 (osx)
2022.1.0-874 (win)
X
netlib 3.9.0-5 3.9.0-5
pypy 7.3.9-7
(for 1.8.1)
7.3.11-0 X
pythran 0.12.1-0 0.12.1-0
qemu-user-static 7.2.0-1 7.2.0-1

variant 1.10.0 now
linux + x64 + mkl + avx512 1 test failure;
scipy/scipy#15533
remains
linux + x64 + openblas + non-avx512 1 test failure
scipy/scipy#16927
remains
linux + netlib - missing symbol cblas_cdotc_sub;
causes ImportError! ☢️
linux + ppc64le test suite disabled due to emulation problems runs through on openblas🥳
missing symbol on netlib
osx + x64 + accelerate 6 test failures 7 test failures
osx + x64 + blis 1 test failure
scipy/scipy#16927
remains
osx + x64 + mkl + CPython - 13-14 test failures
osx + x64 + mkl + PyPy - 2-3 test failures
win + mkl + avx512 1 test failure;
scipy/scipy#15533
remains
win + openblas 7 test failures
scipy/scipy#17869
same failures remain

variant accelerate blis mkl
w/ avx512
mkl
w/o avx512
netlib openblas
w/ avx512
openblas
w/o avx512
sum**
linux / x64 ✔️ ✔️ ✔️ 18F
linux / aarch64 ✔️ 6F
linux / ppc64le ✔️ 6F
osx / arm -
osx / x64 ✔️ ✔️ 18F
win / x64 ✔️ ✔️ ✔️ 18F
sum** 6F 6F 12F 6F 18F 6F 12F 66F

** sum of Segfaults (S), Failures (F), resp. Timeouts (T); out of a total of 138 CI combinations being tested

Build logs:
Azure

(linux or win) + x64 + mkl + avx512: 1 failure
=========================== short test summary info ============================
FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab] - AssertionError: 
Items are not equal:
 ACTUAL: -11
 DESIRED: 0
= 1 failed, 50177 passed, 2294 skipped, 252 xfailed, 10 xpassed, 268 warnings in 758.88s (0:12:38) =
linux + netlib: ImportError! ☢️
import: 'scipy.cluster'
Traceback (most recent call last):
  [...]
    from . import _iterative
ImportError: $PREFIX/lib/python3.10/site-packages/scipy/sparse/linalg/_isolve/_iterative.cpython-310-x86_64-linux-gnu.so: undefined symbol: cblas_cdotc_sub
linux + x64 + openblas + non-avx512: 1 failure
=========================== short test summary info ============================
FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
The list of emitted warnings is: [].
= 1 failed, 50177 passed, 2294 skipped, 250 xfailed, 12 xpassed, 268 warnings in 1219.20s (0:20:19) =
osx + x64 + accelerate: 7 failures
=========================== short test summary info ============================
FAILED linalg/tests/test_cython_blas.py::TestWfuncPointers::test_float_args
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_v0 - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 5 / 100 (5%)
Max absolute difference: 2.77555756e-17
Max relative difference: 3.0222348e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 25 / 100 (25%)
Max absolute difference: 4.16333634e-17
Max relative difference: 4.25664195e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[0] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 70 / 100 (70%)
Max absolute difference: 5.55111512e-17
Max relative difference: 5.8956122e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[1] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 49 / 100 (49%)
Max absolute difference: 4.16333634e-17
Max relative difference: 3.80861356e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[random_state2] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 32 / 100 (32%)
Max absolute difference: 5.55111512e-17
Max relative difference: 5.24318491e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[random_state3] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 28 / 100 (28%)
Max absolute difference: 4.16333634e-17
Max relative difference: 4.38230693e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
= 7 failed, 50158 passed, 2305 skipped, 252 xfailed, 12 xpassed, 265 warnings in 1150.50s (0:19:10) =
osx + x64 + blis: 1 failure
=========================== short test summary info ============================
FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
The list of emitted warnings is: [].
= 1 failed, 38011 passed, 2277 skipped, 139 xfailed, 11 xpassed, 67 warnings in 2058.33s (0:34:18) =
osx + x64 + mkl + CPython: 13-14 failures
=========================== short test summary info ============================
FAILED sparse/linalg/tests/test_propack.py::test_examples[False-complex64]
FAILED sparse/linalg/tests/test_propack.py::test_examples[True-complex64]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex128-array]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex128-csc_matrix]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex128-csr_matrix]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex64-array]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex64-csr_matrix]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex128-array]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex128-csr_matrix]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex64-array]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex64-csc_matrix]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex64-csr_matrix]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[SM-True-complex128-csr_matrix]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[SM-True-complex64-array]
= 14 failed, 50135 passed, 2304 skipped, 253 xfailed, 11 xpassed, 265 warnings in 1244.19s (0:20:44) =
osx + x64 + mkl + PyPy: 2-3 failures
=========================== short test summary info ============================
FAILED linalg/tests/test_decomp_cossin.py::test_cossin[True-40-12-20-float32] - AssertionError: 
Not equal to tolerance rtol=0, atol=1.19209e-06

Mismatched elements: 132 / 1600 (8.25%)
Max absolute difference: 1.2262092
Max relative difference: 2.0007246
 x: array([[ 0.088203, -0.328941, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048775,  0.223086, ...,  0.      ,  0.      ,...
 y: array([[ 0.088203, -0.328941, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048776,  0.223085, ...,  0.      ,  0.      ,...
FAILED linalg/tests/test_decomp_cossin.py::test_cossin[False-40-12-20-float32] - AssertionError: 
Not equal to tolerance rtol=0, atol=1.19209e-06

Mismatched elements: 228 / 1600 (14.2%)
Max absolute difference: 1.2262108
Max relative difference: 2.0003054
 x: array([[ 0.088202, -0.328941, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048775,  0.223086, ...,  0.      ,  0.      ,...
 y: array([[ 0.088203, -0.328942, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048775,  0.223085, ...,  0.      ,  0.      ,...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_F::test_sqr_p_row
= 3 failed, 38009 passed, 2277 skipped, 140 xfailed, 10 xpassed, 67 warnings in 2631.58s (0:43:51) =
windows + openblas: 7 failures
=========================== short test summary info ===========================
FAILED linalg/tests/test_blas.py::TestFBLAS2Simple::test_syr_her - AssertionE...
FAILED linalg/tests/test_decomp.py::TestEig::test_bad_geneig - AssertionError: 
FAILED linalg/tests/test_decomp_ldl.py::test_simple - AssertionError: 
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex64]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex128]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex64]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex128]
= 7 failed, 50138 passed, 2327 skipped, 252 xfailed, 10 xpassed, 273 warnings in 1743.42s (0:29:03) =

@h-vetinari
Copy link
Member Author

Ah well, turns out my rebase wasn't that great after all:

../scipy/sparse/linalg/_propack/meson.build:93:12: ERROR: Unknown variable "cargs_propack".

@h-vetinari
Copy link
Member Author

h-vetinari commented May 5, 2023

Seems something about those patches isn't distutils- and/or windows-compatible; but we'll just exclude them on windows for main then...

%SRC_DIR%\scipy\_build_utils\src\npy_cblas.h(9): fatal error C1083: Cannot open include file: 'numpy/npy_common.h': No such file or directory

@h-vetinari
Copy link
Member Author

h-vetinari commented May 6, 2023

Scipy 1.10.1 (distutils)

This already ran a few days ago to verify a suspicion that scipy/scipy#18371 had something to do with the switch to meson. While looking at the results of testing PROPRACK in #200, I noticed some familiar failures and ended up writing this up because it's relevant.

TL;DR: Switch to meson broke a bunch of things

From 66 failures out of 138 runs for 1.10.1 built with meson (on unix), we're now at 44 failures out of 138 CI runs

Notable

Restrictions & Library Versions

as for 1.10.1


variant 1.10.1 now
linux + x64 + mkl + avx512 1 test failure;
scipy/scipy#15533
remains
linux + x64 + openblas + non-avx512 1 test failure
scipy/scipy#16927
remains
linux + netlib missing symbol cblas_cdotc_sub;
causes ImportError! ☢️
gone! 🥳
osx + x64 + accelerate 7 test failures remain
osx + x64 + blis 1 test failure
scipy/scipy#16927
remains
osx + x64 + mkl + CPython 13-14 test failures gone! 🥳
osx + x64 + mkl + PyPy 2-3 test failures remain
win + mkl + avx512 1 test failure;
scipy/scipy#15533
remains
win + openblas 7 test failures
scipy/scipy#17869
remain

variant accelerate blis mkl
w/ avx512
mkl
w/o avx512
netlib openblas
w/ avx512
openblas
w/o avx512
sum**
linux / x64 ✔️ ✔️ ✔️ ✔️ 12F
linux / aarch64 ✔️ ✔️ -
linux / ppc64le ✔️ ✔️ -
osx / arm -
osx / x64 ✔️ (CPython)
❌ (PyPy)
✔️ ✔️ 14F
win / x64 ✔️ ✔️ ✔️ 18F
sum** 6F 6F 12F 2F - 6F 12F 44F

** sum of Segfaults (S), Failures (F), resp. Timeouts (T); out of a total of 138 CI combinations being tested

Build logs:
Azure

(linux or win) + x64 + mkl + avx512: 1 failure
=========================== short test summary info ============================
FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab] - AssertionError: 
Items are not equal:
 ACTUAL: -11
 DESIRED: 0
= 1 failed, 50177 passed, 2294 skipped, 252 xfailed, 10 xpassed, 268 warnings in 758.88s (0:12:38) =
linux + x64 + openblas + non-avx512: 1 failure
=========================== short test summary info ============================
FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
The list of emitted warnings is: [].
= 1 failed, 50177 passed, 2294 skipped, 250 xfailed, 12 xpassed, 268 warnings in 1219.20s (0:20:19) =
osx + x64 + accelerate: 7 failures
=========================== short test summary info ============================
FAILED linalg/tests/test_cython_blas.py::TestWfuncPointers::test_float_args
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_v0 - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 5 / 100 (5%)
Max absolute difference: 2.77555756e-17
Max relative difference: 3.0222348e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 25 / 100 (25%)
Max absolute difference: 4.16333634e-17
Max relative difference: 4.25664195e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[0] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 70 / 100 (70%)
Max absolute difference: 5.55111512e-17
Max relative difference: 5.8956122e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[1] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 49 / 100 (49%)
Max absolute difference: 4.16333634e-17
Max relative difference: 3.80861356e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[random_state2] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 32 / 100 (32%)
Max absolute difference: 5.55111512e-17
Max relative difference: 5.24318491e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[random_state3] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 28 / 100 (28%)
Max absolute difference: 4.16333634e-17
Max relative difference: 4.38230693e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
= 7 failed, 50158 passed, 2305 skipped, 252 xfailed, 12 xpassed, 265 warnings in 1150.50s (0:19:10) =
osx + x64 + blis: 1 failure
=========================== short test summary info ============================
FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
The list of emitted warnings is: [].
= 1 failed, 38011 passed, 2277 skipped, 139 xfailed, 11 xpassed, 67 warnings in 2058.33s (0:34:18) =
osx + x64 + mkl + pypy: 2-3 failures
=========================== short test summary info ============================
FAILED linalg/tests/test_decomp_cossin.py::test_cossin[True-40-12-20-float32] - AssertionError: 
Not equal to tolerance rtol=0, atol=1.19209e-06

Mismatched elements: 132 / 1600 (8.25%)
Max absolute difference: 1.2262092
Max relative difference: 2.0007246
 x: array([[ 0.088203, -0.328941, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048775,  0.223086, ...,  0.      ,  0.      ,...
 y: array([[ 0.088203, -0.328941, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048776,  0.223085, ...,  0.      ,  0.      ,...
FAILED linalg/tests/test_decomp_cossin.py::test_cossin[False-40-12-20-float32] - AssertionError: 
Not equal to tolerance rtol=0, atol=1.19209e-06

Mismatched elements: 228 / 1600 (14.2%)
Max absolute difference: 1.2262108
Max relative difference: 2.0003054
 x: array([[ 0.088202, -0.328941, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048775,  0.223086, ...,  0.      ,  0.      ,...
 y: array([[ 0.088203, -0.328942, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048775,  0.223085, ...,  0.      ,  0.      ,...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_F::test_sqr_p_row
= 3 failed, 38009 passed, 2277 skipped, 140 xfailed, 10 xpassed, 67 warnings in 2631.58s (0:43:51) =
windows + openblas: 7 failures
=========================== short test summary info ===========================
FAILED linalg/tests/test_blas.py::TestFBLAS2Simple::test_syr_her - AssertionE...
FAILED linalg/tests/test_decomp.py::TestEig::test_bad_geneig - AssertionError: 
FAILED linalg/tests/test_decomp_ldl.py::test_simple - AssertionError: 
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex64]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex128]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex64]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex128]
= 7 failed, 50138 passed, 2327 skipped, 252 xfailed, 10 xpassed, 273 warnings in 1743.42s (0:29:03) =

@h-vetinari
Copy link
Member Author

h-vetinari commented May 6, 2023

Scipy 1.10.1 (meson, with patches)

Now again with meson after the backported patches from #233

TL;DR: OSX+MKL still broken on meson

From 66 failures out of 138 runs for 1.10.1 built with meson (on unix) and 44 failures when built with distutils, we're now at 48 failures out of 138 CI runs

Notable

  • OSX+MKL has the same errors as before the patching, but at least the failures remain stable across CPython python versions (in contrast to what happens when enabling PROPACK, see WIP: Re-enable PROPACK tests #200; a fact that baffles me to no end)

Restrictions & Library Versions

as for 1.10.1


variant 1.10.1 now
linux + x64 + mkl + avx512 1 test failure;
scipy/scipy#15533
remains
linux + x64 + openblas + non-avx512 1 test failure
scipy/scipy#16927
remains
linux + netlib missing symbol cblas_cdotc_sub;
causes ImportError! ☢️
gone! 🥳
osx + x64 + accelerate 7 test failures remain
osx + x64 + blis 1 test failure
scipy/scipy#16927
remains
osx + x64 + mkl + CPython 13-14 test failures remains
in contrast to distutils build
osx + x64 + mkl + PyPy 2-3 test failures remain
win + mkl + avx512 1 test failure;
scipy/scipy#15533
remains
win + openblas 7 test failures
scipy/scipy#17869
remain

variant accelerate blis mkl
w/ avx512
mkl
w/o avx512
netlib openblas
w/ avx512
openblas
w/o avx512
sum**
linux / x64 ✔️ ✔️ ✔️ ✔️ 12F
linux / aarch64 ✔️ ✔️ -
linux / ppc64le ✔️ ✔️ -
osx / arm -
osx / x64 ✔️ ✔️ 18F
win / x64 ✔️ ✔️ ✔️ 18F
sum** 6F 6F 12F 6F - 6F 12F 48F

** sum of Segfaults (S), Failures (F), resp. Timeouts (T); out of a total of 138 CI combinations being tested

Build logs:
Azure

(linux or win) + x64 + mkl + avx512: 1 failure
=========================== short test summary info ============================
FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab] - AssertionError: 
Items are not equal:
 ACTUAL: -11
 DESIRED: 0
= 1 failed, 50177 passed, 2294 skipped, 252 xfailed, 10 xpassed, 268 warnings in 758.88s (0:12:38) =
linux + x64 + openblas + non-avx512: 1 failure
=========================== short test summary info ============================
FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
The list of emitted warnings is: [].
= 1 failed, 50177 passed, 2294 skipped, 250 xfailed, 12 xpassed, 268 warnings in 1219.20s (0:20:19) =
osx + x64 + accelerate: 7 failures
=========================== short test summary info ============================
FAILED linalg/tests/test_cython_blas.py::TestWfuncPointers::test_float_args
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_v0 - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 5 / 100 (5%)
Max absolute difference: 2.77555756e-17
Max relative difference: 3.0222348e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 25 / 100 (25%)
Max absolute difference: 4.16333634e-17
Max relative difference: 4.25664195e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[0] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 70 / 100 (70%)
Max absolute difference: 5.55111512e-17
Max relative difference: 5.8956122e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[1] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 49 / 100 (49%)
Max absolute difference: 4.16333634e-17
Max relative difference: 3.80861356e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[random_state2] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 32 / 100 (32%)
Max absolute difference: 5.55111512e-17
Max relative difference: 5.24318491e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_svd_random_state_2[random_state3] - AssertionError: 
Arrays are not equal
item=0

Mismatched elements: 28 / 100 (28%)
Max absolute difference: 4.16333634e-17
Max relative difference: 4.38230693e-16
 x: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
 y: array([[-0.109858],
       [-0.106207],
       [-0.108685],...
= 7 failed, 50158 passed, 2305 skipped, 252 xfailed, 12 xpassed, 265 warnings in 1150.50s (0:19:10) =
osx + x64 + blis: 1 failure
=========================== short test summary info ============================
FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_failure_to_run_iterations - Failed: DID NOT WARN. No warnings of type (<class 'UserWarning'>,) were emitted.
The list of emitted warnings is: [].
= 1 failed, 38011 passed, 2277 skipped, 139 xfailed, 11 xpassed, 67 warnings in 2058.33s (0:34:18) =
osx + x64 + mkl + CPython: 13-14 failures
=========================== short test summary info ============================
FAILED sparse/linalg/tests/test_propack.py::test_examples[False-complex128]
FAILED sparse/linalg/tests/test_propack.py::test_examples[False-complex64]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex128-array]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex128-csr_matrix]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex64-array]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-False-complex64-csr_matrix]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex128-array]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex128-csr_matrix]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex64-array]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex64-csc_matrix]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[LM-True-complex64-csr_matrix]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[SM-True-complex64-array]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[SM-True-complex64-csc_matrix]
FAILED sparse/linalg/tests/test_propack.py::test_svdp[SM-True-complex64-csr_matrix]
= 14 failed, 50142 passed, 2304 skipped, 253 xfailed, 11 xpassed, 265 warnings in 1276.44s (0:21:16) =
osx + x64 + mkl + pypy: 2-3 failures
=========================== short test summary info ============================
FAILED linalg/tests/test_decomp_cossin.py::test_cossin[True-40-12-20-float32] - AssertionError: 
Not equal to tolerance rtol=0, atol=1.19209e-06

Mismatched elements: 132 / 1600 (8.25%)
Max absolute difference: 1.2262092
Max relative difference: 2.0007246
 x: array([[ 0.088203, -0.328941, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048775,  0.223086, ...,  0.      ,  0.      ,...
 y: array([[ 0.088203, -0.328941, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048776,  0.223085, ...,  0.      ,  0.      ,...
FAILED linalg/tests/test_decomp_cossin.py::test_cossin[False-40-12-20-float32] - AssertionError: 
Not equal to tolerance rtol=0, atol=1.19209e-06

Mismatched elements: 228 / 1600 (14.2%)
Max absolute difference: 1.2262108
Max relative difference: 2.0003054
 x: array([[ 0.088202, -0.328941, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048775,  0.223086, ...,  0.      ,  0.      ,...
 y: array([[ 0.088203, -0.328942, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048775,  0.223085, ...,  0.      ,  0.      ,...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_F::test_sqr_p_row
= 3 failed, 38009 passed, 2277 skipped, 140 xfailed, 10 xpassed, 67 warnings in 2631.58s (0:43:51) =
windows + openblas: 7 failures
=========================== short test summary info ===========================
FAILED linalg/tests/test_blas.py::TestFBLAS2Simple::test_syr_her - AssertionE...
FAILED linalg/tests/test_decomp.py::TestEig::test_bad_geneig - AssertionError: 
FAILED linalg/tests/test_decomp_ldl.py::test_simple - AssertionError: 
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex64]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex128]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex64]
FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex128]
= 7 failed, 50138 passed, 2327 skipped, 252 xfailed, 10 xpassed, 273 warnings in 1743.42s (0:29:03) =

rgommers added a commit to rgommers/scipy that referenced this pull request May 8, 2023
xref conda-forge/scipy-feedstock#224 for
these test failures. The `test_failure_to_run_iterations` issue
that is the only failure for 3 OS/BLAS combinations was already
fixed in `main`, so with this PR we should be close to most jobs
passing again.

[skip cirrus] [skip circle]
@rgommers
Copy link
Contributor

rgommers commented May 8, 2023

All of the entries with a single failure are from the same test, test_failure_to_run_iterations, which is already fixed in SciPy's main branch.

The precision issues in the macOS/Accelerate tests should be fixed by scipy/scipy#18442. Then there is one left:

FAILED linalg/tests/test_cython_blas.py::TestWfuncPointers::test_float_args

that is a crash, probably a faulty G77 ABI wrapper. If that happens again after the current round of -use-g77-abi fixes, I'll investigate more.

rgommers added a commit to rgommers/scipy that referenced this pull request May 8, 2023
xref conda-forge/scipy-feedstock#224 for
these test failures. The `test_failure_to_run_iterations` issue
that is the only failure for 3 OS/BLAS combinations was already
fixed in `main`, so with this PR we should be close to most jobs
passing again.

[skip cirrus] [skip circle]
@h-vetinari
Copy link
Member Author

Closing this as obsolete

@h-vetinari h-vetinari closed this Mar 19, 2024
@h-vetinari h-vetinari deleted the 1.10_blas_vars branch March 19, 2024 05:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants