TEST: 1.11.x + blas variants #242

h-vetinari · 2023-06-26T07:27:23Z

Following the same scheme as #224, #223, #199, #172, #153 & #130. Should not be merged for the same reasons as #153.

PS. Just noticed that it's the third anniversary of my BLAS testing efforts (#130 opened exactly 3 years ago). 🥳

conda-forge-webservices · 2023-06-26T07:27:27Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

conda-forge-webservices · 2023-06-26T07:38:37Z

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipe) and found some lint.

Here's what I've got...

For recipe:

Failed to even lint the recipe, probably because of a conda-smithy bug 😢. This likely indicates a problem in your meta.yaml, though. To get a traceback to help figure out what's going on, install conda-smithy and run conda smithy recipe-lint . from the recipe directory.

conda-forge-webservices · 2023-06-26T07:42:42Z

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

h-vetinari · 2023-06-27T09:42:22Z

Scipy 1.11.0

TL;DR: slow & steady improvements continue

From 48 failures out of 138 runs for 1.10.1, we're now at 22 failures out of 93 CI runs

Notable

Dropped CPython & PyPy 3.8
At first there were 24 failures; two disappeared upon rerun

Good News

Several failures resolved
A long-standing failure on AVX512 should be fixed on main, but the patch is too big to backport

Bad News

Some new failures (flaky crash on blis, hang of PyPy on aarch, another PyPy failure due to apparent memory pressure)

Restrictions

test on aarch64/ppc64le resp. for PyPy only run tests with label='fast', not the full test suite
the following test skips:

# on the slowest agents, these tests take more than 20min in emulation
{% set tests_to_skip = tests_to_skip + " or (test_lsq_linear and test_large_rank_deficient)" %}   # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_iterative and test_precond_inverse[case1])" %}  # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_optimize and TestBrute and test_workers)" %}    # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (TestLevyStable and test_location_scale[pdf-1])" %}   # [aarch64 or ppc64le]
# tests that run into timeouts (set in scipy test suite) in emulation
{% set tests_to_skip = tests_to_skip + " or (test_propack and test_examples)" %}                  # [aarch64 or ppc64le]
# occasionally runs out of memory on windows + pypy
{% set tests_to_skip = tests_to_skip + " or test_bootstrap_against_" %}                           # [(python_impl == "pypy") and win]
{% set tests_to_skip = tests_to_skip + " or test_more_converged" %}                               # [(python_impl == "pypy") and win]
# very slow tests for pypy in emulation
[several]

Details

lib	for 1.10.1	now	updated version	updated build
`scipy`	`1.10.1`	`1.11.0`	X
`numpy`	`1.24.3`	`1.25.0`	X
`libblas`	`3.9.0-16`	`3.9.0-17`		X
`blis`	`0.9.0-0`	`0.9.0-1`		X
`openblas`	`0.3.21-pthreads-3`	`0.3.23-pthreads-0`	X
`mkl`	`2022.2.1-16997` (linux) `2022.2.1-16952` (osx) `2022.1.0-874` (win)	`2022.2.1-16997` (linux) `2022.2.1-16952` (osx) `2022.1.0-874` (win)
`netlib`	`3.9.0-5`	`3.9.0-5`
`pypy`	`7.3.11-0`	`7.3.11-1`		X
`pythran`	`0.12.1-0`	`0.13.1-0`	X
`qemu-user-static`	`7.2.0-1`	`7.2.0-1`

variant	1.10.1	now
linux + x64 + mkl + avx512	1 test failure; scipy/scipy#15533	remains (but should be fixed on `main`)
linux + x64 + openblas + non-avx512	1 test failure scipy/scipy#16927	gone 🥳
linux + aarch64 + pypy	passed	crash or hang results in abandoned job
osx + x64 + accelerate	7 test failures	old 7 are gone; 1 new failure (occasionally passing; once out of 8 runs)
osx + x64 + blis	1 test failure scipy/scipy#16927	old failure gone; one flaky crash (disappeared upon rerun)
osx + x64 + mkl + CPython	13-14 test failures	gone 🥳
osx + x64 + mkl + PyPy	2-3 test failures	2 remain
win + mkl + avx512	1 test failure; scipy/scipy#15533	remains (but should be fixed on `main`)
win + openblas	7 test failures scipy/scipy#17869	1 failure remains (CPython) 2 failures (PyPy)

variant	accelerate	blis	mkl w/ avx512	mkl w/o avx512	netlib	openblas w/ avx512	openblas w/o avx512	sum**
linux / x64	➖	✔️	❌	✔️	✔️	✔️	✔️	4F
linux / aarch64	➖	➖	➖	➖	✔️/ ❌(PyPy)	➖	✔️/ ❌(PyPy)	2F
linux / ppc64le	➖	➖	➖	➖	✔️	➖	✔️	-
osx / arm	❔	➖	➖	➖	❔	➖	❔	-
osx / x64	✔️(1) / ❌(3)	✔️*	➖	✔️/ ❌(PyPy)	✔️	➖	✔️	4F
win / x64	➖	✔️	❌	✔️	✔️	❌	❌	12F
sum**	3F	-	8F	1F	1F	4F	5F	22F

* previously had one (of 4) crash, disappeared upon rerun
** sum of Segfaults (S), Failures (F), resp. Timeouts (T); out of a total of 93 CI combinations being tested

Build logs:
Azure

(linux or win) + x64 + mkl + avx512: 1 failure

=================================== FAILURES ===================================
__________________ test_x0_equals_Mb[bicgstab-nonsymposdef-F] __________________
[gw0] linux -- Python 3.11.4 $PREFIX/bin/python
../_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_plac/lib/python3.11/site-packages/scipy/sparse/linalg/_isolve/tests/test_iterative.py:532: in test_x0_equals_Mb
    assert info == 0
E   assert -11 == 0
        A          = <10x10 sparse matrix of type '<class 'numpy.complex64'>'
	with 19 stored elements in Compressed Sparse Row format>
        b          = array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])
        case       = <nonsymposdef-F>
        info       = -11
        solver     = <function bicgstab at 0x7fc5a542d3a0>
        sup        = <numpy.testing._private.utils.suppress_warnings object at 0x7fc5816b53d0>
        tol        = 1e-08
        x          = array([0.        +0.j, 0.50000358+0.j, 1.25016941+0.j, 2.12653797+0.j,
       3.0674186 +0.j, 4.03716976+0.j, 5.0158146 +0.j, 6.00288608+0.j,
       7.        +0.j, 8.        +0.j])
        x0         = 'Mb'

osx + x64 + accelerate: 1 failure

=========================== short test summary info ============================
FAILED interpolate/tests/test_rbfinterp.py::TestRBFInterpolatorNeighbors20::test_pickleable - AssertionError: 
Arrays are not equal

Mismatched elements: 2 / 50 (4%)
Max absolute difference: 5.55111512e-17
Max relative difference: 2.32789676e-16
 x: array([-0.238461,  0.257024, -0.811244,  0.295313,  0.132623,  0.2038  ,
       -0.99615 ,  0.29755 , -0.019889,  0.230791, -0.516801,  0.279867,
        0.227767,  0.178071, -0.172756,  0.251327, -0.406818,  0.271919,...
 y: array([-0.238461,  0.257024, -0.811244,  0.295313,  0.132623,  0.2038  ,
       -0.99615 ,  0.29755 , -0.019889,  0.230791, -0.516801,  0.279867,
        0.227767,  0.178071, -0.172756,  0.251327, -0.406818,  0.271919,...
= 1 failed, 54558 passed, 2850 skipped, 245 xfailed, 10 xpassed, 10922 warnings in 1648.22s (0:27:28) =

osx + x64 + blis: 1 failure (disappeared upon rerun)

=================================== FAILURES ===================================
________________________ linalg/tests/test_decomp_lu.py ________________________
[gw1] darwin -- Python 3.10.12 $PREFIX/bin/python
worker 'gw1' crashed while running 'linalg/tests/test_decomp_lu.py::TestLU::test_simple_lu_shapes_real_complex[shape5]'

osx + x64 + mkl + pypy: 2 failures

=========================== short test summary info ============================
FAILED linalg/tests/test_decomp_cossin.py::test_cossin[True-40-12-20-float32] - AssertionError: 
Not equal to tolerance rtol=0, atol=1.19209e-06

Mismatched elements: 166 / 1600 (10.4%)
Max absolute difference: 1.2262093
Max relative difference: 2.0001094
 x: array([[ 0.088203, -0.328941, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [-0.233614, -0.048775, -0.223086, ...,  0.      ,  0.      ,...
 y: array([[ 0.088203, -0.328941, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048775,  0.223086, ...,  0.      ,  0.      ,...
FAILED linalg/tests/test_decomp_cossin.py::test_cossin[False-40-12-20-float32] - AssertionError: 
Not equal to tolerance rtol=0, atol=1.19209e-06

Mismatched elements: 251 / 1600 (15.7%)
Max absolute difference: 1.2262111
Max relative difference: 2.0005898
 x: array([[ 0.088202, -0.328941, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048775,  0.223086, ...,  0.      ,  0.      ,...
 y: array([[ 0.088203, -0.328941, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048776,  0.223085, ...,  0.      ,  0.      ,...
= 2 failed, 42003 passed, 2820 skipped, 137 xfailed, 9 xpassed, 4678 warnings in 2755.18s (0:45:55) =

windows + openblas + CPython: 1 failure

================================== FAILURES ===================================
___________________________ TestEig.test_bad_geneig ___________________________
[gw1] win32 -- Python 3.10.12 %PREFIX%\python.exe
..\_test_env\lib\site-packages\scipy\linalg\tests\test_decomp.py:310: in test_bad_geneig
    self._check_gen_eig(A, B)
        A          = [[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, -8.9975, 0], [0, 0, 0, -8.9975]]
        B          = [[0, 0, 1, 0], [0, 0, 0, 1], [1, 0, 0, -0.1], [0, 1, 0.1, 0]]
        k          = 1
        matrices   = <function TestEig.test_bad_geneig.<locals>.matrices at 0x000002BD8AE96950>
        self       = <scipy.linalg.tests.test_decomp.TestEig object at 0x000002BDF3109B40>
..\_test_env\lib\site-packages\scipy\linalg\tests\test_decomp.py:247: in _check_gen_eig
    assert_allclose(w[perm], wt[permt],
        A          = array([[ 1.    ,  0.    ,  0.    ,  0.    ],
       [ 0.    ,  1.    ,  0.    ,  0.    ],
       [ 0.    ,  0.    , -8.9975,  0.    ],
       [ 0.    ,  0.    ,  0.    , -8.9975]])
        B          = array([[ 0. ,  0. ,  1. ,  0. ],
       [ 0. ,  0. ,  0. ,  1. ],
       [ 1. ,  0. ,  0. , -0.1],
       [ 0. ,  1. ,  0.1,  0. ]])
        B0         = array([[ 0. ,  0. ,  1. ,  0. ],
       [ 0. ,  0. ,  0. ,  1. ],
       [ 1. ,  0. ,  0. , -0.1],
       [ 0. ,  1. ,  0.1,  0. ]])
        beta_nonzero = array([ True,  True,  True,  True])
        i          = 3
        length     = array([1., 1., 1., 1.])
        msg        = '\narray([[ 1.    ,  0.    ,  0.    ,  0.    ],\n       [ 0.    ,  1.    ,  0.    ,  0.    ],\n       [ 0.    ,  0.   ... ,  1. ,  0. ],\n       [ 0. ,  0. ,  0. ,  1. ],\n       [ 1. ,  0. ,  0. , -0.1],\n       [ 0. ,  1. ,  0.1,  0. ]])'
        perm       = array([1, 0, 3, 2], dtype=int64)
        permt      = array([1, 3, 2, 0], dtype=int64)
        res        = array([[ 0.00000000e+00-1.11022302e-16j,  0.00000000e+00+1.11022302e-16j,
        -7.77156117e-16-3.96947546e-17j, -7....9065e-15j,  0.00000000e+00-2.44249065e-15j,
         4.44089210e-16+6.52076712e-17j,  4.44089210e-16-6.52076712e-17j]])
        self       = <scipy.linalg.tests.test_decomp.TestEig object at 0x000002BDF3109B40>
        val1       = array([[ 0.00000000e+00-6.71913752e-01j,  0.00000000e+00+6.71913752e-01j,
        -6.69676706e-01+1.21957074e-16j, -6....4557e+00j,  0.00000000e+00-1.98214557e+00j,
        -2.04251395e+00+3.44054481e-16j, -2.04251395e+00-3.44054481e-16j]])
        val2       = array([[-0.00000000e+00-6.71913752e-01j,  0.00000000e+00+6.71913752e-01j,
        -6.69676706e-01+1.61651829e-16j, -6....4557e+00j,  0.00000000e+00-1.98214557e+00j,
        -2.04251395e+00+2.78846810e-16j, -2.04251395e+00-2.78846810e-16j]])
        vr         = array([[ 0.00000000e+00-6.71913752e-01j,  0.00000000e+00+6.71913752e-01j,
        -6.69676706e-01+1.21957074e-16j, -6....9591e-01j,  0.00000000e+00+2.20299591e-01j,
         2.27009053e-01-3.82388976e-17j,  2.27009053e-01+3.82388976e-17j]])
        w          = array([0.00000000e+00+3.05j, 0.00000000e+00-3.05j, 1.00194251e-16+2.95j,
       1.00194251e-16-2.95j])
        w_fin      = array([0.00000000e+00+3.05j, 0.00000000e+00-3.05j, 1.00194251e-16+2.95j,
       1.00194251e-16-2.95j])
        wh         = array([0.00000000e+00+3.05j, 0.00000000e+00-3.05j, 1.00194251e-16+2.95j,
       1.00194251e-16-2.95j])
        wt         = array([0.+3.05j, 0.-3.05j, 0.+2.95j, 0.-2.95j])
        wt_fin     = array([0.+3.05j, 0.-3.05j, 0.+2.95j, 0.-2.95j])
..\_test_env\lib\contextlib.py:79: in inner
    return func(*args, **kwds)
E   AssertionError:
E   Not equal to tolerance rtol=1e-07, atol=1e-07
E
E   array([[ 1.    ,  0.    ,  0.    ,  0.    ],
E          [ 0.    ,  1.    ,  0.    ,  0.    ],
E          [ 0.    ,  0.    , -8.9975,  0.    ],
E          [ 0.    ,  0.    ,  0.    , -8.9975]])
E   array([[ 0. ,  0. ,  1. ,  0. ],
E          [ 0. ,  0. ,  0. ,  1. ],
E          [ 1. ,  0. ,  0. , -0.1],
E          [ 0. ,  1. ,  0.1,  0. ]])
E   Mismatched elements: 3 / 4 (75%)
E   Max absolute difference: 6.
E   Max relative difference: 2.03389831
E    x: array([0.000000e+00-3.05j, 0.000000e+00+3.05j, 1.001943e-16-2.95j,
E          1.001943e-16+2.95j])
E    y: array([0.-3.05j, 0.-2.95j, 0.+2.95j, 0.+3.05j])
        args       = (<function assert_allclose.<locals>.compare at 0x000002BD8AE95FC0>, array([0.00000000e+00-3.05j, 0.00000000e+00+3.05j, 1.00194251e-16-2.95j,
       1.00194251e-16+2.95j]), array([0.-3.05j, 0.-2.95j, 0.+2.95j, 0.+3.05j]))
        func       = <function assert_array_compare at 0x000002BDDD30F010>
        kwds       = {'equal_nan': True, 'err_msg': '\narray([[ 1.    ,  0.    ,  0.    ,  0.    ],\n       [ 0.    ,  1.    ,  0.    ,  0....-0.1],\n       [ 0. ,  1. ,  0.1,  0. ]])', 'header': 'Not equal to tolerance rtol=1e-07, atol=1e-07', 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x000002BDDD324CA0>

windows + openblas + PyPy: 2 failure

Same TestEig.test_bad_geneig failure as CPython, plus:

================================== FAILURES ===================================
_______________________ test_public_modules_importable ________________________
[gw0] win32 -- Python 3.9.16 %PREFIX%\python.exe
..\_test_env\lib\site-packages\scipy\_lib\tests\test_import_cycles.py:14: in test_public_modules_importable
    assert pid.wait() == 0, f'Failed to import {PUBLIC_MODULES[i]}'
E   AssertionError: Failed to import scipy.cluster
E   assert 1 == 0
E    +  where 1 = <bound method Popen.wait of <Popen: returncode: 1 args: ['C:\\bld\\scipy-split_1687779369343\\_test_env\...>>()
E    +    where <bound method Popen.wait of <Popen: returncode: 1 args: ['C:\\bld\\scipy-split_1687779369343\\_test_env\...>> = <Popen: returncode: 1 args: ['C:\\bld\\scipy-split_1687779369343\\_test_env\...>.wait
        i          = 0
        pid        = <Popen: returncode: 1 args: ['C:\\bld\\scipy-split_1687779369343\\_test_env\...>
        pids       = [<Popen: returncode: 1 args: ['C:\\bld\\scipy-split_1687779369343\\_test_env\...>, <Popen: returncode: None args: ['C:...plit_1687779369343\\_test_e...>, <Popen: returncode: None args: ['C:\\bld\\scipy-split_1687779369343\\_test_e...>, ...]
---------------------------- Captured stdout call -----------------------------
Fatal error in cpyext, CPython compatibility layer, calling PyIter_Next
Either report a bug or consider not using this particular extension
<MemoryError object at 0x7ffecb1b28f0>
---------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "%PREFIX%\lib\site-packages\scipy\__init__.py", line 121, in <module>
    from . import _distributor_init
  File "%PREFIX%\lib\site-packages\scipy\_distributor_init.py", line 59, in <module>
    WinDLL(os.path.abspath(filename))
  File "%PREFIX%\Lib\ctypes\__init__.py", line 378, in __init__
    pypy_dll = _ffi.WinDLL(name, mode)
OSError: Cannot load library %PREFIX%\lib\site-packages\scipy\.libs\lib_zpropac.2QW5IRCDOYNJQXT22NTKLJQ4AKNKRHJT.gfortran-win_amd64.dll: The paging file is too small for this operation to complete
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "%PREFIX%\lib\site-packages\scipy\stats\__init__.py", line 608, in <module>
    from ._stats_py import *
MemoryError
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "%PREFIX%\lib\site-packages\scipy\stats\__init__.py", line 608, in <module>
    from ._stats_py import *
MemoryError
Traceback (most recent call last):
  File "%PREFIX%\lib\site-packages\scipy\__init__.py", line 143, in <module>
    from scipy._lib._ccallback import LowLevelCallable
  File "%PREFIX%\lib\site-packages\scipy\_lib\_ccallback.py", line 1, in <module>
    from . import _ccallback_c
ImportError: The paging file is too small for this operation to complete

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "%PREFIX%\lib\site-packages\scipy\__init__.py", line 148, in <module>
    raise ImportError(msg) from e
ImportError: The `scipy` install you are using seems to be broken, (extension modules cannot be imported), please try reinstalling.
RPython traceback:
  File "pypy_interpreter.c", line 30793, in length_hint
  File "pypy_objspace.c", line 20810, in len_w
  File "pypy_objspace.c", line 23488, in _len
out of memory: couldn't allocate the next arena
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "%PREFIX%\lib\site-packages\scipy\ndimage\__init__.py", line 154, in <module>
    from ._interpolation import *  # noqa: F401 F403
  File "%PREFIX%\lib\site-packages\scipy\ndimage\_interpolation.py", line 37, in <module>
    from scipy import special
  File "<frozen importlib._bootstrap>", line 1055, in _handle_fromlist
  File "%PREFIX%\lib\site-packages\scipy\__init__.py", line 189, in __getattr__
    return _importlib.import_module(f'scipy.{name}')
  File "%PREFIX%\Lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<builtin>/frozen importlib._bootstrap_external", line 865, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "%PREFIX%\lib\site-packages\scipy\special\__init__.py", line 772, in <module>
    from . import _ufuncs
ImportError: The paging file is too small for this operation to complete
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "%PREFIX%\lib\site-packages\scipy\__init__.py", line 121, in <module>
    from . import _distributor_init
  File "%PREFIX%\lib\site-packages\scipy\_distributor_init.py", line 59, in <module>
    WinDLL(os.path.abspath(filename))
  File "%PREFIX%\Lib\ctypes\__init__.py", line 378, in __init__
    pypy_dll = _ffi.WinDLL(name, mode)
OSError: Cannot load library %PREFIX%\lib\site-packages\scipy\.libs\libdfitpack.MCFILDAUIYO4QYQL6UCLEJGQTKPQSAWZ.gfortran-win_amd64.dll: The paging file is too small for this operation to complete
RPython traceback:
  File "pypy_module__io_1.c", line 17395, in W_FileIO_descr_init
  File "pypy_interpreter_1.c", line 9247, in ObjSpace_c_int_w
  File "pypy_interpreter.c", line 8413, in W_Root_int_w
  File "pypy_interpreter.c", line 25753, in W_Root_int
out of memory: couldn't allocate the next arena
RPython traceback:
  File "rpython_jit_metainterp.c", line 3019, in ll_portal_runner__pypy_interpreter_baseobjspace_
  File "rpython_jit_metainterp.c", line 4687, in maybe_compile_and_run__star_3
out of memory: couldn't allocate the next arena
RPython traceback:
  File "pypy_module_cpyext.c", line 8290, in wrapper_second_level__star_1_6
  File "pypy_module_cpyext_1.c", line 46051, in make_ref
  File "pypy_module_cpyext_2.c", line 30702, in create_ref
  File "pypy_module_cpyext_6.c", line 17753, in tuple_attach
  File "rpython_rtyper_lltypesystem.c", line 19512, in _ll_dict_setitem_lookup_done__v4342___simple_cal
  File "rpython_rtyper_lltypesystem.c", line 30116, in _ll_dict_resize_to__DICTPtr_Signed
  File "rpython_memory_gctransform.c", line 9230, in ll_malloc_varsize__Signed_Signed_Signed_Signed
RPython traceback:
  File "pypy_objspace_std.c", line 65098, in unmarshal_tuple_1
  File "pypy_objspace_std.c", line 10509, in makespecialisedtuple
out of memory: couldn't allocate a few KB more
RPython traceback:
  File "pypy_objspace_std.c", line 65098, in unmarshal_tuple_1
  File "pypy_objspace_std.c", line 10509, in makespecialisedtuple
out of memory: couldn't allocate the next arena
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "%PREFIX%\lib\site-packages\scipy\linalg\__init__.py", line 206, in <module>
    from ._misc import *
  File "%PREFIX%\lib\site-packages\scipy\linalg\_misc.py", line 4, in <module>
    from .lapack import get_lapack_funcs
  File "%PREFIX%\lib\site-packages\scipy\linalg\lapack.py", line 824, in <module>
    from scipy.linalg import _flapack
ImportError: The paging file is too small for this operation to complete
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "%PREFIX%\lib\site-packages\scipy\__init__.py", line 121, in <module>
    from . import _distributor_init
  File "%PREFIX%\lib\site-packages\scipy\_distributor_init.py", line 59, in <module>
    WinDLL(os.path.abspath(filename))
  File "%PREFIX%\Lib\ctypes\__init__.py", line 378, in __init__
    pypy_dll = _ffi.WinDLL(name, mode)
OSError: Cannot load library %PREFIX%\lib\site-packages\scipy\.libs\lib_zpropac.2QW5IRCDOYNJQXT22NTKLJQ4AKNKRHJT.gfortran-win_amd64.dll: The paging file is too small for this operation to complete
RPython traceback:
  File "rpython_jit_metainterp_4.c", line 24056, in MIFrame_run_one_step
  File "rpython_jit_metainterp_5.c", line 18538, in handler_inline_call_ir_r_1
out of memory: couldn't allocate the next arena
MemoryError
RPython traceback:
  File "pypy_interpreter.c", line 30793, in length_hint
  File "pypy_objspace.c", line 20810, in len_w
  File "pypy_objspace.c", line 23488, in _len
out of memory: couldn't allocate the next arena
Traceback (most recent call last):
  File "%PREFIX%\lib\site-packages\scipy\__init__.py", line 143, in <module>
    from scipy._lib._ccallback import LowLevelCallable
  File "%PREFIX%\lib\site-packages\scipy\_lib\_ccallback.py", line 1, in <module>
    from . import _ccallback_c
ImportError: The paging file is too small for this operation to complete

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "%PREFIX%\lib\site-packages\scipy\__init__.py", line 148, in <module>
    raise ImportError(msg) from e
ImportError: The `scipy` install you are using seems to be broken, (extension modules cannot be imported), please try reinstalling.
RPython traceback:
  File "pypy_module_pypyjit.c", line 940, in portal_4
  File "pypy_interpreter_2.c", line 35363, in handle_bytecode__AccessDirect_None
  File "pypy_interpreter_2.c", line 45590, in dispatch_bytecode__AccessDirect_None
out of memory: couldn't allocate the next arena

h-vetinari · 2023-06-27T21:43:18Z

After a CI rerun, two failures disappeared, so we're at 22/93 now. :)

…nda-forge-pinning 2024.01.08.19.20.26

h-vetinari · 2024-01-10T01:37:12Z

Scipy 1.11.4

TL;DR: the bad old days are beckoning

From 22 failures out of 93 runs for 1.11.0, we're now at 45 failures out of 112 CI runs

Notable

First test after complete switch to meson (also on windows)
Added CPython 3.12
Deactivated pypy on aarch/ppc

Good News

BUG: test failure in test_x0_equals_Mb with bicgstab scipy/scipy#15533 disappeared on win+mkl (but appeared elsewhere...)

Bad News

Lots of (apparently emulation-related) failures on aarch
New failures with ppc
BUG: test failure in test_x0_equals_Mb with bicgstab scipy/scipy#15533 now also affects linux+x64+openblas+avx512 as well as win+blis (the latter regardless of avx512)
There's not a single BLAS flavour that's failure-free 😑

Restrictions

test on aarch64/ppc64le resp. for PyPy only run tests with label='fast', not the full test suite
the following test skips:

# on the slowest agents, these tests take more than 20min in emulation
{% set tests_to_skip = tests_to_skip + " or (test_lsq_linear and test_large_rank_deficient)" %}   # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_iterative and test_precond_inverse[case1])" %}  # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (test_optimize and TestBrute and test_workers)" %}    # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (TestLevyStable and test_location_scale)" %}          # [aarch64 or ppc64le]
{% set tests_to_skip = tests_to_skip + " or (TestShgoArguments and test_19_parallelization)" %}   # [aarch64 or ppc64le]
# tests that run into timeouts (set in scipy test suite) in emulation
{% set tests_to_skip = tests_to_skip + " or (test_propack and test_examples)" %}                  # [aarch64 or ppc64le]
# occasionally runs out of memory on windows + pypy
{% set tests_to_skip = tests_to_skip + " or test_bootstrap_against_" %}                           # [(python_impl == "pypy") and win]
{% set tests_to_skip = tests_to_skip + " or test_more_converged" %}                               # [(python_impl == "pypy") and win]
# very slow tests for pypy in emulation
[several]

Details

lib	for 1.11.0	now	updated version	updated build
`scipy`	`1.11.0`	`1.11.4`	X
`numpy`	`1.25.0`	`1.26.3`	X
`libblas`	`3.9.0-17`	`3.9.0-20`		X
`blis`	`0.9.0-1`	`0.9.0-1`
`openblas`	`0.3.23-pthreads-0`	`0.3.25-pthreads-0`	X
`mkl`	`2022.2.1-16997` (linux) `2022.2.1-16952` (osx) `2022.1.0-874` (win)	`2023.2.0-50496` (linux) `2023.2.0-50500` (osx) `2023.2.0-50497` (win)	X
`netlib`	`3.9.0-5`	`3.9.0-5`
`pypy`	`7.3.11.1`	`7.3.13-1`	X
`pythran`	`0.13.1-0`	`0.15.0-0`	X
`qemu-user-static`	`7.2.0-1`	`8.1.3-1`	X

variant	1.11.0	now
linux + x64 + mkl + avx512	1 test failure; scipy/scipy#15533	now also with openblas, as well as win+blis
linux + x64 + openblas + avx512		scipy/scipy#15533
linux + aarch64	-	66 failures, mostly QR-related; apparently due to emulation
linux + aarch64 + pypy	crash or hang resulting in abandoned job	not run anymore
linux + ppc64le	-	6 failures in `test_distribution_outside_support`
osx + x64 + accelerate	1 flaky test failure (minor tolerance violation)	remains
osx + x64 + mkl + PyPy	2 test failures	1 remains (`test_cossin`)
win + blis	-	scipy/scipy#15533 appeares both with & w/o avx512
win + mkl + avx512	1 test failure; scipy/scipy#15533	gone 🥳
win + openblas	1 failure (CPython) 2 failures (PyPy)	scipy/scipy#17125 remains

variant	accelerate	blis	mkl w/ avx512	mkl w/o avx512	netlib	openblas w/ avx512	openblas w/o avx512	sum**
linux / x64	➖	✔️	❌	✔️	✔️	❌	✔️	10F
linux / aarch64	➖	➖	➖	➖	❌	➖	❌	8F
linux / ppc64le	➖	➖	➖	➖	❌	➖	❌	8F
osx / arm	❔	➖	➖	➖	❔	➖	❔	-
osx / x64	✔️/❌	✔️	➖	✔️/ ❌(PyPy)	✔️	➖	✔️	4F
win / x64	➖	❌	✔️	✔️	✔️	❌	❌	15F
sum**	3F	5F	5F	1F	8F	10F	13F	45F

** sum of Segfaults (S), Failures (F), resp. Timeouts (T); out of a total of 112 CI combinations being tested

Build logs:
Azure

The curious case of `test_x0_equals_Mb[bicgstab-nonsymposdef-F]`

Looking a bit closers around the failures related to scipy/scipy#15533, it's worth noting that while the scipy sources are effectively unchanged (1.11.0 vs. 1.11.4), we have quite surprising behaviour changes.

	before	after	comment
linux + x64 + blis + avx2	✔️	✔️
linux + x64 + blis + avx512	✔️	✔️
linux + x64 + mkl + avx2	✔️	✔️
linux + x64 + mkl + avx512	❌	❌	unchanged
linux + x64 + openblas + avx2	✔️	✔️
linux + x64 + openblas + avx512	✔️	❌	started failing
win + x64 + blis + avx2	✔️	❌	started failing
win + x64 + blis + avx512	✔️	❌	started failing
win + x64 + mkl + avx2	✔️	✔️
win + x64 + mkl + avx512	❌	✔️	stopped failing
win + x64 + openblas + avx2	✔️	✔️
win + x64 + openblas + avx512	✔️	✔️

Aside from linux+openblas suddenly becoming susceptible and win+mkl suddenly passing (the latter perhaps related to the switch to meson? 🤔), the fact that blis could start failing (while being unchanged in terms of build) must almost certainly be related to LAPACK (since blis only provides BLAS, and we add netlib's LAPACK to that), as we also went from 3.9.0-17 to 3.9.0-20 for our blas metapackage.

linux + x64 + {mkl,openblas} + avx512 / win + blis: 1 failure

__________________ test_x0_equals_Mb[bicgstab-nonsymposdef-F] __________________
[gw0] linux -- Python 3.10.13 $PREFIX/bin/python
$PREFIX/lib/python3.10/site-packages/scipy/sparse/linalg/_isolve/tests/test_iterative.py:532: in test_x0_equals_Mb
    assert info == 0
E   assert -11 == 0
        A          = <10x10 sparse matrix of type '<class 'numpy.complex64'>'
	with 19 stored elements in Compressed Sparse Row format>
        b          = array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])
        case       = <nonsymposdef-F>
        info       = -11
        solver     = <function bicgstab at 0x7f65f03e5510>
        sup        = <numpy.testing._private.utils.suppress_warnings object at 0x7f65d03a3700>
        tol        = 1e-08
        x          = array([0.        +0.j, 0.50000358+0.j, 1.25016941+0.j, 2.12653797+0.j,
       3.0674186 +0.j, 4.03716976+0.j, 5.0158146 +0.j, 6.00288608+0.j,
       7.        +0.j, 8.        +0.j])
        x0         = 'Mb'

linux + aarch64: 66 failures

Not showing the 6 failures in test_distribution_outside_support (same as for ppc)

FAILED linalg/tests/test_basic.py::TestLstsq::test_random_overdet - AssertionError: 
Not equal to tolerance rtol=2.98023e-06, atol=2.98023e-06
driver: gelsd
Mismatched elements: 45 / 45 (100%)
Max absolute difference: 23.948622
Max relative difference: 1.0819286
 x: array([[-2.175595e-02,  1.430921e-01,  1.740819e-02],
       [ 2.515154e-02,  2.834489e-02, -6.470084e-03],
       [-5.744661e-05,  1.434425e-02,  4.162818e-02],...
 y: array([[  4.199296,   5.158111,   2.723422],
       [ 11.47436 ,  11.579989,   5.939486],
       [  0.678561,   0.932052,   0.4389  ],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_tall_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.35266802
Max relative difference: 1.35266802
 x: array([[ 0.51534 , -0.066137, -0.282195, -0.078835,  0.126846,  0.063667,
         0.254653,  0.145272,  0.297032, -0.312945,  0.047386, -0.375106],
       [-0.066137,  0.058695,  0.107405, -0.020373, -0.37564 , -0.306869,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_tall_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.38513473
Max relative difference: 1.38513473
 x: array([[ 0.194025,  0.271456,  0.193257, -0.103866,  0.639289,  0.325098,
        -0.308913,  0.225063,  0.009908,  0.531324,  0.418844,  0.486349],
       [ 0.271456, -0.024757,  0.531958, -0.127783,  0.087164,  0.124488,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_delete_last_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.39604267
Max relative difference: 1.39604267
 x: array([[ 7.312894e-01,  2.936634e-02, -2.486336e-01, -3.261323e-01,
         4.044664e-03, -1.884397e-02,  4.260744e-01,  4.652548e-01,
         4.959358e-02,  2.202561e-04, -5.590688e-01, -1.013507e-04],...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_non_unit_strides_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.35266802
Max relative difference: 1.35266802
 x: array([[ 0.51534 , -0.066137, -0.282195, -0.078835,  0.126846,  0.063667,
         0.254653,  0.145272,  0.297032, -0.312945,  0.047386, -0.375106],
       [-0.066137,  0.058695,  0.107405, -0.020373, -0.37564 , -0.306869,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_non_unit_strides_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.37619865
Max relative difference: 1.37619865
 x: array([[ 1.469219e-01,  5.787525e-01,  5.345633e-02, -4.102918e-01,
         5.512089e-01, -6.510306e-02, -3.209547e-01, -2.358620e-01,
         2.342797e-01,  2.828279e-01,  3.863396e-01,  1.333238e-01],...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_neg_strides_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.35266802
Max relative difference: 1.35266802
 x: array([[ 0.51534 , -0.066137, -0.282195, -0.078835,  0.126846,  0.063667,
         0.254653,  0.145272,  0.297032, -0.312945,  0.047386, -0.375106],
       [-0.066137,  0.058695,  0.107405, -0.020373, -0.37564 , -0.306869,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_neg_strides_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.37619865
Max relative difference: 1.37619865
 x: array([[ 1.469219e-01,  5.787525e-01,  5.345633e-02, -4.102918e-01,
         5.512089e-01, -6.510306e-02, -3.209547e-01, -2.358620e-01,
         2.342797e-01,  2.828279e-01,  3.863396e-01,  1.333238e-01],...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_non_itemize_strides_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.35266802
Max relative difference: 1.35266802
 x: array([[ 0.51534 , -0.066137, -0.282195, -0.078835,  0.126846,  0.063667,
         0.254653,  0.145272,  0.297032, -0.312945,  0.047386, -0.375106],
       [-0.066137,  0.058695,  0.107405, -0.020373, -0.37564 , -0.306869,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_non_itemize_strides_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.37619865
Max relative difference: 1.37619865
 x: array([[ 1.469219e-01,  5.787525e-01,  5.345633e-02, -4.102918e-01,
         5.512089e-01, -6.510306e-02, -3.209547e-01, -2.358620e-01,
         2.342797e-01,  2.828279e-01,  3.863396e-01,  1.333238e-01],...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_non_native_byte_order_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.35266802
Max relative difference: 1.35266802
 x: array([[ 0.51534 , -0.066137, -0.282195, -0.078835,  0.126846,  0.063667,
         0.254653,  0.145272,  0.297032, -0.312945,  0.047386, -0.375106],
       [-0.066137,  0.058695,  0.107405, -0.020373, -0.37564 , -0.306869,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRdelete_f::test_non_native_byte_order_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.37619865
Max relative difference: 1.37619865
 x: array([[ 1.469219e-01,  5.787525e-01,  5.345633e-02, -4.102918e-01,
         5.512089e-01, -6.510306e-02, -3.209547e-01, -2.358620e-01,
         2.342797e-01,  2.828279e-01,  3.863396e-01,  1.333238e-01],...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_sqr_1_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 81 / 81 (100%)
Max absolute difference: 1.47812065
Max relative difference: 1.47812065
 x: array([[ 0.427075,  0.052891,  0.034453,  0.178182,  0.20928 , -0.183314,
        -0.029952,  0.108337,  0.05796 ],
       [ 0.052891,  0.173273,  0.570958, -0.143117,  0.093308,  0.188859,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_sqr_p_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 121 / 121 (100%)
Max absolute difference: 1.49504468
Max relative difference: 1.49504468
 x: array([[ 0.025394,  0.222058,  0.609238,  0.497672,  0.354511,  0.041871,
         0.731882, -0.370673, -0.38842 , -0.11726 , -0.131466],
       [ 0.222058,  0.635692, -0.111061,  0.158848,  0.168521, -0.129045,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_tall_1_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 169 / 169 (100%)
Max absolute difference: 1.47401035
Max relative difference: 1.47401035
 x: array([[ 0.522431, -0.118259, -0.003304,  0.450749, -0.030897,  0.1499  ,
         0.07167 , -0.145986, -0.025632, -0.278439, -0.021617, -0.100983,
        -0.252881],...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_tall_p_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 225 / 225 (100%)
Max absolute difference: 1.21126013
Max relative difference: 1.21126013
 x: array([[ 0.008006,  0.003634,  0.082993, -0.105183,  0.469302, -0.159896,
        -0.186092,  0.237852,  0.166177, -0.004724,  0.300496, -0.172572,
         0.054354,  0.01367 ,  0.251432],...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_tall_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.45033643
Max relative difference: 1.45033643
 x: array([[ 0.29902 ,  0.501335,  0.272783,  0.074555,  0.132249, -0.02093 ,
         0.245842, -0.023263,  0.475552,  0.064034, -0.186375,  0.20676 ],
       [ 0.501335,  0.219159, -0.362003, -0.375209, -0.100792, -0.299124,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_tall_p_col_tall - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.52949911
Max relative difference: 1.52949911
 x: array([[ 0.396285, -0.118527,  0.404352,  0.024046,  0.185335,  0.491734,
        -0.084112,  0.179654, -0.282193,  0.263326, -0.229785, -0.033931],
       [-0.118527, -0.145312,  0.166583,  0.136687,  0.248487,  0.227176,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_tall_p_col_sqr - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.29536363
Max relative difference: 1.29536363
 x: array([[ 1.839437e-01,  3.666864e-03,  5.269710e-01, -9.558372e-02,
         4.485931e-01, -1.068802e-01, -3.599494e-01,  2.523912e-01,
         5.202088e-01,  7.875285e-02, -2.388350e-01,  1.856074e-01],...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_tall_p_col_fat - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.22226353
Max relative difference: 1.22226353
 x: array([[ 0.30203 ,  0.289442,  0.491229,  0.232481,  0.069349, -0.017829,
         0.0731  , -0.101897, -0.061624,  0.032908, -0.294819, -0.209391],
       [ 0.289442, -0.027515, -0.225598, -0.00597 ,  0.261011,  0.194242,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_fat_p_row_fat - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 100 / 100 (100%)
Max absolute difference: 1.52807027
Max relative difference: 1.52807027
 x: array([[ 0.476516,  0.678434,  0.487398,  0.170842, -0.530036, -0.049983,
        -0.051561, -0.034272,  0.340691, -0.532899],
       [ 0.678434, -0.52807 , -0.086754,  0.512772,  0.126919,  0.114655,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_fat_p_row_sqr - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.33379635
Max relative difference: 1.33379635
 x: array([[ 0.407052,  0.457873, -0.2282  ,  0.20684 ,  0.335016, -0.217718,
         0.024833, -0.165538,  0.372844,  0.298751, -0.199244, -0.277085],
       [ 0.457873, -0.333796,  0.153424,  0.595791, -0.424009,  0.076896,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_fat_p_row_tall - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 196 / 196 (100%)
Max absolute difference: 1.45184031
Max relative difference: 1.45184031
 x: array([[ 1.306703e-01,  2.308598e-01,  1.503783e-02,  4.989481e-01,
         2.612073e-01, -6.566251e-02, -1.365806e-02, -4.439392e-01,
        -9.274840e-02, -3.816905e-02,  2.477478e-01,  1.054653e-01,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_economic_p_col_eco - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 100 / 100 (100%)
Max absolute difference: 1.37845999
Max relative difference: 1.37845999
 x: array([[ 0.488817,  0.293726,  0.560871, -0.452365,  0.316418,  0.060158,
         0.176355,  0.412072, -0.13077 ,  0.404673],
       [ 0.293726,  0.266032, -0.246029,  0.125679, -0.167877,  0.233249,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_economic_p_col_sqr - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.60243899
Max relative difference: 1.60243899
 x: array([[-3.211651e-01,  3.666854e-03,  5.269712e-01, -1.675753e-01,
         5.411102e-01, -1.068804e-01, -3.599494e-01,  1.086038e-02,
         4.765480e-01,  7.875297e-02, -2.388351e-01,  4.631867e-01],...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_economic_p_col_fat - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.22226353
Max relative difference: 1.22226353
 x: array([[ 0.038019,  0.289442,  0.491229,  0.232482,  0.065881, -0.01783 ,
         0.073099, -0.101897,  0.340713,  0.032909, -0.294819, -0.209391],
       [ 0.289442, -0.027515, -0.225598, -0.00597 ,  0.170632,  0.194242,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_Mx1_1_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 81 / 81 (100%)
Max absolute difference: 1.35482377
Max relative difference: 1.35482377
 x: array([[ 0.204203,  0.081063,  0.547405, -0.497832,  0.44612 , -0.364956,
         0.282041,  0.048404, -0.144208],
       [ 0.081063,  0.102958, -0.048356,  0.717492,  0.188318, -0.23341 ,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_Mx1_p_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 121 / 121 (100%)
Max absolute difference: 1.33370879
Max relative difference: 1.33370879
 x: array([[ 0.421005,  0.417547, -0.32667 , -0.29249 ,  0.282903,  0.253645,
        -0.110355, -0.220882, -0.263948, -0.180241,  0.751357],
       [ 0.417547, -0.047571, -0.367514, -0.214091, -0.303108, -0.364512,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_unit_strides_1_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 81 / 81 (100%)
Max absolute difference: 1.47812065
Max relative difference: 1.47812065
 x: array([[ 0.427075,  0.052891,  0.034453,  0.178182,  0.20928 , -0.183314,
        -0.029952,  0.108337,  0.05796 ],
       [ 0.052891,  0.173273,  0.570958, -0.143117,  0.093308,  0.188859,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_unit_strides_p_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 121 / 121 (100%)
Max absolute difference: 1.49504468
Max relative difference: 1.49504468
 x: array([[ 0.025394,  0.222058,  0.609238,  0.497672,  0.354511,  0.041871,
         0.731882, -0.370673, -0.38842 , -0.11726 , -0.131466],
       [ 0.222058,  0.635692, -0.111061,  0.158848,  0.168521, -0.129045,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_unit_strides_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.45033643
Max relative difference: 1.45033643
 x: array([[ 0.29902 ,  0.501335,  0.272783,  0.074555,  0.132249, -0.02093 ,
         0.245842, -0.023263,  0.475552,  0.064034, -0.186375,  0.20676 ],
       [ 0.501335,  0.219159, -0.362003, -0.375209, -0.100792, -0.299124,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_unit_strides_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.52949911
Max relative difference: 1.52949911
 x: array([[ 0.396285, -0.118527,  0.404352,  0.024046,  0.185335,  0.491734,
        -0.084112,  0.179654, -0.282193,  0.263326, -0.229785, -0.033931],
       [-0.118527, -0.145312,  0.166583,  0.136687,  0.248487,  0.227176,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_neg_strides_1_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 81 / 81 (100%)
Max absolute difference: 1.47812065
Max relative difference: 1.47812065
 x: array([[ 0.427075,  0.052891,  0.034453,  0.178182,  0.20928 , -0.183314,
        -0.029952,  0.108337,  0.05796 ],
       [ 0.052891,  0.173273,  0.570958, -0.143117,  0.093308,  0.188859,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_neg_strides_p_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 121 / 121 (100%)
Max absolute difference: 1.49504468
Max relative difference: 1.49504468
 x: array([[ 0.025394,  0.222058,  0.609238,  0.497672,  0.354511,  0.041871,
         0.731882, -0.370673, -0.38842 , -0.11726 , -0.131466],
       [ 0.222058,  0.635692, -0.111061,  0.158848,  0.168521, -0.129045,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_neg_strides_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.45033643
Max relative difference: 1.45033643
 x: array([[ 0.29902 ,  0.501335,  0.272783,  0.074555,  0.132249, -0.02093 ,
         0.245842, -0.023263,  0.475552,  0.064034, -0.186375,  0.20676 ],
       [ 0.501335,  0.219159, -0.362003, -0.375209, -0.100792, -0.299124,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_neg_strides_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.52949911
Max relative difference: 1.52949911
 x: array([[ 0.396285, -0.118527,  0.404352,  0.024046,  0.185335,  0.491734,
        -0.084112,  0.179654, -0.282193,  0.263326, -0.229785, -0.033931],
       [-0.118527, -0.145312,  0.166583,  0.136687,  0.248487,  0.227176,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_itemsize_strides_1_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 81 / 81 (100%)
Max absolute difference: 1.47812065
Max relative difference: 1.47812065
 x: array([[ 0.427075,  0.052891,  0.034453,  0.178182,  0.20928 , -0.183314,
        -0.029952,  0.108337,  0.05796 ],
       [ 0.052891,  0.173273,  0.570958, -0.143117,  0.093308,  0.188859,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_itemsize_strides_p_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 121 / 121 (100%)
Max absolute difference: 1.49504468
Max relative difference: 1.49504468
 x: array([[ 0.025394,  0.222058,  0.609238,  0.497672,  0.354511,  0.041871,
         0.731882, -0.370673, -0.38842 , -0.11726 , -0.131466],
       [ 0.222058,  0.635692, -0.111061,  0.158848,  0.168521, -0.129045,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_itemsize_strides_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.45033643
Max relative difference: 1.45033643
 x: array([[ 0.29902 ,  0.501335,  0.272783,  0.074555,  0.132249, -0.02093 ,
         0.245842, -0.023263,  0.475552,  0.064034, -0.186375,  0.20676 ],
       [ 0.501335,  0.219159, -0.362003, -0.375209, -0.100792, -0.299124,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_itemsize_strides_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.52949911
Max relative difference: 1.52949911
 x: array([[ 0.396285, -0.118527,  0.404352,  0.024046,  0.185335,  0.491734,
        -0.084112,  0.179654, -0.282193,  0.263326, -0.229785, -0.033931],
       [-0.118527, -0.145312,  0.166583,  0.136687,  0.248487,  0.227176,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_native_byte_order_1_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 81 / 81 (100%)
Max absolute difference: 1.47812065
Max relative difference: 1.47812065
 x: array([[ 0.427075,  0.052891,  0.034453,  0.178182,  0.20928 , -0.183314,
        -0.029952,  0.108337,  0.05796 ],
       [ 0.052891,  0.173273,  0.570958, -0.143117,  0.093308,  0.188859,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_native_byte_order_p_row - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 121 / 121 (100%)
Max absolute difference: 1.49504468
Max relative difference: 1.49504468
 x: array([[ 0.025394,  0.222058,  0.609238,  0.497672,  0.354511,  0.041871,
         0.731882, -0.370673, -0.38842 , -0.11726 , -0.131466],
       [ 0.222058,  0.635692, -0.111061,  0.158848,  0.168521, -0.129045,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_native_byte_order_1_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.45033643
Max relative difference: 1.45033643
 x: array([[ 0.29902 ,  0.501335,  0.272783,  0.074555,  0.132249, -0.02093 ,
         0.245842, -0.023263,  0.475552,  0.064034, -0.186375,  0.20676 ],
       [ 0.501335,  0.219159, -0.362003, -0.375209, -0.100792, -0.299124,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRinsert_f::test_non_native_byte_order_p_col - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.52949911
Max relative difference: 1.52949911
 x: array([[ 0.396285, -0.118527,  0.404352,  0.024046,  0.185335,  0.491734,
        -0.084112,  0.179654, -0.282193,  0.263326, -0.229785, -0.033931],
       [-0.118527, -0.145312,  0.166583,  0.136687,  0.248487,  0.227176,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_tall_rank_1 - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.56318134
Max relative difference: 1.56318134
 x: array([[-0.054046,  0.205639, -0.123517,  0.260355,  0.474265, -0.249005,
         0.177618, -0.301207,  0.618385, -0.037394,  0.023149,  0.252298],
       [ 0.205639,  0.756895, -0.286572,  0.154587,  0.321836,  0.283857,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_tall_rank_p - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.56318134
Max relative difference: 1.56318134
 x: array([[-0.054046,  0.205639, -0.123517,  0.260355,  0.474265, -0.249005,
         0.177618, -0.301207,  0.618385, -0.037394,  0.023149,  0.252298],
       [ 0.205639,  0.756895, -0.286572,  0.154587,  0.321836,  0.283857,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_non_unit_strides_rank_1 - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.56318134
Max relative difference: 1.56318134
 x: array([[-0.054046,  0.205639, -0.123517,  0.260355,  0.474265, -0.249005,
         0.177618, -0.301207,  0.618385, -0.037394,  0.023149,  0.252298],
       [ 0.205639,  0.756895, -0.286572,  0.154587,  0.321836,  0.283857,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_non_unit_strides_rank_p - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.5197227
Max relative difference: 1.5197227
 x: array([[ 0.030667,  0.051494, -0.262698, -0.204111,  0.552454,  0.260673,
        -0.072537,  0.140756,  0.215893, -0.045971,  0.289605, -0.198233],
       [ 0.051494,  0.300886,  0.159218,  0.002972, -0.091647,  0.648182,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_neg_strides_rank_1 - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.56318134
Max relative difference: 1.56318134
 x: array([[-0.054046,  0.205639, -0.123517,  0.260355,  0.474265, -0.249005,
         0.177618, -0.301207,  0.618385, -0.037394,  0.023149,  0.252298],
       [ 0.205639,  0.756895, -0.286572,  0.154587,  0.321836,  0.283857,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_neg_strides_rank_p - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.5197227
Max relative difference: 1.5197227
 x: array([[ 0.030667,  0.051494, -0.262698, -0.204111,  0.552454,  0.260673,
        -0.072537,  0.140756,  0.215893, -0.045971,  0.289605, -0.198233],
       [ 0.051494,  0.300886,  0.159218,  0.002972, -0.091647,  0.648182,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_non_itemsize_strides_rank_1 - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.56318134
Max relative difference: 1.56318134
 x: array([[-0.054046,  0.205639, -0.123517,  0.260355,  0.474265, -0.249005,
         0.177618, -0.301207,  0.618385, -0.037394,  0.023149,  0.252298],
       [ 0.205639,  0.756895, -0.286572,  0.154587,  0.321836,  0.283857,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_non_itemsize_strides_rank_p - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.5197227
Max relative difference: 1.5197227
 x: array([[ 0.030667,  0.051494, -0.262698, -0.204111,  0.552454,  0.260673,
        -0.072537,  0.140756,  0.215893, -0.045971,  0.289605, -0.198233],
       [ 0.051494,  0.300886,  0.159218,  0.002972, -0.091647,  0.648182,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_non_native_byte_order_rank_1 - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.56318134
Max relative difference: 1.56318134
 x: array([[-0.054046,  0.205639, -0.123517,  0.260355,  0.474265, -0.249005,
         0.177618, -0.301207,  0.618385, -0.037394,  0.023149,  0.252298],
       [ 0.205639,  0.756895, -0.286572,  0.154587,  0.321836,  0.283857,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_decomp_update.py::TestQRupdate_f::test_non_native_byte_order_rank_p - AssertionError: 
Not equal to tolerance rtol=0.0001, atol=1.19209e-06

Mismatched elements: 144 / 144 (100%)
Max absolute difference: 1.5197227
Max relative difference: 1.5197227
 x: array([[ 0.030667,  0.051494, -0.262698, -0.204111,  0.552454,  0.260673,
        -0.072537,  0.140756,  0.215893, -0.045971,  0.289605, -0.198233],
       [ 0.051494,  0.300886,  0.159218,  0.002972, -0.091647,  0.648182,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],...
FAILED linalg/tests/test_lapack.py::test_pftri - AssertionError: 
Arrays are not almost equal to 4 decimals

Mismatched elements: 21 / 400 (5.25%)
Max absolute difference: 0.00037931
Max relative difference: 0.64988735
 x: array([[ 5.0451e-02, -1.1576e-03,  2.7290e-04, -1.5024e-03, -1.7656e-03,
        -1.3410e-03, -1.4870e-03, -7.1398e-04, -1.4734e-03, -2.5769e-03,
        -2.0850e-03, -9.8616e-04, -7.3829e-04, -1.9978e-03,  1.7069e-04,...
 y: array([[ 5.0565e-02, -1.0042e-03,  2.1366e-04, -1.5335e-03, -1.5981e-03,
        -1.4567e-03, -1.5799e-03, -5.9890e-04, -1.4266e-03, -2.5949e-03,
        -2.0850e-03, -9.8616e-04, -7.3829e-04, -1.9978e-03,  1.7069e-04,...
FAILED linalg/tests/test_lapack.py::test_sfrk_hfrk - AssertionError: 
Arrays are not almost equal to 4 decimals

Mismatched elements: 165 / 400 (41.2%)
Max absolute difference: 1.14951146
Max relative difference: 14.46566384
 x: array([[ 3.9575e+01,  1.3012e+00,  5.5721e-02,  1.6919e+00,  1.3906e+00,
         1.7789e+00,  8.5231e-01,  8.1178e-01,  1.2308e+00,  2.1547e+00,
         1.7334e+00,  6.0473e-01,  2.1888e-01,  2.0501e+00, -1.1836e-01,...
 y: array([[ 3.9575e+01,  1.3012e+00,  2.2163e-01,  1.2076e+00,  1.4051e+00,
         1.8348e+00,  1.0907e+00,  1.0859e+00,  1.3009e+00,  2.1895e+00,
         1.3301e+00,  1.5722e+00,  1.7302e-01,  1.5784e+00,  1.6231e-01,...
FAILED linalg/tests/test_lapack.py::TestBlockedQR::test_geqrt_gemqrt - AssertionError: 
Not equal to tolerance rtol=0, atol=1.19209e-05

Mismatched elements: 320 / 400 (80%)
Max absolute difference: 1.4186668
Max relative difference: 1.4186668
 x: array([[ 4.153938e-01,  3.214427e-01, -1.697152e-01, -2.782870e-01,
         2.549522e-01,  3.815206e-01, -1.836173e-01, -8.294257e-03,
         2.499608e-01,  1.465432e-02,  1.211796e-01, -2.663902e-01,...
 y: array([[1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0.],
       [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,...
FAILED linalg/tests/test_lapack.py::TestBlockedQR::test_tpqrt_tpmqrt - AssertionError: 
Not equal to tolerance rtol=0, atol=1.19209e-05

Mismatched elements: 1024 / 1600 (64%)
Max absolute difference: 1.2731317
Max relative difference: 1.2731317
 x: array([[ 3.782889e-03,  1.856932e-02,  1.523624e-02, ..., -1.549972e-01,
         2.169387e-01, -3.143149e-01],
       [ 1.856932e-02,  4.894961e-02, -5.576465e-02, ...,  2.150272e-02,...
 y: array([[1., 0., 0., ..., 0., 0., 0.],
       [0., 1., 0., ..., 0., 0., 0.],
       [0., 0., 1., ..., 0., 0., 0.],...
FAILED linalg/tests/test_lapack.py::test_pstrf - AssertionError: 
Not equal to tolerance rtol=0, atol=0.000119209

Mismatched elements: 85 / 100 (85%)
Max absolute difference: 3.529682
Max relative difference: 1.4060138
 x: array([[3.487075, 2.2597  , 1.555289, 2.763015, 3.431685, 2.046993,
        2.797032, 2.700427, 3.422926, 2.103169],
       [2.2597  , 2.130606, 1.477844, 1.633565, 2.117459, 1.866201,...
 y: array([[3.487075, 2.2597  , 1.555289, 2.763015, 3.431685, 2.046993,
        2.797032, 2.700427, 0.      , 0.      ],
       [2.2597  , 2.38366 , 1.007861, 2.456765, 2.693789, 1.169568,...
FAILED linalg/tests/test_lapack.py::test_pstf2 - AssertionError: 
Not equal to tolerance rtol=0, atol=0.000119209

Mismatched elements: 85 / 100 (85%)
Max absolute difference: 3.529682
Max relative difference: 1.4060138
 x: array([[3.487075, 2.2597  , 1.555289, 2.763015, 3.431685, 2.046993,
        2.797032, 2.700427, 3.422926, 2.103169],
       [2.2597  , 2.130606, 1.477844, 1.633565, 2.117459, 1.866201,...
 y: array([[3.487075, 2.2597  , 1.555289, 2.763015, 3.431685, 2.046993,
        2.797032, 2.700427, 0.      , 0.      ],
       [2.2597  , 2.38366 , 1.007861, 2.456765, 2.693789, 1.169568,...

linux + ppc64le: 6 failures

FAILED stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf-1.0-points0-0.0-0.25] - AssertionError: 
Arrays are not almost equal to 7 decimals

Mismatched elements: 10 / 10 (100%)
Max absolute difference: 0.02345738
Max relative difference: inf
 x: array([0.0116484, 0.0116608, 0.0116732, 0.0116857, 0.0116982, 0.0117106,
       0.0117228, 0.0117349, 0.0117468, 0.0234574])
 y: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
FAILED stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf-1.0-points0-0.0-0.5] - AssertionError: 
Arrays are not almost equal to 7 decimals

Mismatched elements: 10 / 10 (100%)
Max absolute difference: 0.0005926
Max relative difference: inf
 x: array([0.0005859, 0.0005867, 0.0005875, 0.0005883, 0.0005891, 0.0005899,
       0.0005907, 0.0005915, 0.0005923, 0.0005926])
 y: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
FAILED stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf-1.0-points0-0.0-0.75] - AssertionError: 
Arrays are not almost equal to 7 decimals

Mismatched elements: 10 / 10 (100%)
Max absolute difference: 2.33537973e-05
Max relative difference: inf
 x: array([-2.3353797e-05, -2.3312442e-05, -2.3271041e-05, -2.3229594e-05,
       -2.3188102e-05, -2.3146564e-05, -2.3104981e-05, -2.3063353e-05,
       -2.3021680e-05, -2.2979962e-05])
 y: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
FAILED stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf--1.0-points2-1.0-0.25] - AssertionError: 
Arrays are not almost equal to 7 decimals

Mismatched elements: 10 / 10 (100%)
Max absolute difference: 0.11131309
Max relative difference: 0.11131309
 x: array([0.8886869, 0.9003975, 0.9004094, 0.9004215, 0.9004337, 0.9004461,
       0.9004586, 0.9004711, 0.9004835, 0.9004958])
 y: array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
FAILED stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf--1.0-points2-1.0-0.5] - AssertionError: 
Arrays are not almost equal to 7 decimals

Mismatched elements: 10 / 10 (100%)
Max absolute difference: 0.01326475
Max relative difference: 0.01326475
 x: array([0.9867353, 0.9867356, 0.9867364, 0.9867372, 0.986738 , 0.9867388,
       0.9867396, 0.9867404, 0.9867412, 0.986742 ])
 y: array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
FAILED stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf--1.0-points2-1.0-0.75] - AssertionError: 
Arrays are not almost equal to 7 decimals

Mismatched elements: 10 / 10 (100%)
Max absolute difference: 0.00165994
Max relative difference: 0.00165994
 x: array([0.9983401, 0.9983401, 0.9983401, 0.9983402, 0.9983402, 0.9983403,
       0.9983403, 0.9983404, 0.9983404, 0.9983404])
 y: array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

osx + x64 + accelerate: 1 failure

FAILED interpolate/tests/test_rbfinterp.py::TestRBFInterpolatorNeighbors20::test_pickleable - AssertionError: 
Arrays are not equal

Mismatched elements: 7 / 50 (14%)
Max absolute difference: 5.55111512e-17
Max relative difference: 9.28510161e-16
 x: array([-0.238461,  0.257024, -0.811244,  0.295313,  0.132623,  0.2038  ,
       -0.99615 ,  0.29755 , -0.019889,  0.230791, -0.516801,  0.279867,
        0.227767,  0.178071, -0.172756,  0.251327, -0.406818,  0.271919,...
 y: array([-0.238461,  0.257024, -0.811244,  0.295313,  0.132623,  0.2038  ,
       -0.99615 ,  0.29755 , -0.019889,  0.230791, -0.516801,  0.279867,
        0.227767,  0.178071, -0.172756,  0.251327, -0.406818,  0.271919,...

osx + x64 + mkl + pypy: 1 failure

=========================== short test summary info ============================
FAILED linalg/tests/test_decomp_cossin.py::test_cossin[False-40-12-20-float32] - AssertionError: 
Not equal to tolerance rtol=0, atol=1.19209e-06

Mismatched elements: 268 / 1600 (16.8%)
Max absolute difference: 1.121828
Max relative difference: 2.0000215
 x: array([[ 0.088203, -0.328942, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048775,  0.223085, ...,  0.      ,  0.      ,...
 y: array([[ 0.088203, -0.328941, -0.031064, ...,  0.      ,  0.      ,
         0.      ],
       [ 0.233614,  0.048776,  0.223085, ...,  0.      ,  0.      ,...
= 1 failed, 42473 passed, 2825 skipped, 137 xfailed, 9 xpassed, 2 warnings in 2072.54s (0:34:32) =

windows + blis: 1 failure

same as for linux+x64+{mkl,openblas}+avx512

=========================== short test summary info ===========================
FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab-nonsymposdef-F] - assert -11 == 0
= 1 failed, 55010 passed, 2871 skipped, 245 xfailed, 11 xpassed, 90 warnings in 1084.88s (0:18:04) =

windows + openblas + CPython: 1 failure

=========================== short test summary info ===========================
FAILED linalg/tests/test_decomp.py::TestEig::test_bad_geneig - AssertionError: 
Not equal to tolerance rtol=1e-07, atol=1e-07

array([[ 1.    ,  0.    ,  0.    ,  0.    ],
       [ 0.    ,  1.    ,  0.    ,  0.    ],
       [ 0.    ,  0.    , -8.9975,  0.    ],
       [ 0.    ,  0.    ,  0.    , -8.9975]])
array([[ 0. ,  0. ,  1. ,  0. ],
       [ 0. ,  0. ,  0. ,  1. ],
       [ 1. ,  0. ,  0. , -0.1],
       [ 0. ,  1. ,  0.1,  0. ]])
Mismatched elements: 3 / 4 (75%)
Max absolute difference: 6.
Max relative difference: 2.03389831
 x: array([0.000000e+00-3.05j, 0.000000e+00+3.05j, 1.001943e-16-2.95j,
       1.001943e-16+2.95j])
 y: array([0.-3.05j, 0.-2.95j, 0.+2.95j, 0.+3.05j])
= 1 failed, 55010 passed, 2871 skipped, 245 xfailed, 11 xpassed, 1 warning in 954.27s (0:15:54) =

h-vetinari · 2024-01-10T02:00:58Z

I had forgotten to update the TL;DR 😅 - unfortunately, while last time it look like things were improving, this has not continued (in fact, regressed). I also added a separate overview about the curious case of scipy/scipy#15533.

h-vetinari · 2024-04-12T03:30:00Z

As 1.13 has been released, I'm closing this as obsolete.

h-vetinari force-pushed the 1.11_blas_vars branch 2 times, most recently from 6beb2ad to 6ab01d3 Compare June 26, 2023 10:25

h-vetinari added 5 commits January 9, 2024 13:45

test for all blas variants

e172254

distinguish mkl/openblas runs for avx512 presence/absence

924ca8b

enable setting pagefile size on windows

76742ac

unskip some tests

b9cda32

MNT: Re-rendered with conda-build 3.28.2, conda-smithy 3.30.2, and co…

30ea0ce

…nda-forge-pinning 2024.01.08.19.20.26

h-vetinari force-pushed the 1.11_blas_vars branch from 6ab01d3 to 30ea0ce Compare January 9, 2024 03:00

h-vetinari mentioned this pull request Jan 10, 2024

BUG: test failure in test_x0_equals_Mb with bicgstab scipy/scipy#15533

Open

h-vetinari mentioned this pull request Jan 23, 2024

TEST: 1.12.x + blas variants #267

Draft

h-vetinari mentioned this pull request Apr 12, 2024

TEST: 1.13.x + blas variants #273

Draft

h-vetinari closed this Apr 12, 2024

h-vetinari deleted the 1.11_blas_vars branch April 12, 2024 03:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TEST: 1.11.x + blas variants #242

TEST: 1.11.x + blas variants #242

h-vetinari commented Jun 26, 2023 •

edited

conda-forge-webservices bot commented Jun 26, 2023

conda-forge-webservices bot commented Jun 26, 2023

conda-forge-webservices bot commented Jun 26, 2023

h-vetinari commented Jun 27, 2023 •

edited

h-vetinari commented Jun 27, 2023

h-vetinari commented Jan 10, 2024 •

edited

h-vetinari commented Jan 10, 2024

h-vetinari commented Apr 12, 2024

TEST: 1.11.x + blas variants #242

TEST: 1.11.x + blas variants #242

Conversation

h-vetinari commented Jun 26, 2023 • edited

conda-forge-webservices bot commented Jun 26, 2023

conda-forge-webservices bot commented Jun 26, 2023

conda-forge-webservices bot commented Jun 26, 2023

h-vetinari commented Jun 27, 2023 • edited

Scipy 1.11.0

TL;DR: slow & steady improvements continue

Notable

Good News

Bad News

Restrictions

Details

h-vetinari commented Jun 27, 2023

h-vetinari commented Jan 10, 2024 • edited

Scipy 1.11.4

TL;DR: the bad old days are beckoning

Notable

Good News

Bad News

Restrictions

Details

The curious case of test_x0_equals_Mb[bicgstab-nonsymposdef-F]

h-vetinari commented Jan 10, 2024

h-vetinari commented Apr 12, 2024

h-vetinari commented Jun 26, 2023 •

edited

h-vetinari commented Jun 27, 2023 •

edited

h-vetinari commented Jan 10, 2024 •

edited

The curious case of `test_x0_equals_Mb[bicgstab-nonsymposdef-F]`