TST: new errors on linux-{aarch64,ppc64le} related to cdf with fft in TestLevyStable

[h-vetinari]

I'm getting some new errors in conda-forge/scipy-feedstock#255, which are most likely triggered somehow by a third-party dependency, as I didn't see these failures on any recent runs, and there's no new scipy version to blame. It started today, as of a couple hours ago, going from all green (modulo some spurious timeouts), to all aarch/ppc64le runs finishing with the exact same set of test failures.

Coincidentally, all the failing tests raise the same RuntimeWarning:

stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf-1.0-points0-0.0-0.25]
stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf-1.0-points0-0.0-0.5]
stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf-1.0-points0-0.0-0.75]
stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf--1.0-points2-1.0-0.25]
stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf--1.0-points2-1.0-0.5]
stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf--1.0-points2-1.0-0.75]
  $PREFIX/lib/python3.10/site-packages/scipy/stats/_levy_stable/__init__.py:1068: RuntimeWarning: Cumulative density calculations experimental for FFT method. Use piecewise method instead.
    warnings.warn(

While the logs talk about missed toleranced, I think the failures must be considered total, given that one side of the comparision is unexpectedly constant zero or one.

test failures incl. main error message

=========================== short test summary info ============================
FAILED stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf-1.0-points0-0.0-0.25] - AssertionError: 
Arrays are not almost equal to 7 decimals

Mismatched elements: 10 / 10 (100%)
Max absolute difference: 0.02345738
Max relative difference: inf
 x: array([0.0116484, 0.0116608, 0.0116732, 0.0116857, 0.0116982, 0.0117106,
       0.0117228, 0.0117349, 0.0117468, 0.0234574])
 y: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
FAILED stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf-1.0-points0-0.0-0.5] - AssertionError: 
Arrays are not almost equal to 7 decimals

Mismatched elements: 10 / 10 (100%)
Max absolute difference: 0.0005926
Max relative difference: inf
 x: array([0.0005859, 0.0005867, 0.0005875, 0.0005883, 0.0005891, 0.0005899,
       0.0005907, 0.0005915, 0.0005923, 0.0005926])
 y: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
FAILED stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf-1.0-points0-0.0-0.75] - AssertionError: 
Arrays are not almost equal to 7 decimals

Mismatched elements: 10 / 10 (100%)
Max absolute difference: 2.33537973e-05
Max relative difference: inf
 x: array([-2.3353797e-05, -2.3312442e-05, -2.3271041e-05, -2.3229594e-05,
       -2.3188102e-05, -2.3146564e-05, -2.3104981e-05, -2.3063353e-05,
       -2.3021680e-05, -2.2979962e-05])
 y: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
FAILED stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf--1.0-points2-1.0-0.25] - AssertionError: 
Arrays are not almost equal to 7 decimals

Mismatched elements: 10 / 10 (100%)
Max absolute difference: 0.11131309
Max relative difference: 0.11131309
 x: array([0.8886869, 0.9003975, 0.9004094, 0.9004215, 0.9004337, 0.9004461,
       0.9004586, 0.9004711, 0.9004835, 0.9004958])
 y: array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
FAILED stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf--1.0-points2-1.0-0.5] - AssertionError: 
Arrays are not almost equal to 7 decimals

Mismatched elements: 10 / 10 (100%)
Max absolute difference: 0.01326475
Max relative difference: 0.01326475
 x: array([0.9867353, 0.9867356, 0.9867364, 0.9867372, 0.986738 , 0.9867388,
       0.9867396, 0.9867404, 0.9867412, 0.986742 ])
 y: array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
FAILED stats/tests/test_distributions.py::TestLevyStable::test_distribution_outside_support[cdf--1.0-points2-1.0-0.75] - AssertionError: 
Arrays are not almost equal to 7 decimals

Mismatched elements: 10 / 10 (100%)
Max absolute difference: 0.00165994
Max relative difference: 0.00165994
 x: array([0.9983401, 0.9983401, 0.9983401, 0.9983402, 0.9983402, 0.9983403,
       0.9983403, 0.9983404, 0.9983404, 0.9983404])
 y: array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
= 6 failed, 42445 passed, 2805 skipped, 134 xfailed, 12 xpassed, 6 warnings in 5683.39s (1:34:43) =

[rgommers]

There are also TestLevyStable failures on s390x reported in gh-17912. Something is not quite right with the C code in stats/_levy_stable/ I think.

[tylerjereddy]

python dev.py test -t scipy/stats/tests/test_distributions.py::TestLevyStable (and the full suite) passes for me with Python 3.9 on the Linux aarch64 architecture below the fold, along with Red Hat GNU toolchain 8.5.0 (and same thing for 11.2.0, and also tests still pass if I go back to NumPy 1.22.4 per your log).

Architecture:        aarch64
Byte Order:          Little Endian
CPU(s):              256
On-line CPU(s) list: 0-255
Thread(s) per core:  4
Core(s) per socket:  32
Socket(s):           2
NUMA node(s):        2
Vendor ID:           Cavium
Model:               0
Model name:          ThunderX2 99xx
Stepping:            0x1
BogoMIPS:            400.00
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            32768K
NUMA node0 CPU(s):   0-127
NUMA node1 CPU(s):   128-255
Flags:               fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics cpuid asimdrdm

(and using python tools/openblas_support.py for OpenBLAS).

My pip list is below:

Package            Version
------------------ ----------
appdirs            1.4.4
attrs              22.1.0
bashlex            0.18
beniget            0.4.1
bracex             2.3.post1
certifi            2022.12.7
charset-normalizer 2.1.1
cibuildwheel       2.12.0
click              8.1.3
cloudpickle        2.2.0
commonmark         0.9.1
Cython             3.0.2
devpy              0.1a3.dev0
doit               0.36.0
exceptiongroup     1.0.4
execnet            1.9.0
filelock           3.9.0
gast               0.5.3
hypothesis         6.87.1
idna               3.4
importlib-metadata 5.2.0
iniconfig          1.1.1
meson              1.2.2
meson-python       0.14.0
ninja              1.10.2.3
numpy              1.26.0
packaging          22.0
pip                23.2.1
platformdirs       2.6.2
pluggy             1.0.0
ply                3.11
pooch              1.6.0
pybind11           2.11.1
pydevtool          0.3.0
Pygments           2.13.0
pyproject-metadata 0.7.1
pytest             7.4.2
pytest-xdist       3.3.1
pythran            0.14.0
requests           2.28.1
rich               12.6.0
rich-click         1.6.0
setuptools         56.0.0
sortedcontainers   2.4.0
threadpoolctl      3.2.0
toml               0.10.2
tomli              2.0.1
typing_extensions  4.4.0
urllib3            1.26.13
zipp               3.11.0

I did notice some weaknesses for pytest running at high concurrency on the slower ARM cores though, having to reduce the level of concurrency because Different tests were collected between gw7 and gw31 type errors.

So I guess I'm saying I can't reproduce locally just yet, but do have convenient access to the hardware if there's a more specific route to reproduction.

[h-vetinari]

Thanks for checking! Good to know you're not hitting it. It might also somehow be related to QEMU, see discussion in #19210.

Though since we didn't observe it for several runs, and then all at once, I'm guessing either infrastructure or third party libs are at fault. Though it's possible of course - like Ralf was saying - that there are lurking bugs already, and perhaps we're just now starting to trigger them.

[h-vetinari]

This is fixed as of 1.12 from the POV of conda-forge, closing.