TST, MAINT: some tests blocking 1.11.0 on MacOS ARM64 with NumPy `1.25.0` #18732

tylerjereddy · 2023-06-23T18:35:19Z

A few more issues blocked me during the release process today on MacOS ARM64 Cirrus CI: https://cirrus-ci.com/build/6410513341480960

Actual test failures appear to be moderate tolerance issues, presumably just showing up now because of NumPy 1.25.0 being in the mix for the final tests in the release-specific Cirrus CI cibuildwheel workflows:

FAILED signal/tests/test_signaltools.py::TestDecimate::test_complex_iir_dlti - AssertionError: 
Arrays are not equal
Mismatched elements: 100 / 100 (100%)
Max absolute difference: 4.20794717e-16
Max relative difference: 3.17627439e-14
 x: array([ 0.003184-3.937370e-03j,  0.006656+7.492975e-03j,
        0.011597+1.481910e-02j,  0.016112+1.665752e-02j,
        0.018049+1.383353e-02j,  0.015539+8.792702e-03j,...
 y: array([ 0.003184-3.937370e-03j,  0.006656+7.492975e-03j,
        0.011597+1.481910e-02j,  0.016112+1.665752e-02j,
        0.018049+1.383353e-02j,  0.015539+8.792702e-03j,...
= 1 failed, 42005 passed, 2819 skipped, 12693 deselected, 135 xfailed, 10 xpassed, 4655 warnings in 288.08s (0:04:48) =

=========================== short test summary info ============================
FAILED signal/tests/test_filter_design.py::TestSOSFreqz::test_fs_param - AssertionError: 
Not equal to tolerance rtol=1e-07, atol=0
Mismatched elements: 1 / 512 (0.195%)
Max absolute difference: 3.14018492e-16
Max relative difference: 2.92960796e-13
 x: array([ 1.000000e+00+0.000000e+00j,  9.996396e-01-2.684364e-02j,
        9.985585e-01-5.367479e-02j,  9.967561e-01-8.048095e-02j,
        9.942321e-01-1.072496e-01j,  9.909856e-01-1.339681e-01j,...
 y: array([ 1.000000e+00+0.000000e+00j,  9.996396e-01-2.684364e-02j,
        9.985585e-01-5.367479e-02j,  9.967561e-01-8.048095e-02j,
        9.942321e-01-1.072496e-01j,  9.909856e-01-1.339681e-01j,...
= 1 failed, 42005 passed, 2819 skipped, 12693 deselected, 135 xfailed, 10 xpassed, 4655 warnings in 325.54s (0:05:25) =

There are also quite a large number of warnings, that perhaps won't cause failures on their own, mostly like this:

  /private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cibw-run-1lcwhxoa/cp311-macosx_arm64/venv-test/lib/python3.11/site-packages/scipy/interpolate/_fitpack_impl.py:347: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    aint, wrk = dfitpack.splint(t, c, k, a, b)

  /private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cibw-run-1lcwhxoa/cp311-macosx_arm64/venv-test/lib/python3.11/site-packages/scipy/stats/_continuous_distns.py:11204: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    return np.clip(np.float64(ufunc(x, k, df)), 0, 1)

The text was updated successfully, but these errors were encountered:

tylerjereddy · 2023-06-24T04:20:53Z

I made a few attempts to reproduce on ARM Mac this evening locally with cibuildwheel -- the process is surprisingly prone to problems, has a mandatory requirement for the Python.org Python over brew Python, and gets confused about all kinds of things related to the presence of brew compiler toolchain, so the promise of low-friction local debugging is still a bit far off. It also required sudo, though that may be our own doing, I didn't check too closely.

I mean, we may just need some simple tol bumps, but it is still nice to repro locally when reasonable.

andyfaff · 2023-06-24T04:41:13Z

Do you need any help on this one? I have an M1 at home.

* deals with the 1 of the 2 Apple silicon test failures in scipygh-18732 * I was able to reproduce locally, but because it was flaky I needed `pytest-repeat` for consistent failure: `CC=clang CXX=clang++ FC=gfortran-12 python dev.py test -t scipy/signal/tests/test_filter_design.py::TestSOSFreqz::test_fs_param -- --count=200` * I verified that with the provided patch/`atol` bump, we consistently pass even with `2000` repeats [skip circle]

* deals with the second part of scipygh-18732 * I was able to reproduce locally, but this one was even flakier--analysis of patch success below, and possibly related to subtle low-level MacOS ARM issues upstream in numpy/numpy#23996 (NumPy `1.25.0`) * before this patch, 4 trials of this command with `pytest-repeat`: `CC=clang CXX=clang++ FC=gfortran-12 python dev.py test -t scipy/signal/tests/test_signaltools.py::TestDecimate::test_complex_iir_dlti -- --count=20000` `81 failed, 19919 passed in 43.68s` `81 failed, 19919 passed in 43.79s` `20000 passed in 33.24s` `43 failed, 19957 passed in 35.17s` * after this patch, 6 trials of the same command: `20000 passed in 30.44s` `20000 passed in 33.08s` `20000 passed in 33.46s` `20000 passed in 33.45s` `20000 passed in 33.56s` `20000 passed in 33.30s` [skip circle]

andyfaff · 2023-06-25T06:34:32Z

I'm seeing unrelated test fails. Python3.10, numpy 1.25, scipy maintenance/1.11.0

scipy/fft/_pocketfft/tests/test_real_transforms.py:382: in test_dst4_definition_ortho
    assert_array_almost_equal(y, y2, decimal=dec)
        dec        = 6
        dt         = dtype('float32')
        mdata_x    = array([ 0.        ,  0.71428571,  1.42857143,  2.14285714,  2.85714286,
        3.57142857,  4.28571429,  5.        ,  5.71428571,  6.42857143,
        7.14285714,  7.85714286,  8.57142857,  9.28571429, 10.        ])
        rdt        = <class 'numpy.float32'>
        x          = array([ 0.        ,  0.71428573,  1.4285715 ,  2.142857  ,  2.857143  ,
        3.5714285 ,  4.285714  ,  5.        ,  5.714286  ,  6.428571  ,
        7.142857  ,  7.857143  ,  8.571428  ,  9.285714  , 10.        ],
      dtype=float32)
        y          = array([ 2.2527182e+01, -3.0485203e+00,  6.8829376e-01, -6.5594608e-01,
        1.3825622e-01, -3.0407611e-01,  2.43382... -1.2823799e-01, -3.5893470e-02, -9.6654691e-02,
       -4.9417440e-02, -7.6473922e-02, -6.1873209e-02], dtype=float32)
        y2         = array([ 2.25271838e+01, -3.04852029e+00,  6.88293833e-01, -6.55946133e-01,
        1.38256458e-01, -3.04076803e-01,  2...4780e-02, -1.28238488e-01, -3.58937516e-02, -9.66546037e-02,
       -4.94173596e-02, -7.64740413e-02, -6.18725511e-02])
/Users/andrew/miniconda3/envs/dev3/lib/python3.10/contextlib.py:79: in inner
    return func(*args, **kwds)
        args       = (array([ 2.2527182e+01, -3.0485203e+00,  6.8829376e-01, -6.5594608e-01,
        1.3825622e-01, -3.0407611e-01,  2.4338...780e-02, -1.28238488e-01, -3.58937516e-02, -9.66546037e-02,
       -4.94173596e-02, -7.64740413e-02, -6.18725511e-02]))
        func       = <function assert_array_almost_equal at 0x107f4f7f0>
        kwds       = {'decimal': 6}
        self       = <contextlib._GeneratorContextManager object at 0x107f7d090>
/Users/andrew/miniconda3/envs/dev3/lib/python3.10/contextlib.py:79: in inner
    return func(*args, **kwds)
E   AssertionError: 
E   Arrays are not almost equal to 6 decimals
E   
E   Mismatched elements: 1 / 15 (6.67%)
E   Max absolute difference: 2.20676786e-06
E   Max relative difference: 1.06287481e-05
E    x: array([ 2.252718e+01, -3.048520e+00,  6.882938e-01, -6.559461e-01,
E           1.382562e-01, -3.040761e-01,  2.433828e-02, -1.844277e-01,
E          -1.595955e-02, -1.282380e-01, -3.589347e-02, -9.665469e-02,
E          -4.941744e-02, -7.647392e-02, -6.187321e-02], dtype=float32)
E    y: array([ 2.252718e+01, -3.048520e+00,  6.882938e-01, -6.559461e-01,
E           1.382565e-01, -3.040768e-01,  2.433815e-02, -1.844278e-01,
E          -1.595948e-02, -1.282385e-01, -3.589375e-02, -9.665460e-02,
E          -4.941736e-02, -7.647404e-02, -6.187255e-02])
        args       = (<function assert_array_almost_equal.<locals>.compare at 0x2c09ecc10>, array([ 2.2527182e+01, -3.0485203e+00,  6.88293...780e-02, -1.28238488e-01, -3.58937516e-02, -9.66546037e-02,
       -4.94173596e-02, -7.64740413e-02, -6.18725511e-02]))
        func       = <function assert_array_compare at 0x107f4f640>
        kwds       = {'err_msg': '', 'header': 'Arrays are not almost equal to 6 decimals', 'precision': 6, 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x107f7d000>
_______________________________________________________________________________________________________________________________________________________ test_uint64_max _______________________________________________________________________________________________________________________________________________________
scipy/ndimage/tests/test_datatypes.py:62: in test_uint64_max
    assert_(x[2] == int(float(big)))
E   AssertionError
        arr        = array([18446744073709550591, 18446744073709550591, 18446744073709550591],
      dtype=uint64)
        big        = 18446744073709550591
        inds       = array([[-0.1,  0.9,  1.9]])
        x          = array([                   0, 18446744073709549568, 18446744073709551615],
      dtype=uint64)
__________________________________________________________________________________________________________________ TestHyp2f1.test_a_b_neg_int_after_euler_hypergeometric_transformation[hyp2f1_test_case0] ___________________________________________________________________________________________________________________
scipy/special/tests/test_hyp2f1.py:530: in test_a_b_neg_int_after_euler_hypergeometric_transformation
    assert_allclose(hyp2f1(a, b, c, z), expected, rtol=rtol)
        a          = -0.5
        b          = -0.9629749245209605
        c          = -15.5
        expected   = (0.9778506962676361+0.044083801141231616j)
        hyp2f1_test_case = Hyp2f1TestCase(a=-0.5, b=-0.9629749245209605, c=-15.5, z=(1.1578947368421053-1.1578947368421053j), expected=(0.9778506962676361+0.044083801141231616j), rtol=1e-12)
        rtol       = 1e-12
        self       = <scipy.special.tests.test_hyp2f1.TestHyp2f1 object at 0x28f5b8ca0>
        z          = (1.1578947368421053-1.1578947368421053j)
/Users/andrew/miniconda3/envs/dev3/lib/python3.10/contextlib.py:79: in inner
    return func(*args, **kwds)
E   AssertionError: 
E   Not equal to tolerance rtol=1e-12, atol=0
E   
E   Mismatched elements: 1 / 1 (100%)
E   Max absolute difference: 2.83181888e-12
E   Max relative difference: 2.89302401e-12
E    x: array(0.977851+0.044084j)
E    y: array(0.977851+0.044084j)
        args       = (<function assert_allclose.<locals>.compare at 0x2f8b7d3f0>, array(0.9778507+0.0440838j), array(0.9778507+0.0440838j))
        func       = <function assert_array_compare at 0x107f4f640>
        kwds       = {'equal_nan': True, 'err_msg': '', 'header': 'Not equal to tolerance rtol=1e-12, atol=0', 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x107f7d000>
_________________________________________________________________________________________________________________________________________ TestHyp2f1.test_region4[hyp2f1_test_case47] _________________________________________________________________________________________________________________________________________
scipy/special/tests/test_hyp2f1.py:1394: in test_region4
    assert_allclose(hyp2f1(a, b, c, z), expected, rtol=rtol)
        a          = -0.5
        b          = -15.5
        c          = -7.949900487447654
        expected   = (-2258.1590330318213+8860.193389158803j)
        hyp2f1_test_case = Hyp2f1TestCase(a=-0.5, b=-15.5, c=-7.949900487447654, z=(0.4172413793103451-0.8724137931034484j), expected=(-2258.1590330318213+8860.193389158803j), rtol=1e-10)
        rtol       = 1e-10
        self       = <scipy.special.tests.test_hyp2f1.TestHyp2f1 object at 0x28f60d060>
        z          = (0.4172413793103451-0.8724137931034484j)
/Users/andrew/miniconda3/envs/dev3/lib/python3.10/contextlib.py:79: in inner
    return func(*args, **kwds)
E   AssertionError: 
E   Not equal to tolerance rtol=1e-10, atol=0
E   
E   Mismatched elements: 1 / 1 (100%)
E   Max absolute difference: 1.09362783e-06
E   Max relative difference: 1.19608052e-10
E    x: array(-2258.159032+8860.19339j)
E    y: array(-2258.159033+8860.193389j)
        args       = (<function assert_allclose.<locals>.compare at 0x2f8b7f130>, array(-2258.159032+8860.19338952j), array(-2258.15903303+8860.19338916j))
        func       = <function assert_array_compare at 0x107f4f640>
        kwds       = {'equal_nan': True, 'err_msg': '', 'header': 'Not equal to tolerance rtol=1e-10, atol=0', 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x107f7d000>
_________________________________________________________________________________________________________________________________________ TestHyp2f1.test_region5[hyp2f1_test_case3] __________________________________________________________________________________________________________________________________________
scipy/special/tests/test_hyp2f1.py:1891: in test_region5
    assert_allclose(hyp2f1(a, b, c, z), expected, rtol=rtol)
        a          = 16.087593263474208
        b          = 1.0
        c          = -15.964218273004214
        expected   = (85592537010.05054-8061416766688.324j)
        hyp2f1_test_case = Hyp2f1TestCase(a=16.087593263474208, b=1.0, c=-15.964218273004214, z=(0.6448275862068968+0.8724137931034486j), expected=(85592537010.05054-8061416766688.324j), rtol=1e-14)
        rtol       = 1e-14
        self       = <scipy.special.tests.test_hyp2f1.TestHyp2f1 object at 0x28f60dcc0>
        z          = (0.6448275862068968+0.8724137931034486j)
/Users/andrew/miniconda3/envs/dev3/lib/python3.10/contextlib.py:79: in inner
    return func(*args, **kwds)
E   AssertionError: 
E   Not equal to tolerance rtol=1e-14, atol=0
E   
E   Mismatched elements: 1 / 1 (100%)
E   Max absolute difference: 0.13428089
E   Max relative difference: 1.66562939e-14
E    x: array(8.559254e+10-8.061417e+12j)
E    y: array(8.559254e+10-8.061417e+12j)
        args       = (<function assert_allclose.<locals>.compare at 0x2f8b7f400>, array(8.5592537e+10-8.06141677e+12j), array(8.5592537e+10-8.06141677e+12j))
        func       = <function assert_array_compare at 0x107f4f640>
        kwds       = {'equal_nan': True, 'err_msg': '', 'header': 'Not equal to tolerance rtol=1e-14, atol=0', 'verbose': True}
        self       = <contextlib._GeneratorContextManager object at 0x107f7d000>

1.11.0 1.25.0 sys.version_info(major=3, minor=10, micro=11, releaselevel='final', serial=0) Build Dependencies: blas: detection method: pkgconfig found: true include directory: /opt/arm64-builds/include lib directory: /opt/arm64-builds/lib name: openblas openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS= NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= SANDYBRIDGE MAX_THREADS=3 pc file directory: /opt/arm64-builds/lib/pkgconfig version: 0.3.21.dev lapack: detection method: pkgconfig found: true include directory: /opt/arm64-builds/include lib directory: /opt/arm64-builds/lib name: openblas openblas configuration: USE_64BITINT= DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS= NO_LAPACK= NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP= SANDYBRIDGE MAX_THREADS=3 pc file directory: /opt/arm64-builds/lib/pkgconfig version: 0.3.21.dev pybind11: detection method: config-tool include directory: unknown name: pybind11 version: 2.10.4 Compilers: c: commands: cc linker: ld64 name: clang version: 14.0.3 c++: commands: c++ linker: ld64 name: clang version: 14.0.3 cython: commands: cython linker: cython name: cython version: 0.29.35 fortran: commands: gfortran linker: ld64 name: gcc version: 12.2.0 pythran: include directory: ../../../../../miniconda3/envs/dev3/lib/python3.10/site-packages/pythran version: 0.13.1 Machine Information: build: cpu: aarch64 endian: little family: aarch64 system: darwin cross-compiled: false host: cpu: aarch64 endian: little family: aarch64 system: darwin Python Information: path: /Users/andrew/miniconda3/envs/dev3/bin/python3.10 version: '3.10'

tylerjereddy · 2023-06-25T14:00:56Z

@andyfaff yup, those are being tracked in gh-18308, which has been open since April, but doesn't seem to affect our current wheel toolchain at least. I fixed the original source of this issue, so I'll close this but leave that open

* deals with the 1 of the 2 Apple silicon test failures in scipygh-18732 * I was able to reproduce locally, but because it was flaky I needed `pytest-repeat` for consistent failure: `CC=clang CXX=clang++ FC=gfortran-12 python dev.py test -t scipy/signal/tests/test_filter_design.py::TestSOSFreqz::test_fs_param -- --count=200` * I verified that with the provided patch/`atol` bump, we consistently pass even with `2000` repeats [skip circle]

* deals with the second part of scipygh-18732 * I was able to reproduce locally, but this one was even flakier--analysis of patch success below, and possibly related to subtle low-level MacOS ARM issues upstream in numpy/numpy#23996 (NumPy `1.25.0`) * before this patch, 4 trials of this command with `pytest-repeat`: `CC=clang CXX=clang++ FC=gfortran-12 python dev.py test -t scipy/signal/tests/test_signaltools.py::TestDecimate::test_complex_iir_dlti -- --count=20000` `81 failed, 19919 passed in 43.68s` `81 failed, 19919 passed in 43.79s` `20000 passed in 33.24s` `43 failed, 19957 passed in 35.17s` * after this patch, 6 trials of the same command: `20000 passed in 30.44s` `20000 passed in 33.08s` `20000 passed in 33.46s` `20000 passed in 33.45s` `20000 passed in 33.56s` `20000 passed in 33.30s` [skip circle]

tylerjereddy added the maintenance Items related to regular maintenance tasks label Jun 23, 2023

tylerjereddy added this to the 1.11.0 milestone Jun 23, 2023

tylerjereddy mentioned this issue Jun 24, 2023

TST: flaky TestSOSFreqz::test_fs_param #18737

Merged

tylerjereddy mentioned this issue Jun 24, 2023

TST: flaky test_complex_iir_dlti #18738

Merged

tylerjereddy closed this as completed Jun 25, 2023

tylerjereddy mentioned this issue Jun 25, 2023

MAINT: more 1.11.0 backports #18743

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TST, MAINT: some tests blocking 1.11.0 on MacOS ARM64 with NumPy `1.25.0` #18732

TST, MAINT: some tests blocking 1.11.0 on MacOS ARM64 with NumPy `1.25.0` #18732

tylerjereddy commented Jun 23, 2023

tylerjereddy commented Jun 24, 2023

andyfaff commented Jun 24, 2023

andyfaff commented Jun 25, 2023

tylerjereddy commented Jun 25, 2023

TST, MAINT: some tests blocking 1.11.0 on MacOS ARM64 with NumPy 1.25.0 #18732

TST, MAINT: some tests blocking 1.11.0 on MacOS ARM64 with NumPy 1.25.0 #18732

Comments

tylerjereddy commented Jun 23, 2023

tylerjereddy commented Jun 24, 2023

andyfaff commented Jun 24, 2023

andyfaff commented Jun 25, 2023

tylerjereddy commented Jun 25, 2023

TST, MAINT: some tests blocking 1.11.0 on MacOS ARM64 with NumPy `1.25.0` #18732

TST, MAINT: some tests blocking 1.11.0 on MacOS ARM64 with NumPy `1.25.0` #18732