CI: Azure Linux PyPy3 tests crash with an illegal instruction.

[WarrenWeckesser]

In recent PRs (e.g. gh-15782, gh-15791, gh-15794), the Azure Linux PyPy3 test is crashing.

The end of the log from gh-15794:

2020-03-22T05:21:47.5132312Z NumPy version 1.19.0.dev0+b1cc781
2020-03-22T05:21:47.5133522Z NumPy relaxed strides checking option: True
2020-03-22T05:21:47.5134185Z ============================= test session starts ==============================
2020-03-22T05:21:47.5135709Z platform linux -- Python 3.6.9[pypy-7.3.1-alpha], pytest-5.4.1, py-1.8.1, pluggy-0.13.1
2020-03-22T05:21:47.5224006Z rootdir: /home/vsts/work/1/s, inifile: pytest.ini
2020-03-22T05:21:47.5225161Z plugins: cov-2.8.1, hypothesis-5.6.0
2020-03-22T05:22:09.4614643Z collected 10877 items / 107 deselected / 10770 selected
2020-03-22T05:22:09.4748033Z 
2020-03-22T05:22:09.4849532Z numpy/compat/tests/test_compat.py .                                      [  0%]
2020-03-22T05:22:09.5053251Z numpy/core/tests/test__exceptions.py ...                                 [  0%]
2020-03-22T05:22:09.5447106Z numpy/core/tests/test_abc.py .....                                       [  0%]
2020-03-22T05:22:12.8217273Z numpy/core/tests/test_api.py .....................                       [  0%]
2020-03-22T05:22:14.7450863Z numpy/core/tests/test_arrayprint.py ..x.................s............... [  0%]
2020-03-22T05:22:14.8121182Z ........                                                                 [  0%]
2020-03-22T05:22:14.8369515Z numpy/core/tests/test_cpu_features.py .ss                                [  0%]
2020-03-22T05:22:15.1816915Z numpy/core/tests/test_datetime.py ...................................... [  1%]
2020-03-22T05:22:16.0520572Z ........................................................................ [  1%]
2020-03-22T05:22:16.7831398Z ........................................................................ [  2%]
2020-03-22T05:22:18.0856415Z ............................x..........x..x..........x.                  [  2%]
2020-03-22T05:22:18.3400512Z numpy/core/tests/test_defchararray.py .................................. [  3%]
2020-03-22T05:22:18.6553630Z ..............................................                           [  3%]
2020-03-22T05:22:18.8717506Z numpy/core/tests/test_deprecations.py .................................. [  3%]
2020-03-22T05:22:18.9070418Z ....                                                                     [  4%]
2020-03-22T05:22:19.2289104Z numpy/core/tests/test_dtype.py ...........................s............. [  4%]
2020-03-22T05:22:19.7590417Z ...........ssssssssssssssssssssssssssssssssssssx........................ [  5%]
2020-03-22T05:22:21.2749281Z ........................................................................ [  5%]
2020-03-22T05:22:21.6616065Z ........................................................................ [  6%]
2020-03-22T05:22:22.0636224Z ........................................................................ [  7%]
2020-03-22T05:22:22.4513506Z ........................................................................ [  7%]
2020-03-22T05:22:22.8107049Z ........................................................................ [  8%]
2020-03-22T05:22:23.1314859Z ........................................................................ [  9%]
2020-03-22T05:22:23.4657531Z ........................................................................ [  9%]
2020-03-22T05:22:23.7767237Z ........................................................................ [ 10%]
2020-03-22T05:22:24.0690820Z ........................................................................ [ 11%]
2020-03-22T05:22:24.2964026Z ...............................................................          [ 11%]
2020-03-22T05:22:29.4489345Z numpy/core/tests/test_einsum.py ...........Fatal Python error: Illegal instruction
2020-03-22T05:22:29.4492134Z 
2020-03-22T05:22:29.4493309Z Stack (most recent call first, approximate line numbers):
2020-03-22T05:22:29.4495182Z   File "/home/vsts/work/1/s/build/testenv/site-packages/numpy/core/tests/test_einsum.py", line 236 in check_einsum_sums
2020-03-22T05:22:29.4497070Z   File "/home/vsts/work/1/s/build/testenv/site-packages/numpy/core/tests/test_einsum.py", line 553 in test_einsum_sums_float32
2020-03-22T05:22:29.4510528Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/_pytest/python.py", line 175 in pytest_pyfunc_call
2020-03-22T05:22:29.4512283Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/callers.py", line 157 in _multicall
2020-03-22T05:22:29.4513043Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/manager.py", line 84 in <lambda>
2020-03-22T05:22:29.4513806Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/manager.py", line 90 in _hookexec
2020-03-22T05:22:29.4514392Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/hooks.py", line 272 in __call__
2020-03-22T05:22:29.4514967Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/_pytest/python.py", line 1477 in runtest
2020-03-22T05:22:29.4515574Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/_pytest/runner.py", line 126 in pytest_runtest_call
2020-03-22T05:22:29.4516187Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/manager.py", line 84 in <lambda> [jit inlined]
2020-03-22T05:22:29.4516829Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/manager.py", line 90 in _hookexec [jit inlined]
2020-03-22T05:22:29.4517628Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/hooks.py", line 272 in __call__ [jit inlined]
2020-03-22T05:22:29.4518475Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/_pytest/runner.py", line 217 in <lambda> [jit inlined]
2020-03-22T05:22:29.4519120Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/_pytest/runner.py", line 237 in from_call [jitted]
2020-03-22T05:22:29.4519756Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/_pytest/runner.py", line 237 in from_call
2020-03-22T05:22:29.4520381Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/_pytest/runner.py", line 204 in call_runtest_hook
2020-03-22T05:22:29.4521166Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/_pytest/runner.py", line 183 in call_and_report
2020-03-22T05:22:29.4521798Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/_pytest/runner.py", line 90 in runtestprotocol
2020-03-22T05:22:29.4522579Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/_pytest/runner.py", line 83 in pytest_runtest_protocol
2020-03-22T05:22:29.4523221Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/callers.py", line 157 in _multicall [jit inlined]
2020-03-22T05:22:29.4524040Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/manager.py", line 84 in <lambda> [jit inlined]
2020-03-22T05:22:29.4524785Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/manager.py", line 90 in _hookexec [jit inlined]
2020-03-22T05:22:29.4535682Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/hooks.py", line 272 in __call__ [jitted]
2020-03-22T05:22:29.4536567Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/hooks.py", line 272 in __call__
2020-03-22T05:22:29.4545685Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/_pytest/main.py", line 260 in pytest_runtestloop
2020-03-22T05:22:29.4547335Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/callers.py", line 157 in _multicall
2020-03-22T05:22:29.4548691Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/manager.py", line 84 in <lambda>
2020-03-22T05:22:29.4549622Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/manager.py", line 90 in _hookexec
2020-03-22T05:22:29.4551523Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/hooks.py", line 272 in __call__
2020-03-22T05:22:29.4552740Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/_pytest/main.py", line 243 in _main
2020-03-22T05:22:29.4553981Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/_pytest/main.py", line 178 in wrap_session
2020-03-22T05:22:29.4556505Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/_pytest/main.py", line 239 in pytest_cmdline_main
2020-03-22T05:22:29.4557790Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/callers.py", line 157 in _multicall
2020-03-22T05:22:29.4559373Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/manager.py", line 84 in <lambda>
2020-03-22T05:22:29.4560873Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/manager.py", line 90 in _hookexec
2020-03-22T05:22:29.4561940Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/pluggy/hooks.py", line 272 in __call__
2020-03-22T05:22:29.4563528Z   File "/home/vsts/.local/lib/pypy3.6/site-packages/_pytest/config/__init__.py", line 95 in main
2020-03-22T05:22:29.4564298Z   File "/home/vsts/work/1/s/build/testenv/site-packages/numpy/_pytesttester.py", line 78 in __call__
2020-03-22T05:22:29.4565647Z   File "runtests.py", line 68 in main
2020-03-22T05:22:29.4566052Z   File "runtests.py", line 2 in <module>
2020-03-22T05:22:29.4566462Z   File "<builtin>/app_main.py", line 803 in execfile
2020-03-22T05:22:29.4566871Z   File "<builtin>/app_main.py", line 90 in run_toplevel
2020-03-22T05:22:29.4567309Z   File "<builtin>/app_main.py", line 562 in run_command_line
2020-03-22T05:22:29.4567731Z   File "<builtin>/app_main.py", line 874 in entry_point
2020-03-22T05:22:29.6943890Z tools/pypy-test.sh: line 44:  5683 Illegal instruction     (core dumped) pypy3/bin/pypy3 runtests.py --debug-info --show-build-log -v -- -rsx --junitxml=junit/test-results.xml --durations 10
2020-03-22T05:22:29.6969941Z 
2020-03-22T05:22:29.7049545Z ##[error]Bash exited with code '132'.
2020-03-22T05:22:29.7084384Z ##[section]Finishing: Run PyPy3 Build / Tests

[mattip]

I can reproduce this by running bash tools/pypy_bash.sh on my local (none-avx512f) machine. The segfault is in OpenBLAS

#0  0x00007ffff00431a8 in sgemm_kernel_direct () \
    from /tmp/openblas/lib/libopenblas.so.0
#1  0x00007fffef0b1892 in cblas_sgemm () 
    from /tmp/openblas/lib/libopenblas.so.0
#2  0x00007ffff0f3fb68 in cblas_matrixproduct ()
   from /home/matti/pypy_stuff/numpy/build/testenv/site-packages/numpy/core/_multiarray_umath.pypy36-pp73-x86_64-linux-gnu.so
#3  0x00007ffff0de06c1 in PyArray_MatrixProduct2 ()

I am not sure why PyPy exhibits this when the other builds don't.

[eric-wieser]

Can we skip the test causing the issue? This is making CI fail on every PR.

[mattip]

Please leave this issue open, since the resolution in gh-15809 is hopefully temporary.

[mattip]

I think we can close this, the PyPy job was restored and still sometimes segfaults but does so differently now. We also moved to OpenBLAS0.3.10. Please reopen if this segfault returns.