Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Recording Pypy Segfaults in Testing #2936

Open
CoolCat467 opened this issue Jan 24, 2024 · 12 comments
Open

Recording Pypy Segfaults in Testing #2936

CoolCat467 opened this issue Jan 24, 2024 · 12 comments

Comments

@CoolCat467
Copy link
Contributor

CoolCat467 commented Jan 24, 2024 •
edited

Gah, well see the segfaults don't exactly provide much information... But I agree that it's annoying. At this point ig we might as well make an issue here (on trio) to track things cause upstream pytest hasn't been able to solve this yet.

Originally posted by @A5rocks in #2930 (comment)

This issue is for recording segfaults during pypy runs in the github actions testing runner.

So far (also feel free to edit and add more):
Windows x64 pypy 3.9 nightly:
https://github.com/python-trio/trio/actions/runs/5705575738/job/15460400267?pr=2724
Ubuntu pypy-3.10-nightly:
https://github.com/python-trio/trio/actions/runs/7326018868/job/19951219901#step:5:1197
Ubuntu pypy-3.9:
https://github.com/python-trio/trio/actions/runs/7607032357/job/20713702030#step:5:1186
Ubuntu pypy-3.9-nightly:
https://github.com/python-trio/trio/actions/runs/7646760307/job/20836270507#step:5:1188

Not segfault but another error during Ubuntu pypy-3.9:
TypeError: expected some sort of stmt, got '@py_assert1'
https://github.com/python-trio/trio/actions/runs/7054572593/job/19203654087#step:5:1147
@jakkdl
Copy link
Member

jakkdl commented Jan 29, 2024 •
edited by CoolCat467

another instance of the ubuntu pypy-3.9: https://github.com/python-trio/trio/actions/runs/7698138955/job/20976753073#step:5:1193

@A5rocks
Copy link
Contributor

A5rocks commented Jan 29, 2024

Are we sure the Windows segfault is this?

@A5rocks
Copy link
Contributor

A5rocks commented Jan 29, 2024 •
edited by CoolCat467

and another ubuntu pypy-3.9 one: https://github.com/python-trio/trio/actions/runs/7703752769/job/20994747752#step:5:1186

@CoolCat467
Copy link
Contributor Author

CoolCat467 commented Jan 30, 2024 •
edited

Are we sure the Windows segfault is this?

No, the logs expired for that one.
EDIT:
Issue comment where I mentioned this particular segfault: #2724 (comment)

@jakkdl
Copy link
Member

jakkdl commented Jan 30, 2024

https://github.com/python-trio/trio/actions/runs/7712303460/job/21019559589?pr=2933

I think this is just a random network error?

macOS pypy-3.9-nightly

Run codecov/codecov-action@v3
  with:
    directory: empty
    token: 87cefb17-c44b-4f2f-8b30-1fff5769ce46
    name: macOS (pypy-3.9-nightly)
    flags: macOS,pypy-3.9-nightly
==> macos OS detected
Error: getaddrinfo ENOTFOUND uploader.codecov.io
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:118:26)

@jakkdl
Copy link
Member

jakkdl commented Jan 31, 2024

ubuntu pypy-3.9
https://github.com/python-trio/trio/actions/runs/7725935804/job/21061210801?pr=2941#step:5:1489

  Fatal Python error: Segmentation fault
  
  Stack (most recent call first, approximate line numbers):
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/jedi/api/completion.py", line 68 in filter_names
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/jedi/api/completion.py", line 141 in complete
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/jedi/api/__init__.py", line 194 in complete
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/jedi/api/helpers.py", line 469 in wrapper
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/trio/_tests/test_exports.py", line 245 in test_static_tool_sees_class_members
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 187 in pytest_pyfunc_call
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 27 in _multicall
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 106 in _hookexec
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 479 in __call__
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/python.py", line 1790 in runtest
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 160 in pytest_runtest_call
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 27 in _multicall
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 106 in _hookexec
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 479 in __call__
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 262 in <lambda>
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 318 in from_call
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 247 in call_runtest_hook
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 219 in call_and_report
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 119 in runtestprotocol
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/runner.py", line 111 in pytest_runtest_protocol
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 27 in _multicall
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 106 in _hookexec
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 479 in __call__
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 338 in pytest_runtestloop
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 27 in _multicall
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 106 in _hookexec
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 479 in __call__
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 321 in _main
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 258 in wrap_session
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_callers.py", line 27 in _multicall
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_manager.py", line 106 in _hookexec
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pluggy/_hooks.py", line 479 in __call__
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 135 in main
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/_pytest/config/__init__.py", line 185 in console_main
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/pytest/__main__.py", line 1 in <module>
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/coverage/execfile.py", line 169 in run
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 810 in do_run
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 608 in command_line
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/coverage/cmdline.py", line 961 in main
    File "/opt/hostedtoolcache/PyPy/3.9.18/x64/bin/coverage", line 3 in <module>
    File "<builtin>/app_main.py", line 1017 in execfile
    File "<builtin>/app_main.py", line 131 in run_toplevel
    File "<builtin>/app_main.py", line 737 in run_command_line
    File "<builtin>/app_main.py", line 1131 in entry_point
  ../../../../../../opt/hostedtoolcache/PyPy/3.9.18/x64/lib/pypy3.9/site-packages/trio/_tests/test_exports.py::test_static_tool_sees_class_members[jedi-trio.lowlevel] ::endgroup::
  ./ci.sh: line 151:  1981 Segmentation fault      (core dumped) COVERAGE_PROCESS_START=$(pwd)/../pyproject.toml coverage run --rcfile=../pyproject.toml -m pytest -ra --junitxml=../test-results.xml --run-slow ${INSTALLDIR} --verbose --durations=10 $flags

@A5rocks
Copy link
Contributor

A5rocks commented Feb 5, 2024

Oh I see I forgot to skip jedi on pypy for test_static_tool_sees_class_members (I fixed that for test_static_tool_sees_all_symbols)

@jakkdl
Copy link
Member

jakkdl commented Feb 13, 2024

another instance of the same ubuntu pypy-3.9 error: https://github.com/python-trio/trio/actions/runs/7886026350/job/21518402835#step:5:1142

succeeded on rerun. Looking closer at the logs, the error is from a runner using Runner image version 20240204.1.0, and the success one uses 20240212.1.0. Although 7 hours ago a 20240204.1.0 ubuntu pypy-3.9 run did succeed: https://github.com/python-trio/trio/actions/runs/7883080425/job/21509366109 so that's probably not a thing either.

@Zac-HD
Copy link
Member

Zac-HD commented Mar 28, 2024

Recently fixed upstream, so I think we're done! pytest-dev/pytest#11168 (comment)

https://www.pypy.org/posts/2024/03/fixing-bug-incremental-gc.html is well worth reading, too.

@Zac-HD Zac-HD closed this as completed Mar 28, 2024
@jakkdl
Copy link
Member

jakkdl commented Mar 31, 2024

Great writeup! Very impressive debugging skillz

@CoolCat467
Copy link
Contributor Author

We had another one in #2972, https://github.com/python-trio/trio/actions/runs/8579945946/job/23515500181#step:5:1232, have we updated pypy or anything?

@CoolCat467 CoolCat467 reopened this Apr 6, 2024
@Zac-HD
Copy link
Member

Zac-HD commented Apr 6, 2024

Looks like the latest version is PyPy 7.3.15, from Jan 15th - we're running this, and the fix just hasn't been released yet. From the post:

At the point of publishing this post, the fixes got merged to the 2.7/3.9/3.10 branches of PyPy, and will be part of the next release (v7.3.16).

Happy to leave this open until PyPy is released, but since we get patch updates automatically via GitHub actions I think we're basically done here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jakkdl @Zac-HD @A5rocks @CoolCat467