Segmentation fault in gc pinning after update from pypy3.5 to pypy3.6 #4900
Comments
|
JFI: The problem can be reproduced with different sets of GC configurations (including the standard one). So the GC configuration doesn't seem to be important. If there are any additional debug or approaches that I can use to localize this issue - please feel free to share them. |
|
The bug also occurs when you turn the gc hooks off, yes? Also, can you give us the steps how to reproduce this? If the software is closed source we cannot really easily help you, unless you can minimize the problem to a reproducer that we can run. |
Exactly. The issue appears periodically. On both debug and non-debug versions of PyPy. To sum up: the issue occurs despite debug / GC / version (any starting from 3.6)
I have not localized this issue yet. It was easy to localize (minimize) the neighbor issue (#4899) caused by gc hooks, because I could find the exact commit in our software, where these hooks were introduced.
The product is a large and complicated VoIP (voice over IP) solution that handles hundreds of calls per second. And I did not manage to minimize the problem. From my point of view, this segmentation fault happens randomly and I cannot relate it to anything. I'm trying to figure out whether this code can be published. Let me describe what I do to face this issue on app level: Could you please give me a hint (from the coredump I've provided) on what event it happened and I'll be able to simplify the steps to reproduce. Or is there any advice on what debug can be enabled? |
|
Greetings Carl, Please check my core dump backtrace - can we assume that this issue is connected to GC? anything else? Also, please hint what additional debug I can add in order to collect some more useful info at the time of segfault or before it. If we narrow it down or will be able to simplify reproducing, but won't have enough info for you to catch the bug - we can join a Zoom session (at your convenience) and investigate it on my system. |
|
Hi @cfbolz, Hope you're doing well. |
|
the backtrace looks like the problem from the other issue (#4899). The fix of that was now merged into the py3.9 and py3.10 branch. does your app still crash with a nightly build? I suspect yes. can you check what the backtrace is with a recent nightly? |
|
or, alternatively, turn your GC hooks off and show the traceback of the crash then. |
|
Hi @cfbolz, Conditions: PyPy 3.6 + your fix from #4899 + debug + without hooks. Unfortunately bt is not complete (not all symbols/functions were decoded), but it looks that GC can't find some element within nursery collection. coredumps_bt_pypy3.6_with_debug_with_fix_4899.zip |
|
Fresh cores from test with the following GC options: We will try with one more set of GC options but I don't expect any new info from it. coredump_small_GC_collections1.log |
|
Small update: we run PyPy with |
|
Hi Carl, What's our next move? |
|
Additional info We've tried to collect codedumps with The results - the picture did not change. E.G.: |
|
At this point we figured out that the problem is in |
|
Fresh backtrace. PyPy 3.6 with debug + patch4 (from Carl) + these options:
|
|
to summarize for myself:
|
|
I'd re-phrase the second statement: there's a chance that fix from #4925 partially solves our case, or we faced several issues and #4925 covers some of them (but not all). So, I'd re-phrase to: with fix from #4925 we still have crashes. Re last statement: we will run additional test, though... to be absolutely sure that it helps. Let me know if/when you need additional data and thanks for your help! |
|
Last portion of coredumps just for history. |
|
JFI: in our case, the problem disappears when we run pypy interpreter with the following environment variable: It's ok for us to live with such a workaround. |
mattip
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Dears,
We develop a SIP application and run it under PyPy.
After the update from 3.5 to 3.6 PyPy, the PyPy process started to fall with a "segmentation fault" once per 20-30 minutes.
Before the update, using pypy3.5, a process was relatively stable and could live for days or weeks without issues.
Our configuration of the GC:
Mentioned PyPy versions:
I also tried to build pypy3.10 or to use different versions of GCC but the picture is the same:
My attempts to analyze the core dumps (attaching example) failed.
Please advise the way to analyze the root of the issue.
If there are additional debug/hints/tricks that can help us to investigate this issue, please share them.
coredump_bt.txt
The text was updated successfully, but these errors were encountered: