New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Python] Segmentation fault when pyarrow is imported in exit handler #38626
Comments
Why do you want to do this...? |
Exit handlers in a complex codebase might incidentally import pyarrow especially via pandas. |
I can confirm the segfault, both with pyarrow 13.0 and main from a few days ago. The gdb bactrace:
|
General cpython issue about issues with threads in finalization: python/cpython#80657 (cc @pitrou) We do have some protection around acquiring the GIL here, but so that is not sufficient: arrow/python/pyarrow/src/arrow/python/common.h Lines 227 to 235 in e62ec62
|
@jorisvandenbossche Is it the full backtrace? Are there other threads running? |
Ok, here is the full backtrace:
|
…tdown Some C++ destructors may be called after the Python interpreter has ceased to exist.
…tdown Some C++ destructors may be called after the Python interpreter has ceased to exist.
…38637) ### Rationale for this change Some C++ destructors may be called after the Python interpreter has ceased to exist. If such a destructor tries to call back in the Python interpreter, for example by calling `Py_DECREF`, we get a crash. ### What changes are included in this PR? Protect `OwnedRef` and `OwneRefNoGIL` destructors against decref'ing a Python object after Python finalization. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: #38626 Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
…38637) ### Rationale for this change Some C++ destructors may be called after the Python interpreter has ceased to exist. If such a destructor tries to call back in the Python interpreter, for example by calling `Py_DECREF`, we get a crash. ### What changes are included in this PR? Protect `OwnedRef` and `OwneRefNoGIL` destructors against decref'ing a Python object after Python finalization. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: #38626 Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
…tdown (apache#38637) ### Rationale for this change Some C++ destructors may be called after the Python interpreter has ceased to exist. If such a destructor tries to call back in the Python interpreter, for example by calling `Py_DECREF`, we get a crash. ### What changes are included in this PR? Protect `OwnedRef` and `OwneRefNoGIL` destructors against decref'ing a Python object after Python finalization. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38626 Authored-by: Antoine Pitrou <antoine@python.org> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Describe the bug, including details regarding any error messages, version, and platform.
Versions
Component(s)
Python
The text was updated successfully, but these errors were encountered: