Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

trio segfaults on import when using CPython 3.12.0b4 on Windows #2714

Closed
joerick opened this issue Jul 23, 2023 · 9 comments
Closed

trio segfaults on import when using CPython 3.12.0b4 on Windows #2714

joerick opened this issue Jul 23, 2023 · 9 comments

Comments

@joerick
Copy link

joerick commented Jul 23, 2023

The test suite for pyinstrument picks up trio, so I noticed this error.

In short, trio segfaults on import on this platform:

$ python
Python 3.12.0b4 (tags/v3.12.0b4:97a6a41, Jul 11 2023, 13:49:15) [MSC v.1935 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import trio

The process just crashed. Running with faulthandler,

$ python -X faulthandler
Python 3.12.0b4 (tags/v3.12.0b4:97a6a41, Jul 11 2023, 13:49:15) [MSC v.1935 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import trio
Windows fatal exception: access violation

Current thread 0x00001d38 (most recent call first):
Windows fatal exception: access violation

(env312) PS C:\Users\joeri\Documents\pyinstrument> 

Frustratingly, the faulthandler crashes when trying to print the stack trace, so it's not possible to see where it was when it crashed.

I'm using the Python.org distribution of CPython 3.12.0b4, on an AMD64 machine.

@joerick
Copy link
Author

joerick commented Jul 23, 2023

Just realised that trio is pure-python, so this must be something upstream of you.

Did a little more digging, it appears to be when cffi loads a library.

Tracked it down to this line:

trio/trio/_util.py

Lines 53 to 57 in a9a9170

import cffi
_ffi = cffi.FFI()
_ffi.cdef("int raise(int);")
_lib = _ffi.dlopen("api-ms-win-crt-runtime-l1-1-0.dll")

Specifically, _ffi.dlopen("api-ms-win-crt-runtime-l1-1-0.dll") seems to be it.

I also tried with cffi installed from the development branch (pip install https://foss.heptapod.net/pypy/cffi/-/archive/branch/default/cffi-branch-default.zip), same result.

my debug script

I'll keep this here in case it's useful later too.

import sys      

depth = 0

def trace(frame, event, what):
    if event not in ('call', 'return'): return
    direction =  1 if event == 'call' else -1
    global depth
    depth += direction

    path = frame.f_code.co_filename
    if '\\cffi\\' in path: return
    if '\\pycparser\\' in path: return
    if '<frozen' in path: return
    if 'Python312\\Lib\\' in path: return
    if '_distutils_hack' in path: return
    
    frame_desc = f"{frame.f_code.co_name} {frame.f_code.co_filename}:{frame.f_code.co_firstlineno}"
    inout = " -->" if event == 'call' else "+ <--"
    print(f"{'+'*depth}{inout} {frame_desc}")

sys.setprofile(trace)

import trio

This produced huge amounts of output, ending in-

...
++++++++++++++++++++ <-- <genexpr> C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\_make.py:997
++++++++++++++++++++ --> <genexpr> C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\_make.py:997
++++++++++++++++++++ <-- <genexpr> C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\_make.py:997
++++++++++++++++++++ --> <genexpr> C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\_make.py:997
++++++++++++++++++++ <-- <genexpr> C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\_make.py:997
+++++++++++++++++++ <-- add_match_args C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\_make.py:996
+++++++++++++++++++ --> build_class C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\_make.py:731
++++++++++++++++++++ --> _create_slots_class C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\_make.py:795
++++++++++++++++++++ <-- _create_slots_class C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\_make.py:795
+++++++++++++++++++ <-- build_class C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\_make.py:731
++++++++++++++++++ <-- wrap C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\_make.py:1489
+++++++++++++++++ <-- <module> C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\_version_info.py:1
++++++++++++ --> AttrsInstance C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\__init__.py:34
++++++++++++ <-- AttrsInstance C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\__init__.py:34
++++++++++++ --> _make_getattr C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\__init__.py:74
++++++++++++ <-- _make_getattr C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\__init__.py:74
+++++++++++ <-- <module> C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\attr\__init__.py:1
+++++++++++ --> <module> C:\Users\joeri\Documents\pyinstrument\env312\Lib\site-packages\trio\_util.py:1
++++++++++++++++++++++ --> <module> <string>:1
++++++++++++++++++++++ <-- <module> <string>:1
+++++++++++++++++++++ --> <module> <string>:1
+++++++++++++++++++++ <-- <module> <string>:1
Windows fatal exception: access violation

Current thread 0x00004ba0 (most recent call first):
  <no Python frame>

I saw that the last call was to the trio\_util.py module, which led me to that line. I had to hide a few libraries as shown in the code to reduce the noise.

@Fuyukai
Copy link
Member

Fuyukai commented Jul 23, 2023

This looks like a cffi issue, not ours.

@A5rocks
Copy link
Contributor

A5rocks commented Jul 29, 2023

Hello! I'm closing this since this is an upstream issue. Here's some stuff:

  1. I can also reproduce this.
  2. Here's what windbg has to say about this:
2:009> gn
ModLoad: 00007ffe`19440000 00007ffe`194ba000   C:\WINDOWS\System32\bcryptprimitives.dll
ModLoad: 00000225`af760000 00000225`af76f000   C:\Users\A5rocks\Documents\python-3.12.0b4\python3.DLL
ModLoad: 00000225`af760000 00000225`af76f000   C:\Users\A5rocks\Documents\python-3.12.0b4\python3.DLL
ModLoad: 00007ffe`09690000 00007ffe`0969a000   C:\Users\A5rocks\Documents\python-3.12.0b4\DLLs\_wmi.pyd
ModLoad: 00007ffe`1acc0000 00007ffe`1ad97000   C:\WINDOWS\System32\OLEAUT32.dll
ModLoad: 00007ffe`193a0000 00007ffe`1943a000   C:\WINDOWS\System32\msvcp_win.dll
ModLoad: 00007ffe`13af0000 00007ffe`13bf0000   C:\WINDOWS\SYSTEM32\PROPSYS.dll
ModLoad: 00007ffe`1af00000 00007ffe`1b289000   C:\WINDOWS\System32\combase.dll
ModLoad: 00000225`b0040000 00000225`b03c9000   C:\WINDOWS\System32\combase.dll
ModLoad: 00007ffd`f34c0000 00007ffd`f34f2000   C:\Users\A5rocks\Documents\trio\py-312-venv\Lib\site-packages\_cffi_backend.cp312-win_amd64.pyd
ModLoad: 00007ffe`197f0000 00007ffe`1999a000   C:\WINDOWS\System32\USER32.dll
ModLoad: 00007ffe`196a0000 00007ffe`196c6000   C:\WINDOWS\System32\win32u.dll
ModLoad: 00007ffe`1ac20000 00007ffe`1ac49000   C:\WINDOWS\System32\GDI32.dll
ModLoad: 00007ffe`18db0000 00007ffe`18ec9000   C:\WINDOWS\System32\gdi32full.dll
ModLoad: 00007ffe`19c30000 00007ffe`19c61000   C:\WINDOWS\System32\IMM32.DLL
(187c.664c): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
python312!unicode_copy_as_widechar+0x49:
00007ffd`9c59530d 66418901        mov     word ptr [r9],ax ds:00000055`d69f0000=????
2:009> tb
(187c.664c): Security check failure or stack buffer overrun - code c0000409 (!!! second chance !!!)
Subcode: 0x2 FAST_FAIL_STACK_COOKIE_CHECK_FAILURE 
_cffi_backend_cp312_win_amd64!PyInit__cffi_backend+0x861:
00007ffd`f34d9fa1 cd29            int     29h
  1. Here's my upstream report: https://foss.heptapod.net/pypy/cffi/-/issues/565#note_304559.

Edit: here's a more complete stack trace:

image

@A5rocks A5rocks closed this as completed Jul 29, 2023
@A5rocks
Copy link
Contributor

A5rocks commented Jul 30, 2023

Err, actually, the comment directly above the line in question does say we could replace with ctypes. That might be worth it for future stability...

@njsmith
Copy link
Member

njsmith commented Jul 30, 2023 via email

@bastimeyer
Copy link

Have you checked whether the issue still occurs when building and installing cffi from the latest commit of their branch/default branch?

They currently can't or don't want to publish a new release because of some py312 issues.
https://groups.google.com/g/python-cffi/c/WBDbhNT0gx8/m/BIn5T-1cAQAJ

Some issues were fixed recently, and one of the open issues is this one here, referring to the comment made by @A5rocks.
https://foss.heptapod.net/pypy/cffi/-/issues/574


I just built cffi in my Windows VM and ran the trio tests. I had two test failures, but those were unrelated. When installing cffi 1.15.1 from PyPI however, the Python 3.12 interpreter crashed.

Please have a look at this and report back to the cffi issue tracker if those crashes are indeed gone.


/c/Python312/python -m venv ~/venv/trio
source ~/venv/trio/Script/activate
python -m pip install -U pip build wheel

git clone https://github.com/python-trio/trio
hg clone https://foss.heptapod.net/pypy/cffi

cd cffi
python -m build --outdir dist --wheel

cd ../trio
python -m pip install -U -r test-requirements.txt
python -m pip install -U --force-reinstall ../cffi/dist/cffi-*.whl
python -m pytest -r a -p trio._tests.pytest_plugin

@A5rocks
Copy link
Contributor

A5rocks commented Sep 22, 2023

I'll check in a while but I suspect you're right -- I don't remember installing from source last time (heck, I don't think I even have mercurial on this computer!!). You might want to comment on the cffi issue yourself since I may forget to do this for a while!

@bastimeyer
Copy link

I don't have an account there and this banner is shown:

Due to a massive storm of spam, registration on this instance now requires explicit administrator approval. Sorry for the added friction, we're also looking into automatic filtering options.

Please just post a quick comment since you're already registered. Thank you.

@A5rocks
Copy link
Contributor

A5rocks commented Sep 22, 2023

Alright, done!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants