runtime/interp/thread state refactoring leads to segmentation fault

[pquentin]

BPO	46320
Nosy	@ericsnowcurrently, @pquentin, @SethMichaelLarson

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericsnowcurrently'
closed_at = <Date 2022-01-12.05:38:40.919>
created_at = <Date 2022-01-09.20:48:32.485>
labels = ['interpreter-core', 'invalid', 'type-crash', '3.11']
title = 'runtime/interp/thread state refactoring leads to segmentation fault'
updated_at = <Date 2022-01-12.15:29:19.814>
user = 'https://github.com/pquentin'

bugs.python.org fields:

activity = <Date 2022-01-12.15:29:19.814>
actor = 'eric.snow'
assignee = 'eric.snow'
closed = True
closed_date = <Date 2022-01-12.05:38:40.919>
closer = 'Quentin.Pradet'
components = ['Interpreter Core']
creation = <Date 2022-01-09.20:48:32.485>
creator = 'Quentin.Pradet'
dependencies = []
files = []
hgrepos = []
issue_num = 46320
keywords = []
message_count = 12.0
messages = ['410166', '410167', '410227', '410231', '410252', '410253', '410254', '410348', '410362', '410370', '410384', '410402']
nosy_count = 3.0
nosy_names = ['eric.snow', 'Quentin.Pradet', 'SethMichaelLarson']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue46320'
versions = ['Python 3.11']

[pquentin]

Since 32a6724 which was introduced through https://bugs.python.org/issue46008 and shipped as part of alpha 3, the urllib3 test suite reliably crashes on Fedora 35 (ie. gives a segfault).

I've narrowed the reproducer code down to https://github.com/pquentin/urllib3/blob/segfault/test/test_reproduce.py, but that still requires cffi, pyOpenSSL and Tornado.

The segfault either happens in the ssl module or the selectors module:

================================================= test session starts =================================================
platform linux -- Python 3.11.0a2+, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/q/pub/urllib3, configfile: setup.cfg
collected 1 item                                                                                                      

test/test_reproduce.py Fatal Python error: Segmentation fault

Thread 0x00007fee9a250640 (most recent call first):
  File "/home/q/pub/install/lib/python3.11/ssl.py", line 1346 in do_handshake
  File "/home/q/pub/urllib3/venv/lib/python3.11/site-packages/tornado/iostream.py", line 1391 in _do_ssl_handshake
  File "/home/q/pub/urllib3/venv/lib/python3.11/site-packages/tornado/iostream.py", line 1478 in _handle_read
  File "/home/q/pub/urllib3/venv/lib/python3.11/site-packages/tornado/iostream.py", line 696 in _handle_events
  File "/home/q/pub/urllib3/venv/lib/python3.11/site-packages/tornado/platform/asyncio.py", line 189 in _handle_events
  File "/home/q/pub/install/lib/python3.11/asyncio/events.py", line 80 in _run
  File "/home/q/pub/install/lib/python3.11/asyncio/base_events.py", line 1858 in _run_once
  File "/home/q/pub/install/lib/python3.11/asyncio/base_events.py", line 591 in run_forever
  File "/home/q/pub/urllib3/venv/lib/python3.11/site-packages/tornado/platform/asyncio.py", line 199 in start
  File "/home/q/pub/install/lib/python3.11/threading.py", line 968 in run
  File "/home/q/pub/install/lib/python3.11/threading.py", line 1031 in _bootstrap_inner
  File "/home/q/pub/install/lib/python3.11/threading.py", line 988 in _bootstrap

Extension modules: tornado.speedups, _brotli, _cffi_backend (total: 3)
zsh: segmentation fault (core dumped)  pytest

================================================= test session starts =================================================
platform linux -- Python 3.11.0a2+, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/q/pub/urllib3, configfile: setup.cfg
collected 1 item                                                                                                      

test/test_reproduce.py Fatal Python error: Segmentation fault

Thread 0x00007fee9a250640 (most recent call first):
  File "/home/q/pub/install/lib/python3.11/ssl.py", line 1346 in do_handshake
  File "/home/q/pub/urllib3/venv/lib/python3.11/site-packages/tornado/iostream.py", line 1391 in _do_ssl_handshake
  File "/home/q/pub/urllib3/venv/lib/python3.11/site-packages/tornado/iostream.py", line 1478 in _handle_read
  File "/home/q/pub/urllib3/venv/lib/python3.11/site-packages/tornado/iostream.py", line 696 in _handle_events
  File "/home/q/pub/urllib3/venv/lib/python3.11/site-packages/tornado/platform/asyncio.py", line 189 in _handle_events
  File "/home/q/pub/install/lib/python3.11/asyncio/events.py", line 80 in _run
  File "/home/q/pub/install/lib/python3.11/asyncio/base_events.py", line 1858 in _run_once
  File "/home/q/pub/install/lib/python3.11/asyncio/base_events.py", line 591 in run_forever
  File "/home/q/pub/urllib3/venv/lib/python3.11/site-packages/tornado/platform/asyncio.py", line 199 in start
  File "/home/q/pub/install/lib/python3.11/threading.py", line 968 in run
  File "/home/q/pub/install/lib/python3.11/threading.py", line 1031 in _bootstrap_inner
  File "/home/q/pub/install/lib/python3.11/threading.py", line 988 in _bootstrap

Extension modules: tornado.speedups, _brotli, _cffi_backend (total: 3)
zsh: segmentation fault (core dumped)  pytest

I can work on a better reproducer, but thought this was already interesting as #29977 states there should have been zero change in behavior.

[pquentin]

Sorry, I pasted the same crash twice. Here's the segmentation fault in selectors:

============================================================================================================= test session starts =============================================================================================================
platform linux -- Python 3.11.0a2+, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /home/q/pub/urllib3, configfile: setup.cfg
collected 1 item                                                                                                                                                                                                                              

test/test_reproduce.py Fatal Python error: Segmentation fault

Thread 0x00007f9683378640 (most recent call first):
  File "/home/q/pub/install/lib/python3.11/selectors.py", line 416 in select
  File "/home/q/pub/install/lib/python3.11/asyncio/base_events.py", line 1822 in _run_once
  File "/home/q/pub/install/lib/python3.11/asyncio/base_events.py", line 591 in run_forever
  File "/home/q/pub/urllib3/venv/lib/python3.11/site-packages/tornado/platform/asyncio.py", line 199 in start
  File "/home/q/pub/install/lib/python3.11/threading.py", line 968 in run
  File "/home/q/pub/install/lib/python3.11/threading.py", line 1031 in _bootstrap_inner
  File "/home/q/pub/install/lib/python3.11/threading.py", line 988 in _bootstrap

Extension modules: tornado.speedups, _brotli, _cffi_backend (total: 3)
zsh: segmentation fault (core dumped)  pytest

(Those results are produced with 32a6724)

[ericsnowcurrently]

I'll look into this today. To reproduce, I should run the urllib3 test suite?

[pquentin]

Yes, exactly. pip install nox && nox -Rs test-3.11. My reproducer is still too big to be useful, and I failed to reproduce this on Docker. If you can't reproduce, I'll work on the reproducer more. Thanks!

[ericsnowcurrently]

I must be missing something. Here is what I did:

$ cd cpython
$ git checkout main
$ make -j8
$ ./python -v venv ../venv-urllib3
$ cd ..
$ git clone https://github.com/urllib3/urllib3
$ cd urllib3
$ ../venv-urllib3/bin/python3 -m pip install nox
$ ../venv-urllib3/bin/nox -Rs test-3.11
nox > Running session test-3.11
nox > Session test-3.11 skipped: Python interpreter 3.11 not found.

I am not familiar with nox so I'm not sure how to trouble-shoot this.

[ericsnowcurrently]

It looks like the urllib3 CI is passing for 3.11.0a3: https://github.com/urllib3/urllib3/runs/4762856431 (Ubuntu 3.11-dev test-3.11).

Is there a urllib3 issue number you could point me at?

[pquentin]

We haven't opened an issue for this, but discussed it in Discord, sorry. It also does not always crash on GitHub Actions. Here's an example of a crash: https://github.com/urllib3/urllib3/runs/4740730329?check_suite_focus=true

If nox can't find your Python version and nox --no-venv -Rs test-3.11 does not run your version of Python 3.11, you can always execute those commands directly:

pip install -r dev-requirements.txt
pip install ".[socks,secure,brotli]"
pytest

If that does not crash on your system, I'd appreciate if you could tell me how I could help get this fixed besides simplifying the reproducer. Thanks.

[ericsnowcurrently]

pip install -r dev-requirements.txt
pip install ".[socks,secure,brotli]"
pytest

I was able to reproduce the crash with these steps. Thanks!

[ericsnowcurrently]

FYI, I get the crash with just:

pytest test/contrib/test_pyopenssl.py::TestHTTPS::test_verify_none_and_bad_fingerprint

[ericsnowcurrently]

It looks like PyThreadState.async_exc is getting set to 0x01 somewhere. There isn't any code like that in 32a6724. However, the struct layout of PyThreadState did change slightly in that commit.

The ABI is generally not stable until the first beta (and sometimes a bit after that). Could it be that the wheel for one or more the dependencies was built against an earlier 3.11 release (with the previous PyThreadState layout)?

If I move fields around in PyThreadState just right, I can no longer reproduce the problem.

[pquentin]

Oh my god. You're right, I had a cffi wheel compiled for Python 3.11 before that commit. But the wheel was not coming from PyPI, it was coming from pip's own cache! And we recently enabled pip's cache in GitHub Actions too.

So the wheel compiled locally for alpha 2 got reused for alpha 3, and 💥.

Sorry for the noise but thanks a lot for your help, this was really baffling. Closing!