Rare CPython 3.10 crashes on Windows

[lazka]

Bug report

This is a bit of a long shot, as I'm out of ideas, please stay with me :)

The MSYS2 project has been plagued by random Python crashes over the last year in CI (GitHub and GitLab), in combination with meson and I finally got some time to reproduce/bisect it, still only in CI:

https://github.com/lazka/python-crash-test (see the workflow file there for the setup)

This sets the error mode to SEM_NOGPFAULTERRORBOX and runs ninja which repeatedly calls python, until python crashes.

Python crashes with STATUS_ACCESS_VIOLATION, only on CI, and only if just-in-time debugging is disabled via SEM_NOGPFAULTERRORBOX, which makes it quite hard to pin down/debug.

I've now taken the time to git bisect it via GHA CI and come to the following result:

• The first commit to introduce the crashes is 4101018 (this was during v3.10.0a3...v3.10.0a4)
• The first commit to make the crashes go away again is ed57b36 (this was during v3.11.0a3...v3.11.0a4)

I'd welcome any ideas on why those commits could have an effect and what I could try to further pin this down.

My initial goal was to revert or backport commits after the bisect, but those commits have so many conflicts and touch so many things that I gave up.

Your environment

• CPython versions tested on: all versions from 3.10.0a1 up to 3.11.3
• Operating system and architecture: on all of Windows Server 2019/2022 (via GHA) Windows Server 2016 (via GitLab CI)

[lazka]

I guess at this point it's unlikely that we'll ever find the cause, so let's close this. However, it is a bit worrying that there are random crashes that have been fixed by accident, and there is no guarantee that they won't come back.

If anyone ends up here because CPython crashes on Windows when using meson, or similar workloads invoking other programs from Python, and/or if SEM_NOGPFAULTERRORBOX is involved, please leave a comment here. If this crash ever comes back it might be good to have this information.

thanks!