gh-146302: make Py_IsInitialized() thread-safe and reflect true init completion

[gpshead]

Summary

• Move the runtime->initialized = 1 store from before site.py import to the end of init_interp_main(), so Py_IsInitialized() only returns true after initialization has fully completed
• Access initialized and core_initialized through new inline accessors using acquire/release atomics, to also protect from data race undefined behavior
• PySys_AddAuditHook() now uses the accessor, so with the flag move it correctly skips audit hook invocation during all init phases (matching the documented "after runtime initialization" behavior) ... We could argue that running these earlier would be good even if the intent was never explicitly expressed, but that'd be its own issue.

Motivation

Py_IsInitialized() returned 1 while Py_InitializeEx() was still running — specifically, before site.py had been imported. See PyO3/pyo3#5900 where a second thread could acquire the GIL and start executing Python with an incomplete sys.path because site.py hadn't finished.

The flag was also a plain int with no atomic operations, making concurrent reads a C-standard data race, though unlikely to manifest.

Regression test:

The added test properly fails on main with ERROR: Py_IsInitialized() was true during site import.

• Issue: Py_IsInitialized() can return 1 before Py_InitializeEx() has completed. #146302

Fixes #146302

---

📚 Documentation preview 📚: https://cpython-previews--146303.org.readthedocs.build/

[gpshead]

this is technically a bug fix, we don't always call these out in the main docs. i'm happy dropping this, but it feels worthwhile to mention for embedders likely to use this API. and if we choose to backport this as a bugfix it'll include proper 3.14.x and 3.13.x attribution to indicate when people need to work around it.

[picnixz]

It is fine to add versionchanged when an important behavior needed to be pointed out in a bugfix release so I don't mind keeping this here.

[picnixz]

However considering what it changed I think it is best to keep it in 3.15 only.

[gpshead]

Agreed, I'm fine not backporting this. It's rare enough and PyO3 already has a workaround.

[gpshead]

Moving this down after site and the rest of the machinery is the part I'm most interested in seeing tested. Intuitively I would not expect any code site may import to ever care about Py_IsInitialized() itself as extension module code "should" never have a reason to use that API as it shouldn't be called outside of an interpreter.

But this is the one thing in this change that might be an observable behavior change. Even if we don't agree with someone relying on the existing behavior.

[picnixz]

OOC there are lots of boolean flags that are still bare ints. Are they all protected against races or?

[gpshead]

OOC there are lots of boolean flags that are still bare ints. Are they all protected against races or?

probably not technically... I doubt the data race potential honestly exposes itself much in this area in practice as filling these in happens so much sooner than most anything that could be consuming them that all memory writes long since land. Py_IsInitialized() being a reader intended for concurrent access feels like an uncommon case. But I didn't try to analyze the others.

asking Opus 4.6 to analyze their use for races (my summary): _initialized, preinitializing, and preinitialized are all fine as used. They're serial during single threaded early internal init phases. Py_FatalError could wind up reading them for informational purposes but accuracy isn't important in that scenario.

[which makes sense to me]

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @gpshead for commit abae231 🤖

Results will be shown at:

https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F146303%2Fmerge

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

[davidhewitt]

Thanks for doing this for PyO3's sake. Does thread-safe here mean that PyO3 could in principle freely call Py_Initialize from multiple threads and they will all block until whichever thread entered the actual initialization pipeline fully completes? I see atomic load/stores but no mutex hence the question.

(I appreciate what PyO3 is doing is slightly weird, because there's no way to reliably know which one of those threads becomes the "main thread". It's good enough for Rust test suites, at least!)

[davidhewitt]

Sorry, I see I misread the title - this is about Py_IsInitialized being thread-safe; this won't make any difference to calling Py_Initialize and PyO3 will presumably continue to need to have a strategy to synchronize that call.

[gpshead]

Does thread-safe here mean that PyO3 could in principle freely call this from multiple threads and they will all block until whichever thread entered the actual initialization pipeline fully completes?

No, CPython does not have a block until initialization completes API. That might make sense as a feature request on its own but I'd wonder what the practical motivation for it was and why we'd want people to do it.

[davidhewitt]

I think it's fine to leave that synchronization downstream in PyO3 rather than complicate the C API. I think in most embedding contexts you want to be deliberate about interpreter lifecycle; what PyO3 does with the concurrent initialization is a quirk of Rust test suites launching multiple threads with no particular need for any proper finalization.

[gpshead]

Agreed. We currently have a variety of C API surfaces (like Initialize and IsInterpreter) that are easier when embedding processes take a wide "hey i'm going to have a Python interpreter, let me go ahead and set that up once up front instead of having everything that wants to use it try to do that at time of first use" approach. That is not really practical for all situations. Your Rust example of using one from some test cases is a good example.

There are others, but they might expand into a larger C API and one interpreter vs multiple interpreters vs subinterpreters question. Imagine two independent libraries linked into a process unaware of each-other both wanting to use an embedded Python interpreter for internal use. As independent code, they have no good way to coordinate in a multithreaded context where they might both be trying at once. Which still ignores the question of if they need isolated interpreters or not. All of that could turn into a discuss.python.org thread...