Python 3.14 Free-Threading Support for duckdb-python

[paultiq]

Per the contributing guidelines, I'd like to propose implementing free threading support for duckdb-python... at least, as an experimental feature.

Python 3.13 released experimental support for free-threading. In Python 3.14 free-threaded Python is officially supported

I have a working fork that's functional, satisfies a set of race tests, that I'm (nearly) ready to open a PR for.

* I also understand if this is something the team will defer, implement internally or otherwise. I'm just hoping to make this a reality sooner rather than later!
** 3.14 is in RC2 with a release scheduled for Oct 7, 2025. Some dependencies haven't yet shipped 3.14 support yet: to make this work, I had to mark pandas, polars, pyarrow and gcovr as python_version < '3.14'.

Current Behavior

The GIL is automatically enabled when duckdb is imported in a free-threaded build of Python:

RuntimeWarning: Extension module was imported but does not declare support for free-threading. The GIL has been re-enabled.

Summary

This change allows free-threaded applications to import and use DuckDB.

Free-threading support does not impact DuckDB itself in any significant way: DuckDB already releases the GIL during query execution, and DuckDB's concurrency wouldn't be changed.

Proposed Behavior / Changes

1. Module Declaration: Pass the required pybind11 flag to denote free threading support: PYBIND11_MODULE(DUCKDB_PYTHON_LIB_NAME, m, py::mod_gil_not_used()) {

2. Module State Management: Move module state (helpfully marked // NOLINT: allow global in pyconnection.cpp) into a DuckDBPyModuleState object for multi-phase init

3. Critical Sections: Add py::scoped_critical_section around specific shared operations, such as connection creation/cleanup, import cache, etc. **. So far, the changes have been fairly localized, owing a lot to duckdb's thread safety.

4. Environmental: Some CI and configuration changes will be needed - especially in this early stage when not all dependencies have 3.14 support out yet.

5. Testing: An additional set of tests in tests/fast/freethreading

6. CI: Add a 3.14t wheel, and set the py::mod_gil_not_used() tag only for 3.14+ to prevent inadvertently adding 3.13t support.

[evertlammerts]

Hey @paultiq, thanks for starting the discussion. We've just started experimenting with / learning about free-threaded python in the past days. Moving to free-threading seems likely to be a relatively large project and we haven't figured out how we want to approach it just yet, so don't expect changes very soon. Nonetheless this is very much on our radar and your input is much appreciated. I'd love to have a browse through your PR when you have it.

[paultiq]

Thanks, I am cleaning up the work so it's actually reviewable. Will be another day (or so).

In the meantime, I'm opening a PR with just the CI changes needed for 3.14 and 3.14t builds (without free-threading), as I'd love to see nightly's post 1.4.0

[paultiq]

Just to update:

• Python 3.14 / 3.14t builds #50: 3.14/3.14t builds - not free-threading, just builds with the GIL. No Windows on 3.14t.
• Free threading 314t - Initial / Experimental Builds #76: Free Threading - experimental ...

More work needs to be done, including:

• Diagnosing Windows issues - there's a series of memory issues / stack corruption, but I'm not sure yet if it's a build issue or an actual issue. * edit - ended up being a build issue: /EHsc needed.
• Testing with Pandas, Polars and PyArrow's free threading builds, as they become available
• TSAN testing, and maybe ASAN