Skip to content

Run pants under free threaded python#23477

Open
tobni wants to merge 9 commits into
pantsbuild:mainfrom
tobni:free-threaded-python
Open

Run pants under free threaded python#23477
tobni wants to merge 9 commits into
pantsbuild:mainfrom
tobni:free-threaded-python

Conversation

@tobni

@tobni tobni commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Switches Pants' own runtime to free-threaded CPython and makes the engine and
Pants' Python code thread-safe without the GIL.

This has been validated as running without error in my companies CI, and nets great speedups.

Most important callouts:

  • Keep a persistent per-thread PyThreadState in the executor. Without
    this, every Python::attach cycle tears down the thread state, and on 3.14t
    that abandons the thread's mimalloc heaps. Created per tokio thread, torn down in on_thread_stop (blocking
    threads are recycled, so the state must not leak per thread creation).
  • Makes run-scoped internals thread-safe: rule_visitor, exception_sink, logging, build_root, run_tracker.
  • PBS does not ship debug-symbol stripped binaries for free threaded python, which means pants-scie downloads a fatter artifact. Fooled by stale validation / docs. Stripped, free threaded PBS builds are available, but requires upstream changes in lift (Accept freethreaded install_only PBS flavors a-scie/lift#200) and pex (no PR but john likely only needs signs of life on our end as per Upgrade to Python 3.14 #22769 (comment).

I used #23437 as a launching point.

I have on speculation set this as shipping in 2.34.x, as I believe 2.33 is too far along the dev-cycles. But I also do not see any reason to delay further.

Blocks pantsbuild/scie-pants#541

@tobni tobni mentioned this pull request Jul 2, 2026
@tobni tobni force-pushed the free-threaded-python branch 2 times, most recently from c68208d to d3a9c73 Compare July 2, 2026 15:30
@cburroughs

Copy link
Copy Markdown
Contributor

(Some adjacent or related issues #22790 #22946 #15840 #21518 so I don't need to search for them again.)

@tobni As I understand it there isn't a stable ABI for free threaded until 3.15. Having dug in to do this work, do you have a sense of how much of a problem or risk that is (or is not)?

@tobni

tobni commented Jul 2, 2026

Copy link
Copy Markdown
Contributor Author

@cburroughs Risk is close to none afaik, it is mostly a pyo3 compat (meaning we couldnt build it) issue. If we are allowed to assume pants is ran as a scie we defined, the internal 3rd party lockfile does the rest. I think that is ok, building a pants dist and targeting a different python would be highly special usecase.

If a plugin uses native deps in rule code that could bite them iff there is no 3.14t wheels for them.

@tobni tobni force-pushed the free-threaded-python branch from d3a9c73 to 6a60274 Compare July 2, 2026 16:44
@tobni tobni force-pushed the free-threaded-python branch from 6a60274 to b03a612 Compare July 2, 2026 17:42
@sureshjoshi sureshjoshi requested review from benjyw, cburroughs, ndellosa95, sureshjoshi and tdyas and removed request for benjyw July 2, 2026 17:56
@sureshjoshi

Copy link
Copy Markdown
Member

Pretty substantial (and consequential) change - so I tacked on a bunch of reviewers

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants