Skip to content

Commit 25e400d

Browse files
committed
Add trio-parent tests for _subint_forkserver
New pytest module `tests/spawn/test_subint_forkserver.py` drives the forkserver primitives from inside a real `trio.run()` in the parent — the runtime shape tractor will actually use when we wire up a `subint_forkserver` spawn backend proper. Complements the standalone no-trio-in-parent `ai/conc-anal/subint_fork_from_main_thread_smoketest.py`. Deats, - new test pkg `tests/spawn/` (+ empty `__init__.py`) - two tests, both `@pytest.mark.timeout(30, method='thread')` for the GIL-hostage safety reason doc'd in `ai/conc-anal/subint_sigint_starvation_issue.md`: - `test_fork_from_worker_thread_via_trio` — parent-side plumbing baseline. `trio.run()` off-loads forkserver prims via `trio.to_thread.run_sync()` + asserts the child reaps cleanly - `test_fork_and_run_trio_in_child` — end-to-end: forked child calls `run_subint_in_worker_thread()` with a bootstrap str that does `trio.run()` in a fresh subint - both tests wrap the inner `trio.run()` in a `dump_on_hang()` for post-mortem if the outer `pytest-timeout` fires - intentionally NOT using `--spawn-backend` — the tests drive the primitives directly rather than going through tractor's spawn-method registry (which the forkserver isn't plugged into yet) Also, rename `run_trio_in_subint()` → `run_subint_in_worker_thread()` for naming consistency with the sibling `fork_from_worker_thread()`. The action is really "host a subint on a worker thread", not specifically "run trio" — trio just happens to be the typical payload. Propagate the rename to the smoketest. Further, add a "TODO — cleanup gated on msgspec PEP 684 support" section to the `_subint_forkserver` module docstring: flags the dedicated-`threading.Thread` design as potentially-revisable once isolated-mode subints are viable in tractor. Cross-refs `msgspec#563` + `tractor#379` and points at an audit-plan conc-anal doc we'll add next. (this patch was generated in some part by [`claude-code`][claude-code-gh]) [claude-code-gh]: https://github.com/anthropics/claude-code
1 parent 82332fb commit 25e400d

4 files changed

Lines changed: 252 additions & 13 deletions

File tree

ai/conc-anal/subint_fork_from_main_thread_smoketest.py

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@
9191
# primitives have moved into tractor proper.)
9292
from tractor.spawn._subint_forkserver import (
9393
fork_from_worker_thread,
94-
run_trio_in_subint,
94+
run_subint_in_worker_thread,
9595
wait_child,
9696
)
9797

@@ -305,18 +305,18 @@ def _child_trio_in_subint() -> int:
305305
'''
306306
CHILD-side `child_target`: drive a trivial `trio.run()`
307307
inside a fresh legacy-config subint on a worker thread,
308-
using the `tractor.spawn._subint_forkserver.run_trio_in_subint`
308+
using the `tractor.spawn._subint_forkserver.run_subint_in_worker_thread`
309309
primitive. Returns 0 on success.
310310
311311
'''
312312
try:
313-
run_trio_in_subint(
313+
run_subint_in_worker_thread(
314314
_CHILD_TRIO_BOOTSTRAP,
315315
thread_name='child-subint-trio-thread',
316316
)
317317
except RuntimeError as err:
318318
print(
319-
f' CHILD: run_trio_in_subint timed out / thread '
319+
f' CHILD: run_subint_in_worker_thread timed out / thread '
320320
f'never returned: {err}',
321321
flush=True,
322322
)

tests/spawn/__init__.py

Whitespace-only changes.
Lines changed: 214 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
'''
2+
Integration exercises for the `tractor.spawn._subint_forkserver`
3+
primitives (`fork_from_worker_thread()` + `run_subint_in_worker_thread()`)
4+
driven from inside a real `trio.run()` in the parent process —
5+
the runtime shape tractor will need when we move toward wiring
6+
up a `subint_forkserver` spawn backend proper.
7+
8+
Background
9+
----------
10+
`ai/conc-anal/subint_fork_blocked_by_cpython_post_fork_issue.md`
11+
establishes that `os.fork()` from a non-main sub-interpreter
12+
aborts the child at the CPython level. The sibling
13+
`subint_fork_from_main_thread_smoketest.py` proves the escape
14+
hatch: fork from a main-interp *worker thread* (one that has
15+
never entered a subint) works, and the forked child can then
16+
host its own `trio.run()` inside a fresh subint.
17+
18+
Those smoke-test scenarios are standalone — no trio runtime
19+
in the *parent*. These tests exercise the same primitives
20+
from inside `trio.run()` in the parent, proving out the
21+
piece actually needed for a working spawn backend.
22+
23+
Gating
24+
------
25+
- py3.14+ (via `concurrent.interpreters` presence)
26+
- no backend restriction (these tests don't use
27+
`--spawn-backend` — they drive the forkserver primitives
28+
directly rather than going through tractor's spawn-method
29+
registry).
30+
31+
'''
32+
from __future__ import annotations
33+
from functools import partial
34+
import os
35+
36+
import pytest
37+
import trio
38+
39+
from tractor.devx import dump_on_hang
40+
41+
42+
# Gate: subint forkserver primitives require py3.14+. Check
43+
# the public stdlib wrapper's presence (added in 3.14) rather
44+
# than `_interpreters` directly — see
45+
# `tractor.spawn._subint` for why.
46+
pytest.importorskip('concurrent.interpreters')
47+
48+
from tractor.spawn._subint_forkserver import ( # noqa: E402
49+
fork_from_worker_thread,
50+
run_subint_in_worker_thread,
51+
wait_child,
52+
)
53+
54+
55+
# ----------------------------------------------------------------
56+
# child-side callables (passed via `child_target=` across fork)
57+
# ----------------------------------------------------------------
58+
59+
60+
_CHILD_TRIO_BOOTSTRAP: str = (
61+
'import trio\n'
62+
'async def _main():\n'
63+
' await trio.sleep(0.05)\n'
64+
' return 42\n'
65+
'result = trio.run(_main)\n'
66+
'assert result == 42, f"trio.run returned {result}"\n'
67+
)
68+
69+
70+
def _child_trio_in_subint() -> int:
71+
'''
72+
`child_target` for the trio-in-child scenario: drive a
73+
trivial `trio.run()` inside a fresh legacy-config subint
74+
on a worker thread.
75+
76+
Returns an exit code suitable for `os._exit()`:
77+
- 0: subint-hosted `trio.run()` succeeded
78+
- 3: driver thread hang (timeout inside `run_subint_in_worker_thread`)
79+
- 4: subint bootstrap raised some other exception
80+
81+
'''
82+
try:
83+
run_subint_in_worker_thread(
84+
_CHILD_TRIO_BOOTSTRAP,
85+
thread_name='child-subint-trio-thread',
86+
)
87+
except RuntimeError:
88+
# timeout / thread-never-returned
89+
return 3
90+
except BaseException:
91+
return 4
92+
return 0
93+
94+
95+
# ----------------------------------------------------------------
96+
# parent-side harnesses (run inside `trio.run()`)
97+
# ----------------------------------------------------------------
98+
99+
100+
async def run_fork_in_non_trio_thread(
101+
deadline: float,
102+
*,
103+
child_target=None,
104+
) -> int:
105+
'''
106+
From inside a parent `trio.run()`, off-load the
107+
forkserver primitive to a main-interp worker thread via
108+
`trio.to_thread.run_sync()` and return the forked child's
109+
pid.
110+
111+
Then `wait_child()` on that pid (also off-loaded so we
112+
don't block trio's event loop on `waitpid()`) and assert
113+
the child exited cleanly.
114+
115+
'''
116+
with trio.fail_after(deadline):
117+
# NOTE: `fork_from_worker_thread` internally spawns its
118+
# own dedicated `threading.Thread` (not from trio's
119+
# cache) and joins it before returning — so we can
120+
# safely off-load via `to_thread.run_sync` without
121+
# worrying about the trio-thread-cache recycling the
122+
# runner. Pass `abandon_on_cancel=False` for the
123+
# same "bounded + clean" rationale we use in
124+
# `_subint.subint_proc`.
125+
pid: int = await trio.to_thread.run_sync(
126+
partial(
127+
fork_from_worker_thread,
128+
child_target,
129+
thread_name='test-subint-forkserver',
130+
),
131+
abandon_on_cancel=False,
132+
)
133+
assert pid > 0
134+
135+
ok, status_str = await trio.to_thread.run_sync(
136+
partial(
137+
wait_child,
138+
pid,
139+
expect_exit_ok=True,
140+
),
141+
abandon_on_cancel=False,
142+
)
143+
assert ok, (
144+
f'forked child did not exit cleanly: '
145+
f'{status_str}'
146+
)
147+
return pid
148+
149+
150+
# ----------------------------------------------------------------
151+
# tests
152+
# ----------------------------------------------------------------
153+
154+
155+
# Bounded wall-clock via `pytest-timeout` (`method='thread'`)
156+
# for the usual GIL-hostage safety reason documented in the
157+
# sibling `test_subint_cancellation.py` / the class-A
158+
# `subint_sigint_starvation_issue.md`. Each test also has an
159+
# inner `trio.fail_after()` so assertion failures fire fast
160+
# under normal conditions.
161+
@pytest.mark.timeout(30, method='thread')
162+
def test_fork_from_worker_thread_via_trio() -> None:
163+
'''
164+
Baseline: inside `trio.run()`, call
165+
`fork_from_worker_thread()` via `trio.to_thread.run_sync()`,
166+
get a child pid back, reap the child cleanly.
167+
168+
No trio-in-child. If this regresses we know the parent-
169+
side trio↔worker-thread plumbing is broken independent
170+
of any child-side subint machinery.
171+
172+
'''
173+
deadline: float = 10.0
174+
with dump_on_hang(
175+
seconds=deadline,
176+
path='/tmp/subint_forkserver_baseline.dump',
177+
):
178+
pid: int = trio.run(
179+
partial(run_fork_in_non_trio_thread, deadline),
180+
)
181+
# parent-side sanity — we got a real pid back.
182+
assert isinstance(pid, int) and pid > 0
183+
# by now the child has been waited on; it shouldn't be
184+
# reap-able again.
185+
with pytest.raises((ChildProcessError, OSError)):
186+
os.waitpid(pid, os.WNOHANG)
187+
188+
189+
@pytest.mark.timeout(30, method='thread')
190+
def test_fork_and_run_trio_in_child() -> None:
191+
'''
192+
End-to-end: inside the parent's `trio.run()`, off-load
193+
`fork_from_worker_thread()` to a worker thread, have the
194+
forked child then create a fresh subint and run
195+
`trio.run()` inside it on yet another worker thread.
196+
197+
This is the full "forkserver + trio-in-subint-in-child"
198+
pattern the proposed `subint_forkserver` spawn backend
199+
would rest on.
200+
201+
'''
202+
deadline: float = 15.0
203+
with dump_on_hang(
204+
seconds=deadline,
205+
path='/tmp/subint_forkserver_trio_in_child.dump',
206+
):
207+
pid: int = trio.run(
208+
partial(
209+
run_fork_in_non_trio_thread,
210+
deadline,
211+
child_target=_child_trio_in_subint,
212+
),
213+
)
214+
assert isinstance(pid, int) and pid > 0

tractor/spawn/_subint_forkserver.py

Lines changed: 34 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,24 @@
5959
returned child pid into tractor's normal actor-nursery/IPC
6060
machinery.
6161
62+
TODO — cleanup gated on msgspec PEP 684 support
63+
-----------------------------------------------
64+
Both primitives below allocate a dedicated
65+
`threading.Thread` rather than using
66+
`trio.to_thread.run_sync()`. That's a cautious design
67+
rooted in three distinct-but-entangled issues (GIL
68+
starvation from legacy-config subints, tstate-recycling
69+
destroy race on trio cache threads, fork-from-main-tstate
70+
invariant). Some of those dissolve under PEP 684
71+
isolated-mode subints; one requires empirical re-testing
72+
to know.
73+
74+
Full analysis + audit plan for when we can revisit is in
75+
`ai/conc-anal/subint_forkserver_thread_constraints_on_pep684_issue.md`.
76+
Intent: file a follow-up GH issue linked to #379 once
77+
[jcrist/msgspec#563](https://github.com/jcrist/msgspec/issues/563)
78+
unblocks isolated-mode subints in tractor.
79+
6280
See also
6381
--------
6482
- `tractor.spawn._subint_fork` — the stub for the
@@ -268,22 +286,29 @@ def _worker() -> None:
268286
return pid
269287

270288

271-
def run_trio_in_subint(
289+
def run_subint_in_worker_thread(
272290
bootstrap: str,
273291
*,
274292
thread_name: str = 'subint-trio',
275293
join_timeout: float = 10.0,
276294

277295
) -> None:
278296
'''
279-
Helper for use inside a forked child: create a fresh
280-
legacy-config sub-interpreter and drive the given
281-
`bootstrap` code string through `_interpreters.exec()`
282-
on a dedicated worker thread.
283-
284-
Typical `bootstrap` content imports `trio`, defines an
285-
async entry, calls `trio.run()`. See
286-
`tractor.spawn._subint.subint_proc` for the matching
297+
Create a fresh legacy-config sub-interpreter and drive
298+
the given `bootstrap` code string through
299+
`_interpreters.exec()` on a dedicated worker thread.
300+
301+
Naming mirrors `fork_from_worker_thread()`:
302+
"<action>_in_worker_thread" — the action here is "run a
303+
subint", not "run trio" per se. Typical `bootstrap`
304+
content does import `trio` + call `trio.run()`, but
305+
nothing about this primitive requires trio; it's a
306+
generic "host a subint on a worker thread" helper.
307+
Intended mainly for use inside a fork-child (see
308+
`tractor.spawn._subint_forkserver` module docstring) but
309+
works anywhere.
310+
311+
See `tractor.spawn._subint.subint_proc` for the matching
287312
pattern tractor uses at the sub-actor level.
288313
289314
Destroys the subint after the thread joins.

0 commit comments

Comments
 (0)