Rework execnet onto a trio-native core (sync facade, uv-provisioned workers, execnet.trio)#422
Draft
RonnyPfannschmidt wants to merge 34 commits into
Draft
Conversation
Move coordinator and worker framed protocol IO into dedicated Trio host threads for local popen + import bootstrap, while keeping the sync Channel/Gateway API and WorkerPool remote_exec. Adds a trio dependency; disable with EXECNET_TRIO_HOST=0. Co-authored-by: Cursor AI <ai@cursor.sh> Co-authored-by: Cursor Grok 4.5 <grok@x.ai>
Replace WorkerPool on the Trio popen worker with TrioWorkerExec: thread-model tasks run via trio.to_thread, main_thread_only hands off to the process main thread. Also wake the protocol writer with a Trio Event instead of blocking to_thread queue.get. Co-authored-by: Cursor AI <ai@cursor.sh> Co-authored-by: Cursor Grok 4.5 <grok@x.ai>
Remove the EventletExecModel and GeventExecModel backends and their get_execmodel branches, leaving only the stdlib thread and main_thread_only models. Narrow the test execmodel fixture, drop the gevent test dependency and the eventlet/gevent mypy overrides, and scrub the docs of the removed backends. This is phase 1 of moving execnet onto Trio: the ExecModel surface is kept intact for now and collapsed further once Trio drives IO on both sides. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Launch the Trio popen worker as `python -m execnet._trio_worker <id> <execmodel> <version>` instead of sending bootstrap source over the wire. The worker imports the installed execnet + trio, writes the `1` handshake after adopting its stdio fds, and serves; the coordinator just waits for the handshake. A rough major/minor version check warns on a real execnet mismatch between coordinator and worker while tolerating patch-level drift. Drops the import-bootstrap source send and the importdir/PYTHONPATH plumbing (no more uninstalled running). Foreign-python and remote transports stay on the legacy path pending the uv-provisioned bootstrap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route `popen//python=<interpreter>` through the Trio path. If the target interpreter already imports execnet + trio, launch the worker module directly on it (preserving sys.executable); otherwise provision an ephemeral environment with uv and run the worker there. New _provision.py builds the launch: - coordinator_requirement(): `execnet==<ver>` for a released coordinator, else a wheel built from the editable source (located via PEP 610 direct_url.json) and cached keyed by version. The wheel path is read from uv's "Successfully built" output rather than reconstructed, since the build-time version can differ from the import-time one. - uv_run_argv(): `uv run --no-project [--python X] --with <req> python -u -m execnet._trio_worker ...`; trio comes in transitively. - target_has_execnet(): cached probe deciding direct vs provisioned. Falls back to the legacy source-copy path when the target lacks execnet and uv is unavailable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an in-process asyncssh server (asyncio-loop thread) with binary-safe command passthrough so the ssh transport can be tested against the system ssh client without an external host. Committed, intentionally-insecure ed25519 test keys back it (see testing/sshkeys/README.md); the fixture copies the client key to a 0600 temp file since git does not preserve it. asyncssh joins the testing extra. The initial test exercises the existing legacy ssh path, validating the harness before the Trio ssh transport. Also make mypy green at the source instead of casting at call sites: TrioHost.call is now generic (Callable[..., Awaitable[T]] -> T) and makegateway_popen_trio returns Gateway. Drop the stale types-gevent from the mypy hook. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route ssh gateways through the Trio host: `ssh -C [-F cfg] <host> '<uv worker command>'` spawns the uv-provisioned worker module on the remote, then the coordinator does the usual b"1" handshake and attaches a Trio session. HostNotFound is raised when ssh exits 255. The popen and ssh factories now share _open_trio_gateway (spawn + handshake + attach); ssh_trio_args builds the shell-quoted remote worker command. test_ssh_roundtrip is parametrized over the trio and legacy paths against the in-process asyncssh server. test_sshconfig_config_parsing (white-box over legacy Popen2IOMaster) is pinned to EXECNET_TRIO_HOST=0, with a new ssh_trio_args test covering -F on the Trio path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A dev coordinator's wheel is not on the remote filesystem, so the ssh remote command becomes a POSIX-sh prelude that reads the wheel bytes from stdin (`head -c N` into a temp dir) and execs uv against it. The coordinator streams those bytes as a preamble before the Message protocol; _open_trio_gateway grew a `preamble` argument. Released coordinators still use `uv run --with execnet==<ver>` with no shipping. Also collapse the worker CLI contract: id, execmodel and coordinator version now travel as a single JSON argument (_provision.worker_cli_arg) consumed by _trio_worker._main, instead of scattered positional args built in three launchers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expose the socket server as a `[project.scripts]` entry point so it can be started directly, e.g. provisioned anywhere with `uvx --from execnet execnet-socketserver :8888`. Refactor the __main__ block into a main() with an argparse CLI (hostport + --once), and thread execmodel explicitly through startserver/exec_from_one_connection instead of relying on a module global (which main()'s local scope broke). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the direct `socket=host:port` transport onto the Trio host: - The socketserver becomes a Trio TCP listener that spawns a `python -m execnet._trio_worker --socket-fd N` subprocess per connection (passing the accepted socket by fd) instead of exec'ing sent source inline. - The worker gains a socket serve mode: it adopts an inherited socket fd into a Trio SocketStream and serves the Message protocol over it. The worker CLI now always carries its config as args, so a future popen socketpair can reuse the same path instead of hijacking stdio. - The coordinator connects a Trio TCP stream, waits for the b"1" handshake, and attaches a Trio session (no local process). A failed connect raises HostNotFound. The worker serve setup is refactored into shared _build_worker_gateway / _run_worker helpers. `installvia` still uses the legacy path for now. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Since execnet is now always installed on both sides, replace the remote_exec source-shipping used by `socket//installvia=<gw>` with a first-class protocol message. The coordinator sends GATEWAY_START_SOCKET (with the bind host) on a request channel; the via gateway's Trio host binds an ephemeral port, replies with the (host, port) on that channel, and serves the one connection by spawning a worker subprocess. The coordinator then connects to it over the Trio socket path. This makes the inline-exec socketserver dead, so drop it: the `execnet-socketserver` script is now only the Trio server + CLI, and the Windows service wrapper launches that. The legacy `__channelexec__` mode, `bind_and_listen`, `startserver`, and the inline `exec` are gone. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With direct socket and installvia both on the Trio path, the legacy socket coordinator is dead. Delete gateway_socket.py (SocketIO, create_io, start_via), drop bootstrap_socket and the socket branch from gateway_bootstrap.bootstrap, and remove the now-unreachable `elif spec.socket:` arm from Group.makegateway. All socket gateways now go through _trio_host.makegateway_socket_trio. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the common `popen//via=<master>` proxy transport onto the Trio host with a new protocol message instead of remote_exec'ing gateway_io source. The coordinator sends GATEWAY_START_POPEN (with the sub worker config) on a request channel; the master's Trio host spawns a `python -m execnet._trio_worker` sub-worker and relays its Message protocol raw over that channel (stdin<-channel, stdout->channel). The coordinator wraps the channel as ChannelByteIO and runs a normal Trio session over it. ChannelByteIO is marked an interim hack: it tunnels sub frames as CHANNEL_DATA (double-framing) and should become a proper relayed transport. ssh/foreign-python via sub-gateways still use the legacy path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Generalize the via relay message from popen-only to a spawn-request dict carrying the sub-spec essentials plus provisioning material: a released coordinator sends a pip requirement, a dev coordinator ships its wheel for the master to materialize into the local wheel cache. The master resolves the launch locally (direct module, uv-provisioned python=, or ssh with the wheel streamed as stdin preamble), so ssh= and python= sub specs now run on the Trio path. Also route ssh=...//via=... through the via path instead of opening a direct ssh connection, and close the request channel with an error on spawn/relay failure instead of crashing the host nursery. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
vagrant_ssh= now launches the uv-provisioned worker through `vagrant ssh <machine> -- -C <command>` (mirroring the ssh argv), both as a direct gateway and as a via sub-gateway through GATEWAY_START_SUB. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Trio host is now the only IO path. Delete gateway_io.py (ProxyIO, Popen2IOMaster, source bootstrap lines) and gateway_bootstrap.py, the Popen2IO sync pipe IO, the thread receiver, and WorkerGateway.serve(); makegateway dispatches directly on the spec. shell_split_path moves to _provision, and HostNotFound moves to gateway_base and now subclasses ConnectionError so generic OSError handling catches unreachable hosts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase B.1+B.2 of the Trio port: - add gateway_base.FrameDecoder, an incremental sans-IO decoder for the 9-byte-header Message framing (feed arbitrary chunks, complete messages come out; close() flags mid-frame EOF) - replace the hand-rolled AsyncByteIO wrappers (ProcessStreamsIO, FdStreamsIO, SocketStreamIO) with trio.StapledStream / plain trio.SocketStream behind a neutral ByteStream protocol (send_all/receive_some/send_eof/aclose) that a future anyio backend can satisfy structurally; the via tunnel keeps its interim bridge as ChannelByteStream - ProtocolSession's reader becomes the uniform receive_some+feed loop; exact reads survive only as the one-byte handshake ack - route worker exec requests through a single FIFO pump task: batched frame decoding removed the per-message awaits that had accidentally serialized main_thread_only exec admission, so admission now happens explicitly in message-arrival order instead of racing tasks - give pre-commit's mypy the trio dependency so trio types are real; drop now-redundant casts Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase B.3 of the Trio port: one portal module replaces the three ad-hoc cross-thread bridges. - new execnet/portal.py: LoopPortal (trio token holder with run/run_sync/post/is_loop_thread; post = run_sync_soon, strict FIFO) and SyncReceiver (loop-to-thread queue whose get() stays KeyboardInterrupt-interruptible on the main thread) - TrioHost owns a LoopPortal; call/call_sync/is_host_thread delegate - ProtocolSession's outbound queue becomes an unbounded trio memory channel; every send is posted through the portal so loop callbacks and foreign threads share one FIFO, and the writer task is a plain async-for -- the recreated-Event wake dance is gone. Blocking and close semantics are unchanged: non-loop threads still wait for the OS write (120s timeout), closed sends still raise OSError, and a finished loop maps trio.RunFinishedError to the same OSError. - TrioWorkerExec's main-thread exec handoff uses SyncReceiver Because each end only needs the other loop's token, the same primitive will serve two-loop setups (facade host loop + user loop) in B.5/B.6. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase B.4 starts the async-native inversion: a new _trio_gateway module hosts AsyncGateway, whose single serve task reads the framed Message protocol off a ByteStream (receive_some + sans-IO FrameDecoder) and dispatches inline -- no receiver thread, no receive lock -- plus a writer task draining an unbounded outbound queue. RawChannel is the low-level half of the two-level channel model: id-routed raw byte payloads with the sync Channel close semantics (CHANNEL_CLOSE both ways, CHANNEL_LAST_MESSAGE as write-EOF leaving the peer sendonly, CHANNEL_CLOSE_ERROR surfacing as RemoteError). Errors keep the execnet contract -- OSError on closed sends, EOFError/RemoteError on receive -- so no trio exception types leak into the API. The ByteStream protocol and RECEIVE_CHUNK move here from _trio_host so the async core sits at the bottom of the dependency stack. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The high level of the two-level channel model: AsyncChannel wraps a RawChannel with dumps/loads per item, per-channel strconfig (RECONFIGURE travels the wire, both ends coerce on load), async iteration, receive timeouts via trio.fail_after surfacing execnet's TimeoutError, and wait_closed mirroring the sync waitclose contract (reraise RemoteError). Channel objects serialize over the wire: a duck-typed save_AsyncChannel emits the existing CHANNEL opcode and AsyncGateway grows a factory adapter the Unserializer resolves ids through, so channels received inside items attach to the local gateway like sync channels do. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…o run open_popen_gateway spawns a _trio_worker subprocess, does the handshake, and serves an AsyncGateway directly in the caller's nursery -- the first end-to-end trio-native path with no host thread. AsyncGateway grows remote_exec accepting the same source kinds (string / pure function / module) as the sync API; exec-finish close, RemoteError propagation, and concurrent execs all flow through the raw/serialized channel layers. Source normalization moves to _exec_source (shared by both coordinators; gateway.py re-exports the old names for its tests), and the transport helpers (staple_*, handshake, popen argv) move down into _trio_gateway so the async core has no dependency on the sync host machinery. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
AsyncGroup is the trio-native group: an async context manager whose nursery serves every makegateway() as a child task. Leaving the block terminates all gateways concurrently with the safe_terminate contract -- GATEWAY_TERMINATE plus a timeout grace, then kill, bounded at roughly twice the timeout even when a kill sticks (issues pytest-dev#43/pytest-dev#221) -- with the cleanup shielded so external cancellation cannot leak workers. open_popen_gateway becomes a thin single-gateway AsyncGroup wrapper, so the popen integration tests now exercise the group termination path too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The via transport no longer double-frames. ChannelFactory grows a raw-receiver registry (CHANNEL_DATA payloads for registered ids route verbatim, no serialization) plus allocate_id, giving the sync gateways the low-level half of the two-level channel model. The master relay forwards the sub-worker's ready byte alone and then runs its stdout through a FrameDecoder, sending exactly one whole sub-protocol frame per CHANNEL_DATA -- and writes coordinator payloads (one frame each, by writer construction) straight to the sub's stdin. Coordinator ends of the tunnel match: RawTunnelStream (sync master, replacing the interim ChannelByteStream hack) and RawChannelStream (async master, wrapping a RawChannel as a ByteStream). AsyncGroup accepts via= specs relayed through a group member, and terminates tunneled gateways before their masters. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The sync coordinator and worker now run their protocol IO on the B.4 async core instead of the parallel ProtocolSession implementation: - AsyncGateway grows per-frame write acknowledgements (outbound queue items carry an optional on_written callback; the writer fails pending frames on shutdown instead of stranding their senders), plus a _finalize hook for subclass shutdown work. - AsyncGroup learns every transport (ssh=, vagrant_ssh=, socket= with installvia=, alongside popen/python=/via=), an overridable gateway factory, per-process reaper tasks, and remoteaddress stamping. - SyncBridgeGateway subclasses AsyncGateway to dispatch messages into the classic sync Message handlers under the receive lock; it keeps the xdist invariant that non-loop-thread sends block until the OS write (120s -> OSError) while loop-thread sends only enqueue, all in one portal-posted FIFO. ProtocolSession is gone; the worker serves on the same bridge. - multi.Group owns a FacadeAsyncGroup task on its TrioHost: makegateway delegates to AsyncGroup.makegateway, and terminate keeps the exit()/join() member contract while the bounded GATEWAY_TERMINATE + grace + kill shutdown runs through AsyncGroup.terminate. - Channel.__del__ posts its close message through the portal without waiting, so GC never blocks on a dying loop. - RawTunnelStream.aclose now feeds EOF to its own reader; previously a terminated via bridge could wait forever on a reader that no longer had a registered raw receiver. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cnet.portal execnet.sync re-exports the blocking facade; the top-level execnet.* names now alias into it. execnet.trio exposes the async-native core (AsyncGroup/AsyncGateway/AsyncChannel, raw channels, stream helpers, shared serialization + errors) for use inside the caller's own trio.run. execnet.portal (LoopPortal/SyncReceiver) is the communicating layer both build on. The trio and portal modules load lazily via module __getattr__ so plain `import execnet` still does not import the trio event loop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The worker created its SyncBridgeGateway inside host.call and only attached it to the sync gateway afterwards on the main thread. A coordinator message arriving in that window (STATUS, CHANNEL_EXEC as the first action on a fresh gateway) dispatched into a gateway whose _trio_session was still None, so the reply fell through to the sync IO stub and killed the session -- the whole test suite under pytest -n 12 showed this as freshly created gateways being dead on arrival. The race predates the B.5 facade (ProtocolSession had the same window). SyncBridgeGateway now attaches itself in __init__, before its serve task can dispatch anything, and the redundant attach calls at the two construction sites are gone. Also drop the apipkg-era unknown-attribute test from the namespace tests -- unknown attributes are plain Python module behaviour now. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
asyncssh stores its redirect cleanup coroutines (Queue.join et al) un-awaited and only runs them inside process.wait_closed(); the server handler never called it, so every ssh test leaked coroutines that pytest's unraisable hook surfaced as RuntimeWarnings during GC -- visible as stray noise in xdist runs. Await wait_closed() in the handler and drop the filterwarnings mark, which never covered the GC-time warning anyway. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…doff Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A ground-up rework of execnet's execution core onto Trio, replacing the thread-per-gateway receiver/writer architecture and the source-shipping bootstrap. The blocking public API is preserved byte-for-byte as a facade; a trio-native API is new.
One protocol engine
AsyncGateway(_trio_gateway.py) is the single protocol engine: one serve task per gateway running a reader (stream → sans-IOFrameDecoder→ dispatch) and a writer draining an outbound queue. Everything else layers on it:execnet.trio):AsyncGroup/AsyncGateway/AsyncChannelawaited directly inside the user's owntrio.run— no host thread. Two-level channel model:RawChannel(id-routed verbatim byte payloads) underneath the serializedAsyncChannel.execnet.sync, aliased by top-levelexecnet.*): unchanged API. Gateways run their IO on a per-GroupTrio host thread; sessions areSyncBridgeGateway— the same engine with dispatch routed into the classic syncMessagehandlers. The existing test suite passes unchanged against the facade.execnet.portal: the cross-thread/cross-loop primitives (LoopPortal,SyncReceiver) both surfaces build on.No source shipping
Workers run
python -m execnet._trio_worker <config-json>and import the installed execnet + trio; nothing is bootstrapped over the wire. Foreign interpreters (python=) and ssh/vagrant remotes are provisioned on demand viauv; dev coordinators (uninstalled checkouts) build and ship a wheel. A rough major/minor version check warns on coordinator/worker mismatch.Transports and coordination
All transports work on the new core: popen,
python=,ssh=,vagrant_ssh=,socket=(withinstallvia=), andvia=sub-gateways. Coordination that used to be shipped source is now protocol messages (GATEWAY_START_SUBfor via spawning with a frame-native byte relay on the master,GATEWAY_START_SOCKETfor installvia). A standaloneexecnet-socketserverconsole command replaces the shipped socketserver script.Semantics kept (pytest-xdist depends on these)
os._exitcannot drop already-"sent" data.remote_execadmission order == message arrival order.main_thread_onlykeeps working (worker main thread integration for GUI-bound code).Group.terminate(timeout)is bounded (~2× timeout) even when a kill sticks.The suite is xdist-clean:
pytest testing/ -n 12passes (and found a real session-startup race on the way, fixed in 51b9053).Breaking changes
eventletandgeventexecmodels are removed;threadandmain_thread_onlyremain.EXECNET_TRIO_HOSTescape hatch and the legacy bootstrap stack are removed.Not in this PR (planned follow-ups)
The phase handoff docs in the repo root (
handoff-phase-*.md) track the running plan:loop=thread|main×exec=thread|main|task(compat mapping fromexecmodel=), asyncremote_exec(exec=task), retiring theExecModel/WorkerPoolinternals behind deprecation shims.ByteStreamprotocol, sans-IO frame decoding) to keep that cheap.Test status
uv run pytest testing/: 514 passed, sequential and under-n 12(repeated runs).pre-commit run -aclean. ssh paths are covered by a real local harness (testing/test_ssh_local.py, asyncssh server + system ssh client).🤖 Generated with Claude Code