0.6.0

Monarch v0.6.0

Released to: https://pypi.org/project/torchmonarch/0.6.0/

New features & API changes

Queue-based actor dispatch is now the default. Python actors dispatch messages through a queue by default, decoupling handler execution from the receive loop; direct dispatch remains available via actor_queue_dispatch=False and warns once (#4211).

New @concurrent_endpoint decorator. An endpoint marked @concurrent_endpoint runs its body in a background task, so the actor keeps servicing other messages while the call is in flight (#4243).

RDMA action API renamed. RDMAAction.read_into / write_from become read_remote / write_remote with destination-first signatures, overlap rules aligned to data direction, and the default timeout raised from 3s to 60s (#4078).

Out-of-cluster clients. New monarch.actor.attach_client(addr) lets a Python client outside a cluster reach an in-cluster mesh through an exposed host gateway (#4142), and Kubernetes gains full out-of-cluster support so an external client can attach to and drive host meshes (#3153, #2657).

SlurmJob batch mode. SlurmJob.apply(client_script=…) runs the workers and the client in a single SLURM allocation that is torn down when the client exits (#4281).

Future async surface hardened. The public Future is now Handle-backed, so get(), await, and as_asyncio() mix freely and a timed-out get() no longer poisons the future (#4363, #4367); Future.get() from a running asyncio loop is still supported and now emits only a UserWarning that it may block — the earlier deprecation was dropped — while calling it on a Tokio worker thread raises up front (#4113, #4368).

Telemetry API simplified. enable_telemetry() always configures the sidecar collector; the legacy in-process path and TelemetryConfig.use_sidecar / batch_size are removed (#4378, #4379). The new distributed telemetry system now launches telemetry actor per-host instead of per-proc, and it is now hosted in the job's sidecar process. Now the distributed telemetry system persists across job invocation. job.query_engine is now removed, use job.query_engine_client instead.

Also: default_bootstrap_cmd() is now public (#4327); the shared per-process asyncio runtime for Python actors was removed (#4242); and the admin TUI gained a ? help overlay (#4114), a live Python handler-execution pane (#4188, #4189), and a --plaintext HTTP option (#4383).

Bug Fixes

• Proc respawn reusing a TCP port. When a proc was respawned and came back on a peer's previous TCP address, DialMailboxRouter kept a dead cached connection keyed by that address and silently sent to it forever, hanging the mesh; the router now detects the closed (stale-session) connection, evicts it, and re-dials the new proc (#4067).
• Actors & endpoints: async-endpoint actors no longer fail to construct when __cleanup__ is not overridden (#3958); returning a mesh from an endpoint no longer triggers an Already borrowed crash (#3973); a concurrent_endpoint that raises now fails the actor instead of hanging the caller (#4300); run_worker_loop_forever binds alias addresses correctly (#4160); and aggregated proc-mesh logs now flush in async IPython/Jupyter cells (#4387).
• Queue dispatch: fixed a debugger deadlock (#4309) and an AttributeError crash under lazy-import builds (#4376).
• RDMA: hosts without a loadable CUDA/HIP driver degrade gracefully instead of crashing across the FFI boundary (#4225); mismatched read/write buffer sizes are handled correctly (#4273); RDMA succeeds on small or unaligned CUDA tensors (#4285); and RDMA action batching was ported to Rust, fixing overlap and inverted-argument bugs (#4079).
• Other: out-of-cluster return-path routing (#4322, #2657); a supervision-propagation topology race (#4336, #3182); and from typing import Self on Python 3.10 (#4060).

Performance & Reliability

• Faster remote mounts. remotemount was reworked into an on-demand per-block FUSE mount with leader-mediated RDMA fan-out and zero-copy buffers, cutting cold torch import from ~25s to ~20s (#4324, #4356).
• RDMA throughput & isolation: added hardware-aware parallelization, native op batching and memory registration caching, providing substantial throughput wins. Host-memory registrations are scattered across all equally-optimal NICs to recover aggregate multi-NIC bandwidth (#4357), and RDMA manager actors run on a dedicated data-plane runtime so buffer-registration work no longer stalls dispatch or supervision (#4302).
• Failure detection & delivery: silently-dead TCP peers are detected via kernel keepalive with bounded reconnect (#4083); senders can observe delivery errors inline via try_post (#4012); and undeliverable handling was rebuilt on a structured DeliveryFailure taxonomy for clearer error reporting (#4016 – #4034).
• Messaging: channel delivery now uses explicit completion sinks and receipts (#4261 – #4264) and receiver-local handler sequencing (#4178); the mesh-attach timeout was raised to 60s (#4299); and the Tokio worker-thread count is configurable (#4347).

Build & Release

• QUIC transport. A new Quinn-backed quic / metaquic channel backend adds ordered and unordered multi-stream delivery (#4152).
• RDMA backends. RDMA now makes it easier to support new NIC types and transport strategies with a backend-agnostic ibverbs device abstraction and generic backend trait (#4162, #4280).
• Toolchains: the default CUDA toolchain was bumped to 13.2 (#3881), and PyTorch was bumped to 2.13 across all builds.
• Packaging: the RDMA Cargo feature was split out of tensor_engine_gpu (#4241); torchx moved to an optional [torchx] extra (#4245).