0.7.0

Monarch 0.7.0 brings a redesigned dashboard and a broader set of observability tools, including distributed profiling, custom metrics, and SQL queries from the command line. Synchronous actors no longer need an asyncio event loop, RDMA transfers can use several network cards at once, and many fixes make actor startup, shutdown, and failure reporting more reliable. This release also includes several breaking changes. Read Breaking changes before upgrading.

New features & API changes

Breaking changes

• Creating Futures: You can no longer construct a Future directly with Future(coro=...). Decorate an async def with @monarch.actor.returns_future instead. Callers can still use either .get() or await on the result (#4545, #4997).
• Queued dispatch is always on: The MONARCH_ACTOR_QUEUE_DISPATCH environment variable and the actor_queue_dispatch config option have been removed. Queued dispatch has been the default since 0.6.0. Actors that need to call back into themselves (reentrant calls) should use concurrent_endpoint (#4681).
• host_mesh_from_store() replaced: Use StoreJob.from_store() to set up torchrun-style, store-based rendezvous through the Jobs API (#4673).
• hosts_from_config moved: Import it from monarch.job instead of monarch.actor (#4761).
• Synchronous actors: An actor whose endpoints are all plain def methods now runs on a single dedicated thread with no asyncio event loop. Its endpoints, supervision, and cleanup run one at a time. As a result, such actors must also define __supervise__ with plain def, and calling an async method through as_endpoint on them now raises TypeError. Actors with any async def endpoint are unaffected (#5006, #5026).
• Operations that start immediately: job.exec_command(), RDMAAction.submit(), RDMABuffer.drop(), and shutdown_context() now begin as soon as you call them. Dropping or cancelling the returned Future no longer cancels the work (#4931, #4888, #4890, #5040).
• Shutdown timeouts: ProcMesh.stop(), HostMesh.stop(), and HostMesh.shutdown() still don't start until you call .get() or await them. If .get(timeout=...) times out, Monarch now cancels the remaining shutdown steps, though steps already under way may still finish (#5000, #5001).
• Messages to actors that are still starting: These messages are now held until the actor is ready instead of failing right away. One tradeoff: a message to an actor that never starts now fails after a 30-second timeout instead of immediately. Set HYPERACTOR_PENDING_ACTOR_DELIVERY_TIMEOUT to change the timeout, or set it to 0 for the old behavior (#4940, #4979).
• Hostname-based TCP transport: The TcpWithHostname transport, which the Slurm and Kubernetes jobs use, no longer picks a loopback address such as 127.0.0.1. If a machine's hostname only resolves to loopback, Monarch now fails with a message telling you to set an explicit address (#4424).
• Port.resolve_and_send() and Port.send_message() were internal helpers and are now private (#5029).

New features

• Redesigned dashboard: A new Overview page, an interactive map of your actors and meshes, and a searchable Explorer. You can capture a Python stack trace from any process with one click, and the dashboard tells you when a topology snapshot is incomplete (#4522, #4968).
• Observability with one call: job.enable_telemetry() now turns on telemetry, the mesh admin service, and topology snapshots together. Snapshots run every 30 seconds by default, and telemetry data is now kept for one hour instead of ten minutes (#4477, #4626).
• Profiling and inspection tools: Job.profile() and monarch profile record a Perfetto trace across every process in a job. monarch query runs SQL against a job's telemetry, and monarch tui shows the job's layout in the terminal (#4781, #4779, #5017, #5032).
• Custom metrics: Use monarch.actor.get_meter() to create your own OpenTelemetry counters, gauges, and histograms. You can query them alongside Monarch's built-in telemetry (#4559, #4554).
• Remote shells and port forwarding: monarch.job.shell() and the monarch shell command open an interactive shell on a remote host. monarch.job.PortForwarder forwards TCP traffic between processes (#4656, #4662).
• Process environment control: BootstrapCommand gains an inherit_env option. Set it to False to start processes from an empty environment. You can also remove an inherited variable by setting its value to None, for example cmd.with_env({"MY_VAR": None}). The docs now include a "Proc Environment" guide (#5041).
• Slurm improvements: SlurmJob now accepts account and qos, can attach to a cluster from a machine outside it, and lets you choose the address workers listen on (#4515, #4653, #4849).
• New worker startup options: start_worker_loop_forever() starts a worker right away. run_worker_loop_until_shutdown() runs a worker inside an existing program and returns when the worker shuts down, so the program itself doesn't exit (#4912, #4845).

Bug Fixes

• Fixed a problem where returning a Port from an async endpoint attached it to the wrong actor (#4711, issue #4658).
• Removed the most common cause of false "actor failed" reports. These happened when another thread was holding Python's global interpreter lock (GIL) (#4971, issue #4938).
• A busy async actor no longer starves its own timers and other endpoints, and async supervision handlers no longer block message processing (#4960, issue #4958, #4999).
• Fixed crashes when Python exits with actors still running, and shutdown now respects the caller's timeout (#4757, issue #4749, #4963).
• Fixed readiness checks and failure reports that pointed at the wrong rank when a mesh was sliced. Also fixed status tracking on clusters where hosts can be reached over both IPv4 and IPv6 (#4433, #4439, #4843).
• Fixed RDMA deadlocks. Memory now stays in place until every transfer that uses it has finished (#4620, #4637).
• RDMA buffers can now span more than one CUDA allocation, and RDMA picks the correct GPU when CUDA_VISIBLE_DEVICES is set (#4562, #4628).
• Mount cleanup is more reliable. Teardown is confirmed and retried if it fails, and stale macOS mounts no longer leak into later jobs (#5035, #5036, #5042).
• Endpoints can now send large NumPy arrays and bytearrays, which used to fail to serialize under pickle protocol 5. CUDA stream and event errors now raise Python exceptions instead of crashing the process (#4850, #4422).
• Fixed a startup hang that could happen when processes were launched from a parent that had already set up CUDA or NCCL (#4820).
• Starting a Monarch client from inside Monarch's own background runtime now raises a clear WouldBlockRuntime error instead of a confusing crash (#4713).
• monarch kill no longer fails on Kubernetes when run from a machine, such as a laptop, whose hostname doesn't resolve (#4970).

Performance & Reliability

• Faster RDMA: RDMA transfers are faster because of a reworked data path (#4836, #4846, #4847).
• Multiple NICs: A single transfer can now be split across several network cards (NICs) and connections. New defaults, tuned on H100 and GB200 systems, register each buffer on every NIC equally close to it and use four connections per peer. Use rdma_qps_per_peer, rdma_max_nics_per_buffer, and related settings in monarch.config.configure() to change them (#4733, #4863, #5022, #4833, #4922).
• Telemetry queries: Queries stay fast as data piles up, and they keep working when individual worker processes fail (#4570, #4622, #4634, #4476).
• RemoteMount: RemoteMount copies files to workers faster. You can now choose the TCP congestion-control algorithm, for example BBR, with the channel_tcp_congestion setting (#4654, #4671, #4937).
• Faster job startup: Processes now start in parallel. On Kubernetes, separate client programs can reuse an existing allocation and share a single port-forward (#4709, #4882, #4955).
• Python overhead: Python code awaiting Monarch results takes the GIL less often, and the mesh admin views stay responsive while they load data (#4991, #4840, #4841).

Build & Release

• Release wheels are now built against PyTorch 2.14.1, up from 2.13.0, with CUDA 13.2. Python 3.10 through 3.13 remain supported (#5009).
• You can now build Monarch from source against PyTorch's pip-installed ROCm 7.14 and 10.0 packages. This release also fixes GPU assignment on multi-GPU AMD hosts and a crash caused by Triton on ROCm. Set MONARCH_SKIP_TRITON_DEEPBIND=1 to turn off the Triton workaround (#4341).
• macOS clients can now use RemoteMount with Linux workers, and they can use GatherMount in read-only mode (#4961, #4957).
• Fixed installation on Python 3.12 and newer. PyTorch extensions now build with C++20, wheels build through the standard Python build tool, and building from source requires the Rust nightly-2026-07-05 toolchain, which rustup installs automatically (#4516, #4464, #4468).
• The torchmonarch[test] extra has been replaced by a test dependency group. Install it with uv sync --group test or pip install --group test (#5013).

Appendix — additional changes

• New documentation: a cookbook, observability and metrics guides, and an example that shares GPUs between actors. Also adds experimental ThreadTimeout support (#4365, #5008, #4795).
• Two experimental runtime projects, Monarch Mini and Chrysalis, with standalone examples and benchmarks (#4618, #4765, #4898, #4900).
• An experimental "direct spawn" mode for actor meshes, off by default and enabled with HYPERACTOR_MESH_USE_DIRECT_SPAWN, now reports failures for the correct rank (#4715, #4950).
• Clearer py-spy errors and a fallback to Python-only stack traces. Perfetto profiles now draw arrows linking each call to its handler and keep spans that were still open when the profile ended (#4782, #4815, #4871, #4933).
• Documented which fork patterns are safe in clients and workers. Removed the mesh admin --diagnose option (#4993, #5003, #4972).
• Pinned Rust dependency versions, updated the RDMA benchmarks, and added fuse3 and rsync to the published Docker images (#4467, #4751, #4767, #4945).