Repository navigation
New features & API changes
- Direct spawn is now the default: Ordinary actor meshes and system meshes with instance IDs use data-backed spawning; restore managed spawning with
use_direct_spawn=FalseorHYPERACTOR_MESH_USE_DIRECT_SPAWN=false, and use an explicit readiness endpoint when you need Python constructor side effects, becauseinitializedguarantees native routability only. #5311, #5336 - Cleanup ordering changes: When an actor's
__cleanup__runs, stop messages have already been sent to its owned meshes and they process no further user messages, but their own cleanup may still be running. #5142 - Custom launcher compatibility: Legacy
service@...addresses,SPMDJob(service_proc_ids=...), and identity-distribution helpers inmonarch.jobare removed; launch workers and attach clients using bare channel URLs with automatic identity discovery. #5066, #5273 - Telemetry persistence: Set
TelemetryConfig.persist_dirand optionallyclient_persist_dirto archive job telemetry, then inspect it withmonarch query --dir PATHormonarch dashboard open PATHusing the same Monarch build that wrote the logs. #5294, #5296 - SQLite tracing is removed: Use telemetry persistence for job-history archives and the native DataFusion query engine for dashboard data. #5298
- Customize delivery failures with generic or kind-specific Python actor hooks and inspect typed reasons through
UndeliverableMessageEnvelope.root_delivery_failure(), with sync and async callbacks and existing defaults preserved unless overridden. #5335 - Inspect host and GPU inventories and aligned resource histories in the dashboard's Resources tab, Overview, and Explorer, with one-second sampling enabled by default. #5127, #5194, #5195, #5196
- Track
monarch execcommand arguments, statuses, and per-rank exit codes in searchable dashboard command history, separately from fleet health. #5238, #5242, #5241 - Launch jobs on PBS clusters with
PBSJob, and read scheduler IDs through the newjob_idproperty onPBSJobandSlurmJob. #5067, #5275 - Install custom Python process launchers with
HostMesh.set_proc_launcher(), including launchers that connect Firecracker microVM processes through vsock. #5151, #5158, #5159 - Use experimental rendezvous tokens to connect trusted Python clients to existing actor meshes, with examples for trainer inspection and controller recovery. #5174, #5168, #5307
- Run an SGLang rollout-engine fleet on Kubernetes or locally using a new example with batched generation, rolling weight updates, routing policies, and replica recovery. #5321
- Select an interpreter for one command with
monarch exec --python-executable PATH. #5051 - Export logs, spans, and metrics over OTLP, and report Python gauge measurements with
get_meter().create_gauge(...).set(...). #5090, #5091, #5116 - Honor Intel XPU environment and allocator settings during actor startup alongside CUDA settings. #4307
Bug Fixes
- Notify controller-based actor mesh owners through
__supervise__when an entire worker host is lost, including idle actors, while preserving intentional shutdown behavior. #5337; fixes #5274 - Allow
flatten()on contiguous actor, proc, and value mesh slices whose size-one dimensions retain parent strides. #5330; fixes #5328 - Raise
KeyErrorfor missingPointlabels so membership checks andget()work with absent dimensions. #5331; fixes #5329 - Reconnect to restarted workers after stale sessions or delivery timeouts while preserving at-most-once delivery. #5099, #5118; fixes #5065
- Fail actors and notify supervisors when
concurrent_endpointbroadcasts raise, instead of silently discarding exceptions. #5283; fixes #5277 - Keep healthy Linux workers alive when their launching Tokio thread retires. #5252; fixes #5248
- Stop remote commands on Ctrl-C and reap
ProcessJobworkers even when the job sidecar has already exited. #5070, #5229 - Run root-client cleanup and stop its asyncio event loop during shutdown. #5152; fixes #4749
- Complete pending replies during supervision teardown and discard expired subscriber notifications without causing additional shutdown failures. #5175, #5235; fixes #5167, #5226
- Fix native InfiniBand
RDMABuffertransfers by selecting the port GID and using LID routing for peers on the same subnet. #5097; fixes #5085 - Preserve telemetry on retention failures, keep remote collectors visible when clients run on worker hosts, fail scans with missing batches, and propagate collector query errors without dropping healthy collectors. #5140, #5132, #5254, #5256, #5317
- Report signal-terminated commands as failures with shell-compatible exit codes. #5164
- Fail promptly when the mesh-admin address is already in use. #5305
Performance & Reliability
- Reduce endpoint and receive latency by decoding results on first observation and sharing them across readers. #5106, #5131, #5146
- Reduce actor-spawn allocation and serialization overhead with smaller per-actor maps, one-time parameter encoding, and fewer system calls. #5218, #5219, #5210
- Keep dashboards responsive during telemetry scans and overlap worker and root queries. #5112, #5255, #5257
- Speed distributed telemetry queries with collector-side partial aggregation, parallel joins and aggregates across up to 16 partitions, and smaller-side join planning. #5317, #5324, #5325
- Reduce per-message tracing overhead by avoiding allocations for static trace metadata and message-status strings. #5322, #5323
- Configure data-actor failure detection independently of orphan cleanup and report lost workers with their process exit code or signal. #5246, #5308
- Keep one mesh admin in the job sidecar across client invocations and register newly connected clients in its topology. #5312, #5313
Build & Release
- Python 3.11 minimum: Python 3.10 support is removed, Python 3.14 wheels are added, and release builds move to PyTorch 2.15. #5221
- Resolve decorated remote functions by module and qualified name on Python 3.14, avoiding unnecessary serialization by value through private pickle helpers. #5306
- Discover ROCm through
hipconfigand configured SDK paths, and use versioned HIP libraries without creating system symlinks. #5143, #5034 - Publish nightlies, prereleases, and official releases through one workflow with consistent trusted-publisher provenance. #5084
- Simplify Linux wheel builds by removing sccache and redundant native compilation. #5267
Appendix — additional changes
- Add WeightBridge sources and integration documentation under
examples/for RDMA model-weight transfers between trainers and inference workers. #5069 - Launch interactive shells with a controlling terminal on macOS and Linux. #5186
- Report explicit dashboard health, retain actor lifecycle history, distinguish the controller from the job sidecar, hide monitor actors with system actors, and improve drawer navigation and small-screen layouts. #5115, #5141, #5334, #5327
- Preserve supervision causes, count failed data actors accurately, and allow subsequent spawns after failures. #5072, #5185
- Clean up children created during actor teardown, run data actors' cleanup before proc shutdown, and stop macOS workers when their hosting process exits. #5123, #5124, #5188
- Allow brief worker pauses and request proc shutdown without waiting for remote actor initialization. #5299, #5309
- Recover complete telemetry frames before damaged log tails and finalize the client archive before graceful sidecar shutdown. #5290, #5291
- Report interrupted QUIC streams as errors and log repeated CUDA-context warnings once per process. #5244, #5064; fixes #5049
- Lock documentation builds with uv, extend frontend CI checks, and remove unnecessary DataFusion and PyO3 dependencies from Arrow-only crates. #5047, #5136, #5130, #5261
- Add a scatter cookbook recipe and use SPMD job names in topology and telemetry. #5251, #5121
- Extend experimental Chrysalis Python postcard examples to separate processes, mixed UDP/vsock carriers, multiple relays, and Python Firecracker guest images. #5318, #5320, #5332, #5333, #5326
- Flush forwarded logs to the socket actually bound, including fallback addresses. #5341
New benchmark results
Benchmark functions must be called from a runnable script that creates a JobTrait.
Published results were obtained from a client and remote CPU workers in one data-centre region.
Both benchmarks were run using 20 warm-up iterations and 100 measured iterations.
Bootstrap
Benchmark: python/benches/bootstrap_benchmark.py
| Hosts | Worker processes | Total bootstrap p50 (s) | Total bootstrap p99 (s) | Proc-mesh p50 (s) | Proc-mesh p99 (s) | Actor-mesh p50 (s) | Actor-mesh p99 (s) | First no-op RPC p50 (s) | First no-op RPC p99 (s) |
|---|---|---|---|---|---|---|---|---|---|
| 1 | 8 | 0.664 | 0.757 | 0.604 | 0.677 | 0.054 | 0.079 | 0.002 | 0.007 |
| 8 | 64 | 0.828 | 0.902 | 0.705 | 0.779 | 0.117 | 0.151 | 0.003 | 0.004 |
| 64 | 512 | 1.056 | 1.190 | 0.842 | 0.982 | 0.193 | 0.238 | 0.018 | 0.023 |
| 128 | 1,024 | 1.113 | 1.265 | 0.864 | 1.021 | 0.222 | 0.253 | 0.032 | 0.038 |
| 512 | 4,096 | 1.886 | 2.448 | 1.383 | 1.969 | 0.388 | 0.438 | 0.116 | 0.130 |
| 1,024 | 8,192 | 2.767 | 2.988 | 1.982 | 2.176 | 0.550 | 0.600 | 0.232 | 0.256 |
| 2,048 | 16,384 | 4.638 | 5.382 | 3.299 | 4.002 | 0.865 | 0.939 | 0.462 | 0.512 |
RPC Ping-Pong (Empty payload)
Benchmark: python/benches/noop_rpc_benchmark.py
| Hosts | Actors per host | Total actors | p50 RPC latency (ms) | p99 RPC latency (ms) |
|---|---|---|---|---|
| 1 | 8 | 8 | 1.193 | 1.627 |
| 8 | 8 | 64 | 1.394 | 3.148 |
| 64 | 8 | 512 | 2.596 | 3.588 |
| 128 | 8 | 1,024 | 3.079 | 4.787 |
| 256 | 8 | 2,048 | 4.017 | 6.469 |
| 512 | 8 | 4,096 | 5.284 | 8.498 |
| 1,024 | 8 | 8,192 | 6.650 | 9.637 |
| 2,048 | 8 | 16,384 | 9.027 | 12.808 |
Hardware configuration
| Component | Client | Remote worker |
|---|---|---|
| CPU | AMD EPYC-Genoa Processor; 1 socket, 96 cores, 96 threads | Intel Xeon D-2191A (Skylake-DE); 1 socket, 18 cores, 36 threads |
| RAM | 235 GiB assigned to the virtual machine | 64 GB DDR4 installed per host; 56,832 MiB assigned to each worker container |
| Storage | 1.23 TB virtual disk; Btrfs root filesystem | 512 GB M.2 PCIe SSD per host; Btrfs root filesystem |
| Virtualization | KVM virtual machine | LXC container on a bare-metal host |
| OS | CentOS Stream 9, x86-64 | CentOS Stream 10, x86-64 |
| Network transport | TLS over TCP/IPv6 | TLS over TCP/IPv6 |
| NIC | Red Hat Virtio network device (eth0, PCI ID 1af4:1000) |
Broadcom Stratus network device; 50 GbE |