Skip to content

0.8.0

Latest

Choose a tag to compare

@dulinriley dulinriley released this 13 Nov 22:33
· 43 commits to main since this release

New features & API changes

  • Direct spawn is now the default: Ordinary actor meshes and system meshes with instance IDs use data-backed spawning; restore managed spawning with use_direct_spawn=False or HYPERACTOR_MESH_USE_DIRECT_SPAWN=false, and use an explicit readiness endpoint when you need Python constructor side effects, because initialized guarantees native routability only. #5311, #5336
  • Cleanup ordering changes: When an actor's __cleanup__ runs, stop messages have already been sent to its owned meshes and they process no further user messages, but their own cleanup may still be running. #5142
  • Custom launcher compatibility: Legacy service@... addresses, SPMDJob(service_proc_ids=...), and identity-distribution helpers in monarch.job are removed; launch workers and attach clients using bare channel URLs with automatic identity discovery. #5066, #5273
  • Telemetry persistence: Set TelemetryConfig.persist_dir and optionally client_persist_dir to archive job telemetry, then inspect it with monarch query --dir PATH or monarch dashboard open PATH using the same Monarch build that wrote the logs. #5294, #5296
  • SQLite tracing is removed: Use telemetry persistence for job-history archives and the native DataFusion query engine for dashboard data. #5298
  • Customize delivery failures with generic or kind-specific Python actor hooks and inspect typed reasons through UndeliverableMessageEnvelope.root_delivery_failure(), with sync and async callbacks and existing defaults preserved unless overridden. #5335
  • Inspect host and GPU inventories and aligned resource histories in the dashboard's Resources tab, Overview, and Explorer, with one-second sampling enabled by default. #5127, #5194, #5195, #5196
  • Track monarch exec command arguments, statuses, and per-rank exit codes in searchable dashboard command history, separately from fleet health. #5238, #5242, #5241
  • Launch jobs on PBS clusters with PBSJob, and read scheduler IDs through the new job_id property on PBSJob and SlurmJob. #5067, #5275
  • Install custom Python process launchers with HostMesh.set_proc_launcher(), including launchers that connect Firecracker microVM processes through vsock. #5151, #5158, #5159
  • Use experimental rendezvous tokens to connect trusted Python clients to existing actor meshes, with examples for trainer inspection and controller recovery. #5174, #5168, #5307
  • Run an SGLang rollout-engine fleet on Kubernetes or locally using a new example with batched generation, rolling weight updates, routing policies, and replica recovery. #5321
  • Select an interpreter for one command with monarch exec --python-executable PATH. #5051
  • Export logs, spans, and metrics over OTLP, and report Python gauge measurements with get_meter().create_gauge(...).set(...). #5090, #5091, #5116
  • Honor Intel XPU environment and allocator settings during actor startup alongside CUDA settings. #4307

Bug Fixes

  • Notify controller-based actor mesh owners through __supervise__ when an entire worker host is lost, including idle actors, while preserving intentional shutdown behavior. #5337; fixes #5274
  • Allow flatten() on contiguous actor, proc, and value mesh slices whose size-one dimensions retain parent strides. #5330; fixes #5328
  • Raise KeyError for missing Point labels so membership checks and get() work with absent dimensions. #5331; fixes #5329
  • Reconnect to restarted workers after stale sessions or delivery timeouts while preserving at-most-once delivery. #5099, #5118; fixes #5065
  • Fail actors and notify supervisors when concurrent_endpoint broadcasts raise, instead of silently discarding exceptions. #5283; fixes #5277
  • Keep healthy Linux workers alive when their launching Tokio thread retires. #5252; fixes #5248
  • Stop remote commands on Ctrl-C and reap ProcessJob workers even when the job sidecar has already exited. #5070, #5229
  • Run root-client cleanup and stop its asyncio event loop during shutdown. #5152; fixes #4749
  • Complete pending replies during supervision teardown and discard expired subscriber notifications without causing additional shutdown failures. #5175, #5235; fixes #5167, #5226
  • Fix native InfiniBand RDMABuffer transfers by selecting the port GID and using LID routing for peers on the same subnet. #5097; fixes #5085
  • Preserve telemetry on retention failures, keep remote collectors visible when clients run on worker hosts, fail scans with missing batches, and propagate collector query errors without dropping healthy collectors. #5140, #5132, #5254, #5256, #5317
  • Report signal-terminated commands as failures with shell-compatible exit codes. #5164
  • Fail promptly when the mesh-admin address is already in use. #5305

Performance & Reliability

  • Reduce endpoint and receive latency by decoding results on first observation and sharing them across readers. #5106, #5131, #5146
  • Reduce actor-spawn allocation and serialization overhead with smaller per-actor maps, one-time parameter encoding, and fewer system calls. #5218, #5219, #5210
  • Keep dashboards responsive during telemetry scans and overlap worker and root queries. #5112, #5255, #5257
  • Speed distributed telemetry queries with collector-side partial aggregation, parallel joins and aggregates across up to 16 partitions, and smaller-side join planning. #5317, #5324, #5325
  • Reduce per-message tracing overhead by avoiding allocations for static trace metadata and message-status strings. #5322, #5323
  • Configure data-actor failure detection independently of orphan cleanup and report lost workers with their process exit code or signal. #5246, #5308
  • Keep one mesh admin in the job sidecar across client invocations and register newly connected clients in its topology. #5312, #5313

Build & Release

  • Python 3.11 minimum: Python 3.10 support is removed, Python 3.14 wheels are added, and release builds move to PyTorch 2.15. #5221
  • Resolve decorated remote functions by module and qualified name on Python 3.14, avoiding unnecessary serialization by value through private pickle helpers. #5306
  • Discover ROCm through hipconfig and configured SDK paths, and use versioned HIP libraries without creating system symlinks. #5143, #5034
  • Publish nightlies, prereleases, and official releases through one workflow with consistent trusted-publisher provenance. #5084
  • Simplify Linux wheel builds by removing sccache and redundant native compilation. #5267

Appendix — additional changes

  • Add WeightBridge sources and integration documentation under examples/ for RDMA model-weight transfers between trainers and inference workers. #5069
  • Launch interactive shells with a controlling terminal on macOS and Linux. #5186
  • Report explicit dashboard health, retain actor lifecycle history, distinguish the controller from the job sidecar, hide monitor actors with system actors, and improve drawer navigation and small-screen layouts. #5115, #5141, #5334, #5327
  • Preserve supervision causes, count failed data actors accurately, and allow subsequent spawns after failures. #5072, #5185
  • Clean up children created during actor teardown, run data actors' cleanup before proc shutdown, and stop macOS workers when their hosting process exits. #5123, #5124, #5188
  • Allow brief worker pauses and request proc shutdown without waiting for remote actor initialization. #5299, #5309
  • Recover complete telemetry frames before damaged log tails and finalize the client archive before graceful sidecar shutdown. #5290, #5291
  • Report interrupted QUIC streams as errors and log repeated CUDA-context warnings once per process. #5244, #5064; fixes #5049
  • Lock documentation builds with uv, extend frontend CI checks, and remove unnecessary DataFusion and PyO3 dependencies from Arrow-only crates. #5047, #5136, #5130, #5261
  • Add a scatter cookbook recipe and use SPMD job names in topology and telemetry. #5251, #5121
  • Extend experimental Chrysalis Python postcard examples to separate processes, mixed UDP/vsock carriers, multiple relays, and Python Firecracker guest images. #5318, #5320, #5332, #5333, #5326
  • Flush forwarded logs to the socket actually bound, including fallback addresses. #5341

New benchmark results

Benchmark functions must be called from a runnable script that creates a JobTrait.

Published results were obtained from a client and remote CPU workers in one data-centre region.

Both benchmarks were run using 20 warm-up iterations and 100 measured iterations.

Bootstrap

Benchmark: python/benches/bootstrap_benchmark.py

Hosts Worker processes Total bootstrap p50 (s) Total bootstrap p99 (s) Proc-mesh p50 (s) Proc-mesh p99 (s) Actor-mesh p50 (s) Actor-mesh p99 (s) First no-op RPC p50 (s) First no-op RPC p99 (s)
1 8 0.664 0.757 0.604 0.677 0.054 0.079 0.002 0.007
8 64 0.828 0.902 0.705 0.779 0.117 0.151 0.003 0.004
64 512 1.056 1.190 0.842 0.982 0.193 0.238 0.018 0.023
128 1,024 1.113 1.265 0.864 1.021 0.222 0.253 0.032 0.038
512 4,096 1.886 2.448 1.383 1.969 0.388 0.438 0.116 0.130
1,024 8,192 2.767 2.988 1.982 2.176 0.550 0.600 0.232 0.256
2,048 16,384 4.638 5.382 3.299 4.002 0.865 0.939 0.462 0.512

RPC Ping-Pong (Empty payload)

Benchmark: python/benches/noop_rpc_benchmark.py

Hosts Actors per host Total actors p50 RPC latency (ms) p99 RPC latency (ms)
1 8 8 1.193 1.627
8 8 64 1.394 3.148
64 8 512 2.596 3.588
128 8 1,024 3.079 4.787
256 8 2,048 4.017 6.469
512 8 4,096 5.284 8.498
1,024 8 8,192 6.650 9.637
2,048 8 16,384 9.027 12.808

Hardware configuration

Component Client Remote worker
CPU AMD EPYC-Genoa Processor; 1 socket, 96 cores, 96 threads Intel Xeon D-2191A (Skylake-DE); 1 socket, 18 cores, 36 threads
RAM 235 GiB assigned to the virtual machine 64 GB DDR4 installed per host; 56,832 MiB assigned to each worker container
Storage 1.23 TB virtual disk; Btrfs root filesystem 512 GB M.2 PCIe SSD per host; Btrfs root filesystem
Virtualization KVM virtual machine LXC container on a bare-metal host
OS CentOS Stream 9, x86-64 CentOS Stream 10, x86-64
Network transport TLS over TCP/IPv6 TLS over TCP/IPv6
NIC Red Hat Virtio network device (eth0, PCI ID 1af4:1000) Broadcom Stratus network device; 50 GbE