Skip to content

ModelExpress v0.7.0

Latest
Latest

Choose a tag to compare

View all tags
@nv-nmailhot nv-nmailhot released this 03 Oct 18:28
v0.7.0
ac6d03a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

ModelExpress v0.7.0 Release Notes

The big picture

ModelExpress exists to eliminate the wait, and to become the weight and artifact management layer for both inference and RL systems. v0.6.0 landed RL post-training weight refit end to end: the reshard-refit engine plus the first trainer and generator clients.

v0.7.0 turns that foundation into a production RL weight-update system. The centerpiece is a canonical checkpoint and delta-refit pipeline built on S3: trainers publish version-owned, immutable checkpoints to S3, and generators refit from them — either a full checkpoint or a target-version delta replayed onto what's already resident. Around that, the release adds a P2P-first active refit path, an RL cold-start load-strategy chain with distributed reconciliation, trainer-only MoE weight updates, and a hardened immutable-checkpoint lifecycle, with the vLLM weight transfer engine now integrated directly. Two other themes round out the release: smarter P2P source selection — topology-aware, bandwidth/load-aware, and PCI-affinity-aware — and operational observability, with PodMonitors, alert rules, a Grafana dashboard, and phase-level timing across model load, artifact validation, and P2P attempts. CRDs are now generated from the Rust types, giving Kubernetes a single source of truth.

What's new in this release

v0.7.0 lands four things that matter:

  1. Canonical S3 checkpoints + delta refit: version-owned canonical S3 publication and generator integration, full-checkpoint cadence with optional checksums, and target-version delta replay, so RL generators refit from an immutable, S3-backed source of truth. (#685, #686, #705, #710)
  2. P2P-first active refit and cold starts: a P2P-first active refit path, an RL cold-start load-strategy chain with distributed reconciliation, and generator P2P received directly into live tensors. (#740, #727, #738, #767)
  3. Smarter P2P source selection: topology-aware and bandwidth/load-aware source selection, with PCI root complex retained in affinity scoring. (#512, #510, #739)
  4. Operational observability: PodMonitors, alert rules, and a Grafana dashboard, plus phase-level timing for model load, artifact validation, and P2P attempts. (#681, #706, #744, #745)

RL weight refit

  • vLLM weight transfer engine integrated: the vLLM weight transfer engine is wired directly into the RL refit path. (#684)
  • Canonical S3 publication and generator: version-owned canonical S3 checkpoint publication and the matching canonical S3 generator integration. (#685, #686)
  • Full checkpoints and delta replay: support a full-checkpoint cadence with optional checksums, and target-version delta replay onto resident weights. (#705, #720, #710)
  • Immutable checkpoint lifecycle: harden the immutable checkpoint lifecycle, and harden and optimize checkpoint materialization. (#709, #717)
  • P2P-first active refit: an active refit path that prefers P2P, receives generator P2P directly into live tensors, and protects active P2P readers during donor mutation. (#740, #767, #757)
  • Cold starts: an RL cold-start load-strategy chain, with distributed cold-start reconciliation. (#727, #738)
  • Trainer-only MoE updates: support trainer-only MoE weight updates. (#704)
  • Non-CUDA FSDP shards: publish the served device ordinal for non-CUDA FSDP shards. (#683)
  • Native→HF capture + minimal-read generator: capture native→HF conversion and add a minimal-read generator. (#680)
  • Version identity and path model: allow caller-supplied version UIDs, remove optional weight version numbers, compose the RL weight-update paths, reuse manifest metadata, and rename the checkpoint path settings (with a new S3 delta refit doc). (#694, #691, #690, #735, #701)
  • Performance: batch the initial layout capture. (#753)
  • CI and recipes: add Vime + Dynamo vLLM CI and example recipes for S3 delta refit. (#736)

P2P source selection

  • Topology-aware selection: choose P2P sources based on cluster topology. (#512)
  • Bandwidth/load-aware selection: a load_aware strategy that accounts for available bandwidth. (#510)
  • PCI-affinity scoring: retain the PCI root complex in affinity scoring for better peer pairing. (#739)

Observability and metrics

  • Dashboards and alerts: PodMonitors, alert rules, and a Grafana dashboard. (#681)
  • Phase-level timing: time the model load and its phases, time validate and extract per artifact archive, and split a P2P source attempt into the phases the strategy owns. (#706, #744, #745)

Transfer plane and performance

  • UCX/RDMA tuning: default UCX_MEM_EVENTS=n for NIXL agent creation, and spread RDMA rails across ranks while reporting a throughput floor. (#688, #689)
  • RDMA P2P config fix: fix RDMA P2P being silently disabled when model_loader_extra_config is set. (#718)
  • Stale publish markers: reclaim stale publish markers after a worker restart. (#675)

Engine and client fixes

  • SGLang: avoid duplicate model allocation on retry, and support model_path in GDS loading. (#611, #605)
  • vLLM: refresh host attention scales after RDMA. (#711)
  • Snapshot resolution: install weights into the snapshot root the engine resolved. (#721)
  • Heartbeats: re-register the source when the server rejects heartbeats. (#658)

Kubernetes, CRDs, and Helm

  • CRDs from Rust types: generate CRD manifests from the Rust types, bound uint32 fields in the OpenAPI schema, and use the canonical CRD manifest in CI. (#722, #746, #723)
  • S3 in the schema: accept S3 in the ModelCacheEntry provider enum and regenerate model_pb2 so ModelProvider exposes S3. (#699, #693)
  • RBAC: remove the unused CRD reader role. (#750)
  • Helm: make the chart's non-root default actually deployable, and keep apiVersion out of the SPDX comment in ingress.yaml. (#682, #707)
  • Docs: document CRD updates for existing clusters. (#696)

Documentation

  • Reorganize the ModelExpress documentation. (#702)
  • Unify coding-agent instructions under AGENTS.md and shared skills. (#726)

CI, build, dependencies, and housekeeping

  • CI: bump the GHA version and cargo matrix and add upstream guards, install cargo-audit with locked dependencies, and add a DCO compliance skill. (#657, #724, #725)
  • Dependencies and crypto: bump rustls to 0.23.45 for RUSTSEC-2026-0285, and keep ring out of OpenSSL builds. (#756, #728)
  • Release bookkeeping: update attributions for 0.5.1, bump the version 0.5.0 → 0.5.1, and bump the version 0.5.1 → 0.7.0. (#663, #618, #772)

Where we're headed

v0.7.0 makes RL weight refit a production checkpoint-and-delta system and gives it the source selection and observability to run at scale. The next releases focus on:

  • Refit throughput and scale: pushing delta-replay and P2P-first refit across larger topologies and more training frameworks, building on canonical S3 checkpoints toward end-to-end verification.
  • Selection and placement: extending topology- and bandwidth-aware selection into fuller placement decisions across heterogeneous fabrics.
  • Observability-driven operations: turning the dashboards and phase-level timing into SLOs, non-disruptive rolling upgrades, and multi-tenant isolation.

The longer arc is unchanged: ModelExpress becomes the weight and artifact management layer for inference and RL systems, the piece that makes model placement, scaling, refit, and migration fast enough that the orchestrator can treat GPU memory as a fungible resource across the cluster.

New Contributors

Contributors

Thank you to everyone who contributed to this release, and especially to the team that built out the canonical S3 checkpoint and delta-refit pipeline across dozens of PRs, and to the reviewers and testers who validated the source-selection, observability, and CRD paths across Kubernetes and multi-node setups.

Full changelog

v0.6.0...v0.7.0

Assets 2
Loading