Skip to content

v0.1.3

Latest
Latest

Choose a tag to compare

View all tags
@HJSang HJSang released this 23 Oct 03:25
· 196 commits to main since this release
v0.1.3
74f47d3

TL;DR

Native LoRA without Megatron-Bridge. LoRA now works in Megatron raw mode (--megatron-to-hf-mode raw) for the Llama, Qwen2/3, Qwen3.5/3.6, GLM-4/4.7/5.x, DeepSeek-V3/V3.2, Kimi-K2.5/K3 and Inkling families on one shared implementation. GLM-5.3-Flash is the first model to use it, with a launcher validated on 2x8 H200.

Rebuilt P2P weight transfer. --update-weight-transfer-mode p2p no longer keeps a full CPU copy of the model on every sender. Senders now use two reusable pinned buffers instead. In one measured setup, registered memory dropped from 28 GiB to 2 GiB per sender and the first update from 99.5 s to 3.9 s. A failed write now fails the update instead of being logged and ignored. Each write can be checksummed on arrival. Trained MTP layers now reach the speculative draft.

Weight updates that survive cell failures. With fault tolerance on and P2P transfer, a rollout cell that fails its weight update is taken out of service and the rest still get the new weights. A trainer cell that dies or hangs mid-update is given up instead of blocking the run. Independent data parallelism keeps training when the surviving cells do not divide the global batch.

Fully async RL, colocated and resumable. --fully-async --colocate runs generation and training on the same GPUs, down to a single GPU. Fully async runs now checkpoint and restore their buffer and in-flight work. New metrics report how stale each rollout's prompt KV cache is.

Faster loss head and session server. The new opt-in --log-probs-backend fused computes log-probs and entropy in one Triton kernel. That cuts trainer peak memory by a third in a 30K-token GRPO run. The session server does much less work per turn.

New stack. The image moves to SGLang v0.5.22 with torch 2.14 wheels, a Miles-built Transformer Engine 2.17.0+miles and transformers 5.19.0.

Upgrade carefully. Rollout-side checkpoints have a new layout, so v0.1.2 runs cannot be resumed on v0.1.3. The DataSource API changed. MTP training now needs --enable-mtp-training. Rollout fault tolerance now requires P2P weight transfer. See Breaking Changes & Upgrade Notes.

Highlights

287 PRs from 17 contributors.

Model Support

Model Training What changed PRs Docs
GLM-5.3-Flash LoRA (native) New LoRA training mode for an already-supported model #3098 @yushengsu-thu, #1792 @Zhichenzzz link
DeepSeek-V4 Flash Full, MXFP8 (Blackwell) New mode: train directly from the HF checkpoint and serve MXFP8 rollout weights from the trainer #3135 @bobboli —

GLM-5.3-Flash LoRA. scripts/run_glm5_3_flash_lora.py trains adapters on the KDA, DSA (except kv_b_proj) and dense/shared MLP projections. Routed experts stay frozen. On the full model on 2x8 H200 (TP8, EP16, DAPO-Math-17k, 8K responses, R3), reward rose from 0.74 to 0.81 over 100 rollouts. The train/rollout log-prob difference averaged 0.022. The rollout side needs sgl-project/sglang#44862, which is in the locked sglang-miles (#3098 @yushengsu-thu).

DeepSeek-V4 Flash from HF, with MXFP8. --init-model-source hf builds the Megatron trainer straight from the official HF checkpoint through Megatron-Bridge, so no offline conversion is needed. --rollout-weight-source trainer --rollout-mxfp8 serves rollout weights produced by the trainer. Requirements:

  • Blackwell, because MXFP8 is rejected on H100/H200.
  • dsv4_impl=megatron.

Evidence so far is a 3-step smoke run on 8x4 GB300, not a long run (#3135 @bobboli).

New recipe pages for models on dedicated images. These pages document runs of Step-5-Preview (#4104 @HJSang), Muse Spark Open (#4112, #4144 @guapisolo) and Beam 1.0 (#4138 @zhihengy). Their model code is not in this release. Each runs from its own image: radixark/miles:step5, radixark/miles:muse-spark-open or radixark/miles:beam. See the Step-5-Preview, Muse Spark Open and Beam 1.0 pages.

Features

Native LoRA for Megatron raw mode. Before this release, raw-mode LoRA existed only as one-off implementations for Inkling and Kimi K3. A single backend under miles/backends/megatron_utils/lora/native/ now covers every registered architecture: llama, qwen2/3, qwen3_moe, qwen3_5/3_6 (dense and MoE), qwen3_next, mimo, glm4, glm4_moe, glm4_moe_lite, glm_moe_dsa, glm5_next, deepseek_v3/v32, kimi_k2/k25/k3, joyai_llm_flash and inkling. --save-hf merges the adapters into the base weights. Limits:

  • Single LoRA, colocated only.
  • Rejected: --overlap-param-gather, --overlap-grad-reduce and --moe-shared-expert-overlap.
  • Routed-expert adapters only for Inkling and Kimi K3 (with --experts-shared-outer-loras). Use Bridge mode for routed-expert LoRA on other models.

CI registers H200 end-to-end cases for Qwen3-0.6B (TP2+SP), GLM-5.2 5-layer, Kimi-K2.5 2-layer, GLM-4.7-Flash and Qwen3.5-35B-A3B (docs, #1792 @Zhichenzzz).

P2P weight transfer, rebuilt. Each sender used to hold a full pinned CPU copy of the model. Now it builds storage-free replicas of each engine's layout and loads weights through two reusable pinned buffers, registered once. On Qwen3-30B-A3B (TP2, 1 GiB buffers), compared with the previous P2P path:

  • Registered memory per sender fell from 28.45 to 2.00 GiB.
  • Peak host RSS fell from about 125 to 88 GiB.
  • The first update, including connect, fell from 99.5 to 3.9 s.
  • Steady-state updates on H200 were about 6% slower.

Other changes:

  • Writers are assigned from the engines' real sizes and placement, and Bridge mode is accepted.
  • Weight mappings come from probing the engine's own loader, so models need no hand-written mapping.
  • Every write error or timeout fails the update, and each engine gets its own write queue.
  • With MTP training and same-model EAGLE, both the target and the draft are updated, so trained MTP weights reach speculative decoding.

Requirements: trainer and engines must run the same SGLang build on the same GPU architecture, and P2P cannot be combined with --colocate (docs, #3939–#3943, #3945, #3950, #3959, #3969, #3985 @yueming-yuan, plus fixes by @fzyzcjy).

Weight updates that tolerate failed cells. --ft-components gains weight_update, which is now part of the default when fault tolerance is on. During a P2P update:

  • A rollout cell whose write fails is blamed, agreed on across trainer ranks and taken out of service. The other cells still move to the new version.
  • Every engine request is time-bounded (--update-weight-engine-request-timeout, default 300 s).
  • A trainer cell whose update never returns (--update-weights-timeout, default 600 s) or that dies mid-update is given up.

--check-weight-transfer-checksum hashes every P2P write on both ends; it is on by default only in CI. Engine checksums are tied to the published weight version and the engine incarnation. Weight-update FT requires Megatron, P2P and no colocate (#3440–#3460, #3833–#3849 @fzyzcjy).

Fully async, colocated. --fully-async --colocate lets engines and trainer take turns on the same GPUs. The driver pauses generation, flushes and offloads, trains, then reloads and updates weights. No samples are produced during the train window, and --max-weight-staleness behaves as before. examples/retool_v2 runs a multi-turn tool-call recipe this way on one GPU. Restrictions:

  • Megatron only.
  • Use --pause-generation-mode retract.
  • No rollout fault tolerance.

See the docs, the retool v2 example and #3375–#3378 @fzyzcjy.

Fully async runs resume where they stopped. Each training step takes one whole batch from the buffer. Stale groups are dropped or recycled across the whole buffer on every read. The buffer, the retry queue and in-flight groups are checkpointed and restored, and generated but untrained outputs are replayed on resume (#3404–#3420 @fzyzcjy).

Prompt KV staleness. Every sample records the weight versions that computed its prompt KV cache. New weight_version/prefill_* metrics include lag and the stale-token ratio. --namespaced-radix-cache partitions SGLang's radix cache by the rollout call a sample started under, and turns on automatically for fully async in-place pausing (#3368–#3374 @fzyzcjy).

Independent DP with uneven survivors. With --indep-dp (turned on by --ft-components train), the run keeps training when the number of live trainer cells does not divide the global batch, for example 256 samples over 3 cells. An end-to-end test kills cells in a 4-cell run (#3430, #3431 @fzyzcjy).

Optimizations

Fused log-prob and entropy. --log-probs-backend fused (opt-in; the default stays torch) computes log-probs and entropy in one Triton kernel, normalizing over the true vocabulary as SGLang does. In a microbenchmark on [65536, 129280] bf16 logits, forward+backward ran 13–16x faster on H200 and B300, and the extra memory dropped from tens of GiB to under 1 MB. End to end, on Qwen3-1.7B GRPO with 30K-token responses on 4x B300 (1 trainer GPU, 32K tokens per GPU):

  • Peak trainer memory fell from 117.3 to 79.0 GiB.
  • Actor-update throughput rose by 20%.
  • Step time fell by 7%, since the step is rollout-bound.

Not compatible with --true-on-policy-mode or sampling-support replay. Megatron needs --vocab-size (#3741 @HJSang).

Session server and agentic rollout. Per-turn work in the session server is cut down:

  • Turn arguments are no longer deep-copied.
  • The backend proxy uses aiohttp, and JSON goes through orjson.
  • /samples replies are decoded as views over one buffer.
  • Large collections are materialized off the event loop.

In CPU benchmarks:

  • 16 sessions x 40 turns went from about 46 to 75 turns/s.
  • The worst event-loop stall for one 256K collection fell from 1.7 s to 0.19 s.

These benchmarks now run in CI (#3963, #3967, #3972–#3975, #4044 @guapisolo, #3736 @nanjiangwill).

Refactors

Rollout-side state and the data source. Rollout-side checkpoints now live in one directory per rollout, {save}/rollout/{id}/, written atomically. That directory holds the data source, the rollout functions, the executor outputs and a snapshot of the event log. DataSource loses add_samples and saves to a directory (#3379, #3397–#3403 @fzyzcjy). Megatron models carry a small CPU-only "model companion" holding the weight version and the record of consumed samples, which makes an opt-in sample-ownership checker possible (#3382–#3396, #3423 @fzyzcjy).

Fault injection through hooks. CI fault injection moves to named hook points in the trainer, controller and orchestrator (--ci-fault-hooks). The FT API server can target a specific process incarnation and refuses stale targets. Soak tests run through one soak runner that records evidence across failures (#3771–#3832 @fzyzcjy).

Full release notes by category below; breaking changes are near the end.

Weight Update

  • P2P transfer through storage-free engine replicas and two reusable pinned buffers; writers assigned from real engine sizes; loader-derived mappings; failing writes fail the update; replicas and registered memory kept across reconnects: #3939, #3940, #3941, #3942, #3943, #3945, #3950, #3959, #3969 @yueming-yuan
  • P2P updates the speculative draft from trained MTP weights: #3985 @yueming-yuan
  • P2P write-path and Mooncake fixes, including honoring the MOONCAKE_PROTOCOL/MOONCAKE_DEVICE settings, keeping registered memory after uncertain completions, cancelling writes queued from the previous connection, and waiting for running writes before releasing buffers: #4168, #4169, #4170, #4185, #4186, #4187, #4196 @fzyzcjy
  • Weight versions stored with the Megatron model state; disk-delta sequencing separated from weight versions; explicit publication versions on the torch-native backends: #3423, #3424, #3425, #4084 @fzyzcjy
  • Disk-delta startup sync publishes the loaded trainer weights; optional weight version on executor eval for already-synchronized eval pools; a rollout-cell restart fault action: #4161 @zianglih
  • Weight checksums: per-write P2P checksums (--check-weight-transfer-checksum), engine checksums bound to version and incarnation, unchanged-weight detection with exclusions, collection before the update window closes, and audit failure after three consecutive misses: #3833, #3834, #3835, #3836, #3844, #3845, #3846, #3847, #3848, #3849, #4175, #4192, #4193, #4194 @fzyzcjy
  • Respect FP8 quantization exclusions (modules_to_not_convert / ignored_layers) during online weight refit; fixes GLM-5.3-Flash's kv_b_proj and indexer weights being quantized: #3900 @kaixih
  • Sync and check draft weights only when the trainer trains MTP: #4043 @guapisolo

Fault Tolerance

  • Weight-update FT across rollout and trainer cells (--ft-components weight_update): blame, agree on, and retire cells that miss an update; time-bounded engine requests; give up on hung or dead trainer cells; send from every alive trainer cell: #3440, #3443, #3446, #3447, #3448, #3449, #3450, #3451, #3454, #3455, #3456, #3457, #3458, #3459, #3460, #4173 @fzyzcjy
  • Independent DP: typed training parallel config, rollout shard storage moved to the caller, and training continues when live cells do not divide the global batch; 4-cell trainer-fault e2e: #3426, #3427, #3428, #3429, #3430, #3431 @fzyzcjy
  • Worker identity and incarnations: pod UID in identity reports, RPC handles pinned to the expected boot UUID, incarnation-guarded fault injection over the API server (GET .../fault-target, HTTP 412 on stale targets), remote exception types preserved in RPC failures: #3771, #3772, #3773, #3774, #3775, #3776, #3777, #3857 @fzyzcjy
  • Fault hooks: --ci-fault-hooks / --ci-fault-hooks-path, hook points in trainer/controller/orchestrator, runtime set/clear over RPC and the API server (POST .../fault-hook), delayed and expiring requests, stop/freeze/deadlock/sleep actions, cross-worker delivery: #3820, #3821, #3822, #3823, #3824, #3825, #3826, #3827, #3828, #3829, #3830, #3831, #3832, #3839, #4172, #4177 @fzyzcjy
  • Soak runner: interfaces for runners, observations and fault actions; scenarios (random crash, rollout-deterministic, gsm8k, hot restart) moved onto it; fsynced evidence, archived training evidence, recovery budgets, pod/process observation, quiescent-only injection, hook-triggered faults by default, coverage-first fault selection: #3782–#3819, #3850–#3856, #4176, #4178–#4184, #4191, #4195, #4197 @fzyzcjy
  • Bitwise fault comparisons: trainer all-gather faults and a receiver killed mid P2P send leave training bitwise intact against a fault-free run; deterministic P2P recipe with the overlap scheduler disabled: #3840, #3841, #3842, #3843 @fzyzcjy
  • Launch plumbing: Ray submission IDs and Helm upgrade timeouts, shared in-cluster/kubeconfig Kubernetes client loading, launch guards around Helm cleanup and upgrades: #3780, #3781, #4171 @fzyzcjy
  • Handle the frozen model companion in the weight checksum and dumper: #4119 @guapisolo, #4174 @fzyzcjy

Rollout & Fully Async

  • --fully-async --colocate, with pause/continue/flush fan-outs on the inference controller and a single-GPU retool_v2 recipe: #3375, #3376, #3377, #3378 @fzyzcjy
  • Fully async buffer: whole batches per step, stale-group recycling over the whole buffer, consumed-only staleness metrics, a no-progress warning, failing fast when the worker ends, own retry buffer, state_dict/load_state_dict, and checkpoint/restore with replay of untrained outputs: #3380, #3404–#3418, #3420, #3421, #3422 @fzyzcjy
  • Prefill KV weight versions on samples, prompt-KV staleness metrics, and a namespaced radix cache (--namespaced-radix-cache): #3366, #3367, #3368, #3369, #3370, #3371, #3372, #3373, #3374 @fzyzcjy
  • Data source and rollout-side checkpoints: add_samples removed, legacy buffer only for partial rollout, one atomic directory per rollout, rollout state loaded only for a trained step, refusing to start fresh when the trainer resumed: #3379, #3381, #3397, #3398, #3399, #3400, #3401, #3402, #3403, #3419, #4082 @fzyzcjy
  • Sample lineage and opt-in sample ownership checks (--enable-sample-ownership-checker, on by default only under --ci-test): model companions, optimizer consumption in checkpoints, issued/dropped sample records, trainer witnesses: #3382–#3396, #4188 @fzyzcjy
  • Rollout executor robustness: rollout functions and save/load on dedicated event loops, loop-local HTTP clients, RPC shutdown cleanup, router worker-removal failures propagated for retry, rollout cleanup before controllers close, aligned LoRA rollout numbering: #4081, #4083, #4085, #4086, #4087, #4189, #4190 @fzyzcjy
  • mask_truncated_zero_reward sample filter (opt-in via --rollout-sample-filter-path): #3920 @lizamd
  • Warn that --over-sampling-batch-size is ignored under --fully-async: #3921 @lizamd
  • Engines launched by Miles keep HTTP connections alive for 60 s by default (SGLANG_TIMEOUT_KEEP_ALIVE), longer than the client's expiry: #3987 @zyzshishui

Agentic & Session

  • Session-server and rollout CPU work: no per-turn deep copies, aiohttp proxy, orjson, view-based /samples decoding, pre-faulted reply buffers, sample materialization off the event loop, and CPU benchmarks in CI: #3963, #3967, #3972, #3973, #3974, #3975, #4044, #4103 @guapisolo, #3736 @nanjiangwill
  • Request session top-k candidates as flat base64 arrays, preserving -inf; older servers fall back to nested rows: #3952 @HJSang
  • swe-agent-harbor example: agent-server pool limits on the transport, engine context capped to the run's sequence budget, restarts resume from save_dir: #3917, #3918, #3919 @lizamd

RL Algorithms & Training

  • Fused Triton log-prob + entropy (--log-probs-backend fused, opt-in): #3741 @HJSang
  • FSDP/torchtitan per-token loss is divided by the step's DP-total token count, matching Megatron; it previously used the token sum, inflating grad norms: #4099 @HJSang
  • Score centering turns on the FP32 rollout LM head by default: #4031 @Shi-Dong
  • --skip-actor-forward-only accepts custom model providers, before-train-step hooks and rollout-data postprocessors: #4114 @guapisolo
  • MTP layers built and trained only with --enable-mtp-training; MTP heads detached by default: #4041 @guapisolo
  • Megatron BSHD routing replay uses the router's token order with micro-batch size ≥ 2: #3146 @guapisolo
  • Resume checkpoint selection: explicit --ckpt-step honored without a tracker file, separate selectors for reference/teacher loads (with radixark/Megatron-LM#129): #2688 @nanjiangwill
  • Shared W&B workers attach without resuming, fixing intermittent 404s: #4021 @guapisolo
  • Resolve the Ray head placement without the dashboard API: #3640 @nanjiangwill

LoRA

  • Native LoRA for Megatron raw mode across registered architectures, merged --save-hf export, --lora-A-init-method, LoRA optimizer resume fixes, Kimi-K2.5 raw mode: #1792 @Zhichenzzz
  • GLM-5.3-Flash LoRA through native LoRA: #3098 @yushengsu-thu

Low Precision

  • DeepSeek-V4 direct-HF initialization with MXFP8 training and trainer-sourced MXFP8 rollout weights (Blackwell): #3135 @bobboli
  • tools/convert_hf_to_fp8.py --extra-high-precision-layers-hf keeps chosen layers unquantized: #1623 @nanjiangwill
  • Respect FP8 quantization exclusions during weight refit: #3900 @kaixih

Models & Backends

  • FSDP: one ArchAdapter per architecture; HF padding-free kwargs for GatedDeltaNet packing; NemotronH reference model no longer leaks state across packed documents: #3935 @HJSang
  • Route the training path's device calls through miles.platforms (groundwork for other accelerators; no behavior change on CUDA): #4209 @HJSang

Dashboard & Metrics

  • Sandboxes page in the dashboard, fed by miles.dashboard.hooks.report_sandbox(...) from agent code: #4150 @Shi-Dong
  • Prompt-KV staleness metrics (weight_version/prefill_*): see Rollout & Fully Async.

AMD / ROCm

ROCm is not part of release CI; the evidence below is from the PR descriptions.

  • ROCm images move to the SGLang v0.5.22 ROCm bases (rocm724-mi35x, rocm10-mi35x, rocm10-mi30x), and the MI350 nightly adds the DeepSeek-V4.1 4-layer test, the DSA indexer test and eight single-GPU kernel test files; the PR reports these passing on MI350X: #4202 @XinyuJiangCMU
  • Import megatron_to_hf lazily in native LoRA, fixing the ROCm image build's import check: #4201 @michaelzhang-ai

Docker & Build

  • Bump to SGLang v0.5.22, torch 2.14 wheels, Transformer Engine 2.17.0+miles, transformers 5.19.0: #4125 @guapisolo

CI & Release Infrastructure

Docs Updates

Dependencies

  • SGLang: sglang-miles locked at 3a938213c1 (v0.1.2: b84bec5761). The Miles patch stack is rebased onto SGLang v0.5.22 (#4125 @guapisolo). New in the stack:

    SGLang v0.5.22 itself returns top logprobs as flat base64 arrays on request (sgl-project/sglang#42890).

  • Megatron-LM: radixark/Megatron-LM@miles-main locked at 2fbe25a442 (v0.1.2: fd15ee20a4). Changes:

    • Explicit checkpoint selection (radixark/Megatron-LM#129).
    • A reshardable DP parameter state that round-trips through its loader (#121).
    • MoE dispatcher routing tensors released after combine (#118).
    • Optimizer master weights and chained-optimizer state staging bounded by the offload chunk (#119, #120).

  • Base image and wheels: lmsysorg/sglang:v0.5.22 (was v0.5.21). Prebuilt wheels move to cu130-torch214-x86_64 / cu130-torch214-aarch64 (were torch213).

  • Transformer Engine: 2.17.0+miles, built from radixark/TransformerEngine miles-main (NVIDIA's release commit plus the fixes Miles carries). This replaces upstream 2.17.0 plus build-time patches. cuDNN now comes from the base image; the separate nvidia-cudnn-cu13==9.22.0.52 install is gone, and nvidia-cudnn-frontend is a >=1.28 floor.

  • Python requirements: transformers==5.19.0 (was 5.12.1, matching SGLang v0.5.22). Adds aiohttp and orjson for the session server. Ray stays >=2.56.0,<2.59.0.

Docker

docker pull radixark/miles:v0.1.3
  • NVIDIA: CUDA 13 only, on linux/amd64 and linux/arm64, built on lmsysorg/sglang:v0.5.22. The manifest digest is sha256:c3736aedfc1aa8e8df828daab7f5de2dcee8cb223d76275055281ec558202bbf. Sources: miles 74f47d37c8, sglang-miles 3a938213c1, Megatron-LM 2fbe25a442.
  • AMD: the ROCm images build on the SGLang v0.5.22 ROCm bases (v0.5.22-rocm724-mi35x, v0.5.22-rocm10-mi35x, v0.5.22-rocm10-mi30x) (#4202 @XinyuJiangCMU).

Breaking Changes & Upgrade Notes

  • v0.1.2 rollout-side checkpoints cannot be resumed.
    • Old: rollout-side state was saved as {save}/rollout/global_dataset_state_dict_{id}.pt, and a missing file only caused a warning.
    • New: state lives in {save}/rollout/{id}/, and resuming a trainer step without that directory fails at startup.
    • Action: finish v0.1.2 runs on v0.1.2, or start a new run. No migration tool exists.
    • Sources: #3398, #3399, #3403 @fzyzcjy.
  • DataSource API.
    • Old: the API had add_samples, save(rollout_id) and load(rollout_id), and the default class was RolloutDataSourceWithBuffer.
    • New: add_samples is removed. save(directory) and load(directory) write inside the given directory. The default class is miles.rollout.data_source.RolloutDataSource. The buffered class is renamed LegacyRolloutDataSourceWithBuffer; it is chosen automatically only for --partial-rollout, and it cannot load a checkpoint.
    • Action: port custom data sources to the new methods. Replace any explicit RolloutDataSourceWithBuffer path. Custom rollouts that called add_samples must keep their own retry queue.
    • Sources: #3379, #3381, #3419 @fzyzcjy.
  • Custom fully async data buffers.
    • Old: get() returned one group.
    • New: get(*, num_groups, ...) returns a whole batch, and checkpointing calls state_dict() / load_state_dict(). The base class raises NotImplementedError for both. --async-data-buffer-capacity-factor must be ≥ 1.0, and --buffer-filter-path is rejected under --fully-async.
    • Action: update --custom-async-data-buffer-path classes.
    • Sources: #3410, #3412, #3415 @fzyzcjy.
  • MTP training is explicit.
    • Old: --mtp-num-layers N built and trained MTP layers, and their gradients reached the policy.
    • New: MTP layers are built only with --enable-mtp-training (Megatron only), and --mtp-detach-heads is on by default. Without the flag, --save-hf exports carry no MTP weights.
    • Action: add --enable-mtp-training to keep training an MTP/EAGLE draft. Use --no-mtp-detach-heads for the old gradient flow.
    • Sources: #4041, #4043 @guapisolo.
  • Fault tolerance needs P2P weight transfer.
    • Old: --use-fault-tolerance worked with broadcast or colocated weight updates, and the default component was rollout.
    • New: the default components are rollout weight_update. Both require --update-weight-transfer-mode p2p without --colocate, and weight_update also requires Megatron.
    • Action: switch to disaggregated P2P, or pass --ft-components train only.
    • Sources: #3447, #3460, #4173 @fzyzcjy.
  • Independent DP batch settings.
    • New: --indep-dp, and therefore --ft-components train, requires --use-dynamic-batch-size with --max-tokens-per-gpu. It rejects --use-dynamic-global-batch-size.
    • Sources: #3430 @fzyzcjy.
  • Disk-delta weight sync on resume.
    • Old: --update-weight-transfer-mode disk-delta with --load pointing at a Megatron training checkpoint silently never pushed the restored weights to the engines.
    • New: this combination is a startup error.
    • Action: start disk-delta runs from an HF checkpoint, or resume with another transfer mode.
    • Sources: #3425 @fzyzcjy.
  • Removed flags (no replacement). --log-probs-max-tokens-per-gpu was never read (#4118 @guapisolo). --p2p-transfer-num-workers is replaced by one write thread per engine (#3941 @yueming-yuan). Action: drop them from launch commands.
  • CI fault-injection interfaces.
    • Old: --ci-ft-test-actions / --ci-ft-test-actions-path, and POST /api/v1/cells/{name}/inject-fault.
    • New: --ci-fault-hooks / --ci-fault-hooks-path, and GET .../fault-target followed by POST .../fault-hook.
    • Action: only scripts that drove these test interfaces need to change.
    • Sources: #3826, #3829 @fzyzcjy.
  • Native LoRA checkpoints.
    • Kimi K3's old megatron_rank_sharded LoRA checkpoints no longer load; its HF adapters still do.
    • GLM-4 native adapters are exported as gate_up_proj.
    • A LoRA resume from a checkpoint without optimizer state now fails unless --no-load-optim is passed.
    • A --lora-adapter-path with no adapter fails.
    • Sources: #1792 @Zhichenzzz.
  • Gradient scale on FSDP/torchtitan with per-token loss.
    • Old: with --calculate-per-token-loss, the gradient was the token sum.
    • New: it is the per-token mean over the step's DP-total tokens.
    • Action: revisit --clip-grad and the learning rate if they were tuned around the old scale.
    • Sources: #4099 @HJSang.
  • Score centering serves an FP32 LM head.
    • New: --loss-type score_centering now enables --sglang-enable-fp32-lm-head, which costs some rollout throughput. Per-engine overrides still take precedence.
    • Sources: #4031 @Shi-Dong.
  • FSDP architecture registries.
    • Old: out-of-tree FSDP patches used the register_* registries.
    • New: those registries are replaced by one ArchAdapter per architecture.
    • Action: port out-of-tree FSDP patches.
    • Sources: #3935 @HJSang.
  • Environment pins.
    • transformers==5.19.0 (was 5.12.1). Transformer Engine is now 2.17.0+miles, and the image no longer pins cuDNN separately.
    • Action: rebuild custom environments against these versions.
    • Sources: #4125 @guapisolo.
  • Session request parsing is stricter. The session server now returns 400 for JSON with NaN/Infinity, a BOM or UTF-16/32 encoding. Malformed base64 routed-expert or indexer buffers now fail session collection with 422 (#3967, #3973 @guapisolo).

Known Issues

  • Trainer-cell fault tolerance with independent DP may fail to recover when a trainer cell dies right before a training step starts.
    • The surviving cell can wait in the cross-cell barrier until torchft's 120 s timeout.
    • Before that, the heartbeat checker stops it at about 60 s, and the run ends with "all alive cells failed".
    • This shows up in two newly enabled soak tests that inject random trainer and rollout faults on dp2/cp2: tests/e2e/ft/test_random_crash__kill_train_rollout__dp2_cp2.py and its fully async twin, tests/e2e/ft/test_random_crash_fully_async__kill_train_rollout__dp2_cp2.py.

All Contributors

@bobboli @fzyzcjy @guapisolo @HJSang @kaixih @lizamd @michaelzhang-ai @nanjiangwill @Rockdu @Shi-Dong @XinyuJiangCMU @yueming-yuan @yushengsu-thu @Zhichenzzz @zhihengy @zianglih @zyzshishui

Full Changelog: v0.1.2...v0.1.3

Contributors

kaixih, fzyzcjy, and 15 other contributors
Assets 2
Loading