Skip to content

v0.3.0

Latest
Latest

Choose a tag to compare

View all tags
@acisseJZhong acisseJZhong released this 03 Sep 03:51
v0.3.0
086bf6c

What's changed

Compared with v0.2.2, TorchTitan 0.3.0 adds typed Python configuration, expands model and declarative parallelism support, and enhances MoE execution, graph-native training, training efficiency, SFT, and RL.

Configuration architecture and extensibility

Recipes now support Python inheritance and typed CLI overrides. An opt-in override registry also lets external packages replace selected model or training components without modifying TorchTitan core.

Key PRs:

  • Python configuration refactor (#2386)
  • External override registry (#3396)

Models and architecture

This release adds Qwen3.5 with variable-length attention, Kimi K2.7 with DistributedMuon, Muse Glimmer 30B, DeepSeek V3 MTP and fused MLA, and Flux compilation with MXFP8.

Key PRs:

  • Qwen3.5 (#3371)
  • Qwen3.5 variable-length attention (#3801)
  • Kimi K2.7 (#3532)
  • Kimi K2.7 DistributedMuon overlap (#4060)
  • Muse Glimmer 30B (#4106)
  • Muse Glimmer FA3 and vLLM support (#4126)
  • DeepSeek V3 MTP (#3392)
  • DeepSeek V3 fused MLA (#4134)
  • Flux compile and MXFP8 (#2579)

Declarative SPMD

spmd_types is now the default SPMD backend, providing declarative state and activation sharding across Llama 3, Qwen3, DeepSeek V3, GPT-OSS, Flux, Qwen3.5, Kimi K2.7, Muse Glimmer, and the Transformers modeling backend.

Some GraphTrainer configurations continue to use the legacy partial_dtensor backend, where model-parallel axes use DTensor while FSDP is managed separately.

Read the SPMD types integration devlog for more details.

Key PRs:

MoE execution and expert parallelism

MoE is reorganized around a unified token-dispatcher interface that separates routing, token movement, and expert computation. Standard, MinimalAsyncEP, DeepEP, and HybridEP backends support communication overlap, CUDA graphs, and composition with TP, CP, PP, activation checkpointing, and compilation.

Key PRs:

  • Token dispatcher (#2842)
  • Dispatcher decomposition (#3389)
  • Unified dispatcher API (#3970)
  • SPMD sparse-mesh transitions (#3654)
  • MinimalAsyncEP (#3561)
  • MinimalAsyncEP composition (#4089)
  • HybridEP compilation (#3360)
  • DeepEP v2 (#3808)

GraphTrainer and GraphPP

GraphTrainer captures the forward, loss, and backward step as one graph that can be transformed, precompiled once, and reused across ranks. Its graph passes manage activation memory and overlap FSDP or expert-parallel communication with computation.

GraphPP partitions this graph into pipeline stages and supports schedules including DualPipeV.

Key PRs:

  • Whole-step tracing (#2561)
  • Rank-agnostic precompilation (#2713)
  • Activation-memory framework (#3118)
  • FSDP communication overlap (#3156)
  • EP communication overlap (#3328)
  • CUDA Graph execution (#2868)
  • GraphPP stage partitioning (#2726)
  • Backward splitting (#2727)
  • FSDP collective splitting (#3088)
  • GraphPP runner (#3089)
  • DualPipeV validation (#3090)

Precision, memory, and training efficiency

BF16 optimizer states, MXFP8, Float8, and NVFP4 reduce memory use and enable lower-precision training. Composable activation-checkpointing policies reduce activation memory, while Trainer CUDA graphs reduce CPU launch overhead.

Key PRs:

Supervised fine-tuning

ChatDataset and a new SFT recipe support training on chat-formatted conversations.

Key PR: ChatDataset and SFT (#2556)

TitanRL

Experimental TitanRL workflows add asynchronous and multi-turn rollouts, continuous batching, multi-generator routing, overlapped weight synchronization, checkpoint and resume support, and Search-R1 and DAPO recipes.

TitanRL requires additional dependencies. Follow the TitanRL quick-start guide before running these workflows.

Key PRs:

  • Async RL loop (#3642)
  • Continuous batching and multi-turn rollouts (#3593)
  • Overlapped weight sync (#3810)
  • Checkpoint and resume (#3751)
  • Search-R1 (#3602)
  • DAPO (#3951)

Compatibility

Component Validated version
Python 3.11, 3.12
PyTorch 2.14.0
torchvision 0.29.0
torchao 0.18.0

Installation

uv pip install torchtitan==0.3.0
Assets 2
Loading