v0.5.21

Release Highlights

• Dynamo rollout token metadata. ART now reads Dynamo's nvext.engine_data alongside vLLM's token metadata, and trainable models' openai_client() requests it automatically. ServerlessBackend users whose rollouts are served by W&B Inference no longer hit Trainable Choice is missing vLLM prompt_token_ids/token_ids (#934).
• RL training stability. Zero-variance GRPO updates are skipped (#1118), policy importance ratios are bounded before exponentiation (#1113), TIS is skipped when original logprobs are unavailable (#1116), and several logprob and gradient paths now use FP32 (#1105, #1112, #1117).
• Trainer. Versioned trainer graphs and logical-rank APIs (#925), with the pre-#925 trainer execution restored behind API-name aliases (#1107); many trainer-rank memory planning fixes; Megatron SFT packing with shared prefix trees (#905); compact logprobs and token-aligned expert routing (#1050).
• Models. Qwen 3.5 9B Megatron support (#896) and Qwen3.5 adapter loading fixes (#959).
• Docs moved to Astro (#1129).

What's Changed

• fix: drain full trajectory batches after completion bursts (#1124)
• Migrate docs to Astro and deploy main to CoreWeave (#1129)
• fix: preserve FP32 target gradients in Megatron TE cross entropy (#1117)
• fix: use population std for native Tinker advantages (#1115)
• fix: isolate legacy Tinker branch contexts (#1114)
• fix: skip TIS when original logprobs are unavailable (#1116)
• fix: bound policy importance ratios before exponentiation (#1113)
• fix: stabilize Unsloth logprobs in fp32 (#1112)
• fix: skip zero-variance GRPO updates (#1118)
• fix(trainer-rank): keep head row normalization outside the statistics checkpoint (#1120)
• revert: restore pre-#925 trainer execution with API-name aliases (#1107)
• ci: pin uv to 0.12.24 so Prek reuses its prebuilt cache (#1119)
• fix(trainer-rank): combine cross-entropy target and softmax gradients in FP32 (#1105)
• fix: remove unused datrie from tinker dependencies (#1104)
• fix(trainer-rank): honor oversized model output placement (#1099)
• perf: avoid numeric type tuples in compact decoding (#1097)
• perf: reuse alignment across chat history masks (#1098)
• test: separate builder expiry from signal escalation (#1095)
• fix: bound image builder lifetime and request resources (#1094)
• fix(ci): verify prospective manual GPU validation reuse (#1093)
• fix(trainer-rank): retain HybridEP planner replay dimensions (#1088)
• fix(megatron): skip TE grouped GEMM when every group is empty (#1089)
• fix(trainer-rank): price TP2/TP4 sequence-parallel recompute term by term (#1075)
• fix(trainer-rank): preserve graph pricing in oversized fallbacks (#1086)
• feat(tau): expose bounded transport lifecycle observations (#1085)
• fix(ci): bound GPU capacity waits and retain cleanup receipts (#1043)
• fix(trainer): require coordinated EP oversized admission (#1084)
• fix(trainer-rank): credit staged adapter gradients and shared captures in graph admission (#1081)
• fix(trainer): remember reloadable immutable checkpoint sources (#1077)
• fix(ci): initialize image cleanup path on the runner (#1082)
• fix(trainer-rank): reserve eager head workspaces in admission and replay (#1068)
• fix(trainer-rank): preserve dense no-grad floors beside adapter backward peaks (#1067)
• fix(trainer): honor oversized admission for supported EP plans (#1080)
• fix(ci): retain image cleanup receipts and surface failures (#1079)
• feat: add versioned trainer graphs and logical-rank APIs (#925)
• perf(checkpoints): read snapshot components once per block (#1078)
• fix: make live trajectory compaction explicit (#1076)
• test(trainer-rank): pin packing fingerprint inputs with golden digests (#1074)
• fix: preserve process-pool readiness and lifetime ownership (#1073)
• feat(trainer-rank): expose exact prefix packing and input commitments in telemetry (#1072)
• fix(trainer-rank): rank-local memory admission for asymmetric DP batches (#1071)
• fix: restrict serial-tool prefix certificates by producer policy (#989)
• fix(megatron): give MoE methods separate Dynamo caches (#1056)
• docs: document the fixed LoRA alpha rationale (#1027)
• fix(inference): preserve streams when history observation cannot render (#1026)
• fix(ci): match normalized image repositories when resolving the pushed digest (#1069)
• perf: validate prefix digests with a compiled ASCII-hex pattern (#933)
• fix(tau): remove implicit active HTTP connection limits (#1066)
• fix(trainer-rank): account for pending adapter gradients during recompute (#1002)
• Pass CPU sequence boundaries into packed GDN (#1065)
• Allow bounded owners to set Tau environment idle timeout (#1064)
• Speed up bounded JSON encoding for planner reports (#1057)
• Clarify Dynamo scope in trainer compile telemetry (#1062)
• fix: Preserve bounded planner coverage after report exhaustion (#1063)
• Only trust checkpoint code for models that require it (#1060)
• fix(tinker): bound server workers and reserve listening ports (#1059)
• feat: preserve compact prompt metadata across protocols (#1054)
• chore(deps): update Tinker SDK to 0.31.0
• Preserve BF16 SiLU rounding in fused GDN convolution (#1055)
• Add compact logprobs and token-aligned expert routing (#1050)
• fix: release GDN LoRA components before output addition (#1048)
• Release terminal trajectory capture owners (#1052)
• Avoid canonical re-encoding for freshly generated planner reports (#1051)
• Skip planner report construction only after definitive budget exhaustion (#1049)
• Skip body parsing for ineligible trajectory capture (#1047)
• Allow a configured free Kubernetes context for required GPU CI (#1045)
• Snapshot physical LoRA state before expanding checkpoint payloads (#1041)
• Release direct HF source tensors after their last logical alias (#1038)
• Reuse owned JSON containers during compact decoding (#1042)
• Defer LoRA packing until snapshot capture or transport owns it (#1039)
• Build missing checkpoint moments from captured CPU shards (#1037)
• Release the trainer queue after immutable checkpoint capture (#1035)
• Skip prefix-tree construction for single-history masks (#1036)
• Allow explicit free Kubernetes context for GPU CI (#1034)
• Avoid allocating nodes for existing token prefixes (#1033)
• Recompute grouped planner reports from bounded runtime facts (#1028)
• Preserve planner reports when tensor nesting exceeds encoding limits (#1032)
• Bound planner diagnostic tensor list structure (#1030)
• Distinguish planner retention exhaustion from persistence failures (#1025)
• Preserve bounded CPU inference inputs in planner diagnostics (#1020)
• Keep nested event loop patches out of imports (#1023)
• Preserve recorded conditioning by default during tokenization (#1014)
• Move TrainerRank checkpoint-slot and dynamic-optimizer management into trainer_rank/_slots and _optimizer (#1001)
• Cache bounded immutable chat template normalization results (#1017)
• Fix same-head TrainerRank GPU reuse (#1016)
• Move TrainerRank micro-batch planning into trainer_rank/_micro_batch_planner (#1000)
• Move TrainerRank memory accounting into trainer_rank/_memory (#999)
• Preserve literal reasoning tags in assistant history (#1009)
• Preserve string subclasses during string interning (#1013)
• Use identity for exact serialization type checks (#1012)
• Release source rebinding closures after tokenization (#1011)
• Reuse checkpoint memory floor within native planning cost (#1008)
• Reuse exact GPU validation on draft promotion (#1007)
• perf: Skip scalar recursion in compact trajectory numeric lists (#1006)
• Reuse GDN parent row indices across paired state tensors (#1005)
• Decompose _tokenize_chat_view into a staged _ChatViewTokenizer (#997)
• Add a byte-exact golden regression harness for trajectory tokenization (#996)
• Preserve Tau deletion cleanup through repeated parent cancellation (#994)
• Avoid repeated MoE workspace scans for GDN layer floors (#998)
• Price warm waves by the fit later plans measured, not the first plan's (#991)
• Preserve running GPU image adoption (#993)
• Validate canonical checkpoint optimizer counters (#992)
• Price HybridEP recompute outputs at the live communication extent (#971)
• Clean up cancelled GPU image prewarm pods (#964)
• Price TP4 sequence-parallel boundary shards in the checkpoint floor (#988)
• Release HybridEP dispatcher state after combine (#975)
• Reuse exact chat-template prefixes during long-history tokenization (#960)
• Price the shared-expert return under context and expert parallelism (#966)
• Load Qwen3.5 adapters with the pinned model's attention shape (#959)
• Preserve literal think markers in thinking-off histories (#967)
• Collect dynamo's cyclic garbage after compiling a checkpointed layer (#955)
• Fix Responses history observation after tool argument normalization (#962)
• Run the MoE planner memory tests in the Megatron CI lane (#958)
• Price HybridEP expert work and buffer growth at EP>1 (#954)
• Price trainer-rank memory profiles by packed tokens, not sharing ratio (#950)
• Apply trainer-rank memory floors per context-parallel rank (#951)
• Preserve contained sampled-part bounds during history correction (#943)
• Enforce cumulative execution allowances for planner reports (#946)
• Account for grouped checkpoint, head, and enclosing forward buffers (#898)
• Preserve planner refusal attribution and compact report context (#945)
• Budget physical-memory recovery at backward handoff (#888)
• Preserve supplied Responses fields in saved trajectories (#938)
• Defer Tinker backend imports for inference clients (#937)
• Retain scalar planner admission and failure evidence (#942)
• Use Qwen 3.8 27B and ART main in serverless examples (#941)
• Freeze request options for delayed planner reports (#940)
• Add opt-in planner admission override and reproducible miss reports (#939)
• Support native Dynamo rollout token metadata alongside vLLM (#934)
• Release prior vocabulary-head chunk references before the next projection (#929)
• Calibrate unseen TrainerRank signatures before reusing batch width (#927)
• Render structured tool-call arguments in append-only observer (#928)
• fix: Preserve native token boundaries during trajectory replay (#924)
• Bound native CUDA cache recovery by measured forward work (#900)
• chore: Account for dense CP rank loads in retained-activation memory (#922)
• Clean up cancelled GPU image builders (#923)
• Use restricted workload identity for GPU image smoke launches (#921)
• Model retained activations and allow selective recompute in TrainerRank (#915)
• fix: Broadcast native FLA CP chain state from the group's global owner rank (#920)
• Reject unsupported selective recompute in TrainerRank planning (#914)
• Return full source-order context-parallel trainer outputs (#912)
• Preserve proven sampled boundaries after context text matches (#910)
• Pack Megatron SFT batches with shared prefix trees (#905)
• Centralize tokenizer defaults and preserve sampled inference history (#909)
• fix(trainer-rank): coordinate pure planning failures across ranks (#907)
• Preserve group types and whitespace boundaries in trajectory tensorization (#904)
• Preserve NaN log probabilities in trajectory JSON (#903)
• Refresh final TrainerRank admission and propagate counter failures (#899)
• Register Qwen 3.5 9B Megatron support (#896)
• Exclude pending native allocator bytes from reusable credit (#895)
• trainer_rank: retain completed split-forward memory peaks (#894)
• trainer_rank: preserve memory estimates and account for overlapping expert buffers (#893)
• GDN planner: price the dense projection work per GDN-owned token; refit the chain overheads and gate (#884)
• fix: Price the known eager MoE FC2 input/output envelope (#891)
• Keep first recorded MoE routes on token overlaps (#890)
• fix(trainer-rank): keep retained checks paired with their layout (#889)
• fix: Bind sampled source identity to captured conditioning (#886)
• Preserve proved boundaries in mixed-stop captured histories (#882)
• fix: exclude inactive requests from memory profile ratios (#880)
• perf: Speed up assistant span prefix comparisons (#887)
• Decouple TrainerRank CI results from attached log transport (#879)
• fix: Fix trainer-rank memory estimates and split pruning at profile boundaries (#877)
• Release completed microbatch outputs and preserve admission evidence (#872)

Full Changelog: v0.5.20...v0.5.21