Skip to content

Halo 1.3.0rc1: faster GRPO at scale, score centering and FP8 rollouts

Pre-release
Pre-release

Choose a tag to compare

@hivaze hivaze released this 08 Nov 22:07
7292e89

Halo 1.3.0rc1 is a release candidate for faster reinforcement learning at scale. On synchronous GRPO with Qwen3.5-35B-A3B on two 8× B300 nodes, a step takes 0.137 s per 1k response tokens against Miles v0.1.3's 0.176, and 0.102 against 0.111 with rollouts overlapped. The gain comes from packing rows into token-budget micro-batches, taking importance weights from the loss forward instead of a separate no-grad pass, and syncing weights to several rollout servers through one group. The release also adds score centering for environmental GRPO on SGLang, rollouts on block-FP8 weights, GRPO training of GLM-5.3-Flash, adapter-only weight sync to SGLang for LoRA policies, striped gathered expert-parallel (EP) saves, and adapter merges that load no model. It lifts a DeepEP dispatch limit that the installed DeepEP no longer has, and moves the Qwen3.5/3.6 recipes to FlashAttention-2 with packing.

Startup now refuses several setups that used to fail hours into a run or train silently wrong. The halo package also regroups its top-level modules under halo/models, halo/environments and halo/trainers into subpackages, so imports of the old paths break, and num_rollout_workers is now rollout_actor_budget. This is a pre-release: the moving image tags stay on 1.2.0 until a final 1.3.0.

MIGRATION-1.3.0rc1.md, attached to this release, maps every moved module and every changed config key.

Highlights

  • GRPO against Miles. Qwen3.5-35B-A3B on GSM8K, 32 prompts × 8 rollouts, one 8× B300 trainer node and one SGLang node, three alternated runs each with prompts identical token for token, in seconds per 1k response tokens. On-policy (steps 3-12) Halo takes 0.137 and Miles v0.1.3 0.176, a step of 14.0 against 18.6 s, with the trainer node peaking at 63 against 109 GiB. With rollouts overlapped (steps 4-14) Halo takes 0.102 and Miles 0.111 (GRPO against Miles).
  • Packing. On the same setup with Halo alone, three alternated runs, the trainer phase and the whole step in seconds per 1k response tokens read 0.034 and 0.163 at 8 rows per micro-batch, 0.018 and 0.141 packed, and 0.014 and 0.143 packed with loss-forward weights. The trainer phase falls by more than half while the whole step moves far less, because generation and sync dominate it.
  • Recipes on short rounds. Qwen3.6-35B-A3B, 4 turns × 1,024 tokens, react_math on GSM8K, one run, steps 2-3, whole step with sync, from SDPA to FA2 with packing: environmental full fine-tuning at ep4 from 23.9 to 10.1 s, environmental LoRA on SGLang from 18.0 to 8.6 s, and an online smoke run from 6.8 to 4.3 s. With the code-contests recipe's own objective settings, FA2 takes 21.2 s and packed 16.7 s (steps 2-4, evaluation pass removed, noisy).
  • Loss-forward weights in online GRPO. On Qwen3.5-35B-A3B under FSDP2, one run per arm, steps 3-5, the 1.7 s no-grad phase disappears and the whole step falls from 23.6 to 22.3 s (-5.6%).
  • Weight sync through one group. Total push time against one server, with rollout_sync_topology: single_group, is 1.27× faster for 4 vLLM servers, and for 8 servers across 2 nodes 1.27× on vLLM and 1.76× on SGLang.
  • Long rows. Update pass, seconds per 1k loss tokens at 16k, 64k and 128k tokens: Qwen3-8B on FA2 0.261, 0.581 and 1.010, and on FA4 at 64k and 128k 0.270 and 0.397; Qwen3.5-35B-A3B 0.193, 0.260 and 0.399, with peaks of 78 and 81 GiB at 128k.

What is new

GRPO packing and importance weights. packed_microbatch_tokens (off by default) plans each round into token-budget bins and runs every bin as one varlen call, rescaling row-mean losses so a window sums to the per-row loss. M-RoPE models are admitted, and a GPU gate scores rows alone and packed for each decoder family. loss_forward_importance_weights (off by default) skips the separate no-grad pass on aligned one-step rounds and raises at construction where something reads the weights before the loss. Online GRPO packs through the same code and supports every vllm_importance_sampling_mode. balance_rank_tokens deals online rows to ranks by attended tokens, and SDPG accepts it too. SDPG also projects its teacher through the chunked head, with no [rows, completion, vocab] plane under ZeRO-2 or ZeRO-3. log_phase_times adds per-phase step timers.

Weight sync. rollout_sync_topology: single_group broadcasts each chunk once to the workers of every server, while per_server stays the default, and a server that sees one GPU per container gets an error that names the remedy. Chunks are staged once, expert layers are gathered only to the sending rank, every bounded wait logs its server, and startup verdicts join over the c10d store. rollout_adapter_sync pushes the frozen base to SGLang once and after that only the LoRA adapter. Episodes keep the adapter version they started with and route around a restarted server.

Score centering. For environmental GRPO on SGLang (arXiv 2609.20807), score_centering_top_k records the sampler's top-K candidates per token and centers each token's gradient on them, with score_centering_weights set to none, tis or mis. The candidate head matches a dense fp32 reference at vocabulary 248k (log-prob error below 4e-6) at the speed of the [n] path. End to end on SGLang, one run per arm, steps 2-6, the whole step takes 14.3 s off and 14.1 s on, rollout tokens/s stay within 1%, and the correction is 1-3e-5 on a bf16 sampler and 5e-5 to 1.5e-4 on an FP8 sampler. vLLM is refused, because returning candidates cost 65% of its rollout throughput at K=16.

FP8-weight rollouts. rollout_weight_precision: fp8_block serves a block-FP8 checkpoint made by scripts/before_training/export_block_fp8.py and quantizes on push in batched calls per expert bank, for Qwen3.5/3.6 MoE only. A push is bit-exact against a fresh boot of the export on vLLM and SGLang. On two servers, one run per arm, steps 2-6, sync is slower than bf16 (5.6 against 2.7 s on vLLM, 7.8 against 2.2 s on SGLang) and so is the whole step (35.9 against 33.5 s on vLLM, +7%, and 39.7 against 33.5 s on SGLang, +19%). The value is engine memory and SGLang decode, which is 26% faster at 256 × 1,024 on one server. SGLang needs SGLANG_ENABLE_JIT_DEEPGEMM=0, which the compose file now passes through, and the preflight now refuses any quantized server a run did not declare.

Long trajectories. The DeepEP wire-index dispatch ceiling is removed. The installed DeepEP V2 addresses its buffers with 64-bit offsets, so a row has no dispatch ceiling within one NVLink domain, and the cross-domain Gin ceiling stays. Dispatch and combine are bit-exact past the old guard at 262,144 tokens per rank on ep2, and a 140k-token row per rank trains on ep2 under both backends with losses equal to ep1 within bf16 (elastic peak 219.8 GiB against ep1's 216.0). routing_replay: recompute keeps captured routing ragged on the host, which takes the update-pass peak at 24 rows of 64k tokens from +2.03 to +0.16 GiB. The DeepEP arena takes 1.5× headroom so a rising high-water mark stops rebuilding it mid-step (long trajectories).

RL at scale. TP and ETP followers skip rollout collection, and Ray actor pools are per node. New startup checks cover an unsharded KL reference that will not fit, the cross-node dispatch token budget, free disk for a checkpoint rotation and engine turns slower than request_timeout, and a run halts when its episode error rate stays over 0.5 for 3 rounds. An engine routing mask that names an expert twice in a token's top-k is refused before dispatch.

GLM-5.3-Flash under GRPO. The family syncs and saves experts one by one, as its hub checkpoint and both engines name them, and unfuse_moe_experts.py converts older checkpoints. After the first push the trainer scores one row against the engine and warns when the gap passes the trainer-versus-engine floor.

Checkpoints at scale. Gathered EP saves are striped over one writer per node (HALO_CHECKPOINT_WRITERS_PER_NODE) on a shared filesystem, with the same tensors, keys and index as one writer, and a plain save writes weights on each node's save rank when nodes have their own disk. merge_peft_adapters.py folds an adapter into the base shards tensor by tensor with no model load, native expert LoRA included.

Reasoning profiles. Each level in reasoning_effort_profiles sets its own reasoning_budget and reasoning_price, and the budget and the price now count reasoning tokens of a whole episode.

Fixes and smaller changes.

  • balance_token_mass weighs a cut turn's negative mass with the other negatives, so a balanced round nets to zero. Before, it left that mass unopposed and pushed down harder than no balance.
  • Logged tokens/s and MFU count attended tokens for SMPO, embedding and PP, and EfficiencyCallback logs data-wait time.
  • A chat template whose assistant turns end in no terminator is refused on padded rows, and wrapped packing keeps documents apart. padding_free is allowed on M-RoPE text models.
  • The training images pin nvidia-cublas==13.8.0.4, because 13.1.0.3 drops trailing rows of some bf16 matmuls (pytorch/pytorch#203481). 0 of 135 matmul reproducers are wrong on 13.8.0.4 against 14 on 13.1.0.3, and SFT throughput is unchanged.
  • The benchmark docs add the SFT comparison against TRL 1.17.1, NeMo AutoModel 0.7.0 and Megatron Bridge, and the GRPO comparison against Miles.

Images

Pull public.ecr.aws/whitecircle/halo:blackwell-1.3.0rc1 or :hopper-1.3.0rc1, and the rollout images :vllm-0.32.0-1.3.0rc1 and :sglang-0.5.22-1.3.0rc1. The moving tags (:blackwell, :hopper, :vllm-0.32.0, :sglang-0.5.22) stay on 1.2.0 until a final 1.3.0, so a release candidate is only ever pulled by its pinned tag. The relocked uv.lock and the cuBLAS pin need rebuilt blackwell and hopper images. Update the rollout servers together with the trainer, because every client refuses a server that does not report the toolkit's image. All four images are built from 7292e894dc0816c792c6234b89c91dabe059a8fe.

Pinned digests:

  • blackwell-1.3.0rc1: sha256:32fe426f2842439947c4d571542019283992dc05f80c430d79012749677a2b33
  • hopper-1.3.0rc1: sha256:2f429b0c33aa1839bf3c6d72ae60d30abb90e882463c6ff4ba128b72cb202927
  • vllm-0.32.0-1.3.0rc1: sha256:4510fff0d3ce2b92c1f7f347e07a79867a5559cf91f9bfabfbfe1f9ad4499b63
  • sglang-0.5.22-1.3.0rc1: sha256:9741585a459eab60a2023b9da3cdc79cd2e4c2bf120a53ca564a1483ae1718c7

Online and async GRPO on Qwen3.8-Flash-Next stay refused on both engines. SGLang 0.5.22 keeps a padded copy of a hyper-connection weight that an update does not reach on B200 and B300, so syncing that family to SGLang needs a later SGLang image than this release's.

Breaking changes

MIGRATION-1.3.0rc1.md lists every moved path and key with its replacement.

  • Module paths. Top-level modules of halo/models, halo/environments and halo/trainers moved into subpackages, with no shims and no code changes. For example halo.models.config_access is now halo.models.introspection.config_access, halo.trainers.world_metrics is halo.trainers.metrics.world_metrics, halo.environments.episode is halo.environments.rollouts.episode_loop, and halo.models.nemotron_h is halo.models.families.nemotron_h. Forty-one modules moved in all.
  • num_rollout_workers. It becomes rollout_actor_budget, a per-node budget (default 256, cluster-wide with ray_address), and the old key raises with the conversion: without ray_address, multiply the old value by the node's rollout ranks; with it, keep the value.
  • Reasoning keys. The per-level keys thinking_tokens and enforce_thinking_cap raise as unknown keys that list the allowed reasoning_budget and reasoning_price. The top-level reasoning_price and reasoning_price_past_budget are gone too: the price is set per level in reasoning_effort_profiles, and it always counts only the reasoning past the level's budget.
  • Changed defaults. Gathered EP saves are striped, the DeepEP arena allocates 1.5×, logged tokens/s and MFU count attended tokens, the startup checks are on, the rollout concurrency cap is one wave of the round instead of 4× the actors, a run whose episode error rate stays over 0.5 for 3 rounds halts, the parity probe warns after the first push, and a quantized server is refused unless declared. The Qwen3.5/3.6 recipes use FA2 and packing. Everything else is opt-in.

Validation

The CPU tier passes: 36,581 tests across its 12 shards, with the two tests that need the network and three sandbox-timeout tests passing when run alone. On the release images, SFT, LoRA, QLoRA and gpt-oss EP2 smoke runs end within 0.003 of the 1.2.0 images' losses, and 15 core GRPO rows and the single-GPU vLLM and SGLang rows pass. The GPU rows quoted above ran on the commits they name, and make test-gpu-core was not run on the final head of this release candidate.

Known issues

Most of 1.2.0's known issues still apply, among them:

  • With more than one EP group, a LoRA adapter or an added embedding row starts identical across groups only when every rank is seeded alike, as the training scripts seed it.
  • PP's logged rate counts every padded row at max_length, so it reads above the attended rate.
  • A full fine-tune that does not fit is not refused up front, so GLM-5.3-Flash at ep8 with data parallelism runs out of memory about 3 minutes in.
  • An MTP head is not carried by online weight sync, and SGLang starts on a checkpoint with no head and misses every draft.
  • Qwen3.8-Flash-Next is experimental, with exact resume checked only on a four-layer cut.
  • FlashAttention-4 4.0.0b37 is a beta, and a pinned FA4 on Gemma 2 in the padded-batch scripts is refused only when the first padded batch reaches it, so leave attn_implementation unset there.
  • A text SFT run on the packed or padding-free routes whose chat template ends assistant turns with no terminator is not refused. With train_on_completions_only, set eos_token to the token that closes a turn.
  • FLA's Triton kernels are autotuned per TRITON_CACHE_DIR, so runs that must match bit for bit should share a warm cache.
  • SGLang 0.5.22 fails to start a Gemma 4 server with --enable-return-routed-experts, so leave that flag off for it.
  • With PEFT 0.18.1, AutoModelForCausalLM.from_pretrained on a directory that holds adapter_config.json raises ModuleNotFoundError, so load adapters with PeftModel.from_pretrained.
  • Every rank still contacts the Hub at startup for a text model's tokenizer, so at large world sizes download the model first and set HF_HUB_OFFLINE=1.
  • A python tool with no sandbox= runs in-process, so a call that times out returns while its thread keeps running until the process exits.
  • Trainer and engine disagree on identical weights through bf16 routing flips, which is not a sync error (trainer and engine noise).

Full changelog: v1.2.0...v1.3.0rc1