Skip to content

v2.56.0: leftover-DP packed expert shards and router topk under FSDP checkpointing

Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 12 Oct 16:09
· 2 commits to main since this release
v2.56.0
72b2153

Fixes

  • leftover-DP packed expert shards and router topk under FSDP checkpointing

    Packed expert safetensors copies now keep this rank's leftover data-parallel rows when the checkpoint slice is still the expert-parallel block. Indexed expert stacks already did that; 3-D packed slices did not, so leftover-DP loads failed with a destination/source shape mismatch.

    Activation checkpointing on blocks that hold packed experts saves router topk (and device-to-host copies) so recompute cannot change expert assignment or all-to-all splits.

    GRPO learn-phase timing tests advance a fake clock for the no-grad forward delay so the charge does not depend on wall sleep.

Full Changelog: v2.55.0...v2.56.0

Assets 4
Loading