Skip to content

v4.6.0 - segmented checkpointing, multi-gpu training accuracy fixes, example training configs updated

Choose a tag to compare

View all tags
@bghira bghira released this 03 Aug 21:34
· 400 commits to release since this release
v4.6.0
c0da1e5
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Features

  • Segmented checkpointing
    • When gradient checkpointing is enabled, you lose half your speed, or more
    • Segmented checkpointing doesn't save as much VRAM as full layer checkpointing, but kills your speed far less
    • When combined with regional compile, the segments really show some good wall clock savings
  • Attention activation offload
    • Moves activations to CPU instead of requiring gradient checkpointing to be enabled in every scenario
    • Sometimes, transferring the activations is cheaper than recomputing them from checkpoints
      • High batch sizes on fat models meet this criteria
    • Can be used with full layer or torch-ffn/unsloth-ffn feed forward gradient checkpointing as well as interval and stride based configurations (segmented checkpointing)
  • Updated all training configuration examples
    • Verified all of the example run and train a valid result on L40S and H100

What's Changed

  • segmented checkpointing for better cache hits / latency by @bghira in #2920
  • Support Hugging Face Hub attention kernels by @bghira in #2924
  • Fix distributed padding and preserve padded occurrences by @hjinnkim in #2897
  • Normalize configured API base URLs by @cupkk in #2919
  • Map HunyuanVideo flavours to Diffusers repos by @bghira in #2928
  • Improve MageFlow loading defaults by @bghira in #2929
  • Add Wan S2V example config by @bghira in #2983
  • Add SanaVideo example config by @bghira in #2982
  • Add LTXVideo 0.9.5 example config by @bghira in #2981
  • Add LongCat Video example config by @bghira in #2980
  • Update Wan example preset by @bghira in #2975
  • Add Anima example config by @bghira in #2976
  • Add Kolors example config by @bghira in #2978
  • Add SD1x DreamShaper example config by @bghira in #2979
  • Add DeepFloyd IF example config by @bghira in #2977
  • Update PixArt example preset by @bghira in #2974
  • Update OmniGen example preset by @bghira in #2973
  • Update LTXVideo2 example preset by @bghira in #2972
  • Update LongCat Image example preset by @bghira in #2971
  • Update Kandinsky5 Video example preset by @bghira in #2970
  • Update Kandinsky5 Image example presets by @bghira in #2969
  • Update HiDream example preset by @bghira in #2968
  • Update HeartMula example preset by @bghira in #2967
  • Update Cosmos example preset by @bghira in #2966
  • Update Stable Cascade example preset by @bghira in #2965
  • Update ACE-Step example preset by @bghira in #2964
  • Improve audio validation metadata handling (ACE-Step, HeartMuLa, OmniGen and Wan S2V) by @bghira in #2926
  • Validate HiDream mixed precision settings by @bghira in #2927
  • Add Wan latent I2V conditioning by @bghira in #2931
  • Fix LongCat pipeline compatibility by @bghira in #2963
  • Fix Kandinsky5 pipeline compatibility by @bghira in #2962
  • Add Qwen text encoder path override by @bghira in #2985
  • Add segmented checkpointing base controls by @bghira in #2932
  • Add ACE-Step segmented checkpointing support by @bghira in #2933
  • Add AuraFlow segmented checkpointing support by @bghira in #2934
  • Add Boogu Image segmented checkpointing support by @bghira in #2935
  • Add Chroma segmented checkpointing support by @bghira in #2936
  • Add Cosmos segmented checkpointing support by @bghira in #2937
  • Add Cosmos 3 segmented checkpointing support by @bghira in #2938
  • Add ERNIE segmented checkpointing support by @bghira in #2939
  • Add Flux segmented checkpointing support by @bghira in #2940
  • Add Flux.2 segmented checkpointing support by @bghira in #2941
  • Add HiDream segmented checkpointing support by @bghira in #2942
  • Add HunyuanVideo segmented checkpointing support by @bghira in #2943
  • Add Ideogram segmented checkpointing support by @bghira in #2944
  • Add Kandinsky 5 segmented checkpointing support by @bghira in #2945
  • Add Kolors ControlNet checkpointing compatibility by @bghira in #2946
  • Add Krea 2 segmented checkpointing support by @bghira in #2947
  • Add LongCat segmented checkpointing support by @bghira in #2948
  • Add LTXVideo segmented checkpointing support by @bghira in #2949
  • Add LTXVideo2 segmented checkpointing support by @bghira in #2950
  • Add Lumina2 segmented checkpointing support by @bghira in #2951
  • Add MageFlow segmented checkpointing support by @bghira in #2952
  • Add PixArt segmented checkpointing support by @bghira in #2953
  • Add Qwen Image segmented checkpointing support by @bghira in #2954
  • Add Sana segmented checkpointing support by @bghira in #2955
  • Add SanaVideo segmented checkpointing support by @bghira in #2956
  • Add SD3 segmented checkpointing support by @bghira in #2957
  • Add Stable Cascade segmented checkpointing support by @bghira in #2958
  • Add Wan segmented checkpointing support by @bghira in #2959
  • Add Z-Image segmented checkpointing support by @bghira in #2960
  • Add ZLab I1 segmented checkpointing support by @bghira in #2961
  • Fix automatic oversubscription scheduling with gradient accumulation by @hjinnkim in #2918
  • fix(data): only restore a checkpointed shard when every rank has one by @hjinnkim in #2986
  • fix(data): keep post-split shard length when filtering a sample by @hjinnkim in #2987
  • add licensing info to feature matrix by @bghira in #2991
  • Document model license statuses by @bghira in #2992
  • (#2990 continued) LoRA state dict key naming cleanup by @bghira in #2993
  • Fix checkpoint rotation after successful saves by @hjinnkim in #2989
  • Fix RamTorch prefetch-order save race by @hjinnkim in #2988
  • merge by @bghira in #2994

New Contributors

Full Changelog: v4.5.2...v4.6.0

Contributors

bghira, hjinnkim, and cupkk
Assets 2
Loading