Skip to content

v4.11.0 - CuTe attention by default, FP32 EMA accumulation, bugfixes

Latest
Latest

Choose a tag to compare

View all tags
@bghira bghira released this 28 Oct 16:38
v4.11.0
be6dff9
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Breaking changes

  • The examples are now sorted by model family, flavour, and VRAM levels, since the directory listing was getting to be quite long
  • EMA accumulation now occurs in FP32 to avoid losing small deltas that led to EMA being stuck in place or degrading
  • KohakuFA-CuTe dependencies now included in cuda and cuda13 extras so that kohakufa-cute can be used as default attention_mechanism
  • cuDNN attention is now forcibly disabled due to ongoing correctness issues in each cudnn-frontend release
  • batch sizes, repeats, and other dataset config options will override the previous checkpoints' cached values during restart

Features

  • Initial Ampere (3090) performance-tuned support for CuTe attention now included
  • Ada (4090, L40S) performance-tuned CuTe attention improves over previous release's speed
  • VAE tiling is automatically enabled upon OOM instead of requiring the user to enable it manually and restart the trainer
  • LoRA/LyCORIS adapters can now be trained using fp32 weights with option --adapter_model_precision=no_change|fp32
    • This is helpful for some models like MOSS Image where bf16 LoRA/LyCORIS weights take longer to converge due to the loss of small update deltas

Bugfixes

  • AnyFlow improvements for correctness and reliability (especially for MOSS Image)
  • MOSS Image regional compile will no longer reach the torch dynamo cache limit by default
  • REPA multiGPU training now serialises the model download + load procedure to avoid multiple processes racing to finish it first
  • LyCORIS dependency is updated to fix LoKr kernel dispatch issue

What's Changed

  • Organize training examples by model family, flavour, and VRAM by @bghira in #3282
  • Port KohakuFA key centering and precision diagnostics by @bghira in #3283
  • Keep EMA accumulation in FP32 during mixed-precision training by @bghira in #3287
  • Serialize DINOv2 cache loading across REPA workers by @bghira in #3288
  • Target repeated MOSS transformer blocks for regional compilation by @bghira in #3289
  • Require LyCORIS 4.1.3 to fix LoKR kernel dispatch by @bghira in #3290
  • Refresh Webshart caption and image metadata caches on worker reload by @bghira in #3291
  • Sync CuTe attention with KohakuFA-CuTe 9280c20 by @bghira in #3293
  • Optimize SM120 CuTe attention with full-width D512 tiles by @bghira in #3294
  • Add SM86 Ampere support and improve Ada CuTe attention by @bghira in #3295
  • Disable cuDNN SDPA to prevent nonfinite attention gradients by @bghira in #3296
  • Enable VAE tiling automatically after CUDA OOM by @bghira in #3297
  • Respect caption dropout in AnyFlow derivative guidance by @bghira in #3301
  • Compute AnyFlow latent interpolation in FP32 by @bghira in #3300
  • Add optional FP32 storage for trainable adapter parameters by @bghira in #3299
  • Preserve runtime dataset settings when reloading metadata caches by @bghira in #3298
  • Sync KohakuFA-CuTe forward tuning and launch-grid fixes by @bghira in #3303
  • Include KohakuFA-CuTe dependencies in CUDA extras by @bghira in #3302
  • merge by @bghira in #3304

Full Changelog: v4.10.1...v4.11.0

Contributors

bghira
Assets 2
Loading