Skip to content

Load balancing Fxi

Pre-release
Pre-release

Choose a tag to compare

@smithblack-0 smithblack-0 released this 05 Jun 20:31
· 81 commits to master since this release
24452c5

The deepseek style load balancing was too slow and did not correct itself as the imbalance worsens. This update implements loss choices that correct this issue. The gshard, ce, and bce options are all available, and the ce and bce have familiar logprob properties which make them crank up the pain as load balancing worsens. Like Deepseek, we still perform load balancing on decoupled logits.