Repository navigation
Load balancing Fxi
Pre-release
Pre-release
·
81 commits
to master
since this release
The deepseek style load balancing was too slow and did not correct itself as the imbalance worsens. This update implements loss choices that correct this issue. The gshard, ce, and bce options are all available, and the ce and bce have familiar logprob properties which make them crank up the pain as load balancing worsens. Like Deepseek, we still perform load balancing on decoupled logits.