Description
Many recent frontier models are using MoE architecture (e.g. DeepSeek-V3, OLMoE). We recently implemented MoE architecture in our training framework Levanter and would like to learn more about the effectiveness of this architecture.
OLMoE has done some incredible work analyzing different configurations of MoE and compared with dense models. Their results showed that olmoe (moe) significantly outperforms olmo (dense) in evaluation in many evaluation tasks. However, there are a lot of differences (e.g. tokens trained and batch size) between their moe and dense model. Also, we found that our dense model training has a much higher MFU than theirs, so it may close the gap because it trains much faster than moe training.
Hypothesis or Goal
This experiment aims to replicate the resuls of olmoe but with a more balanced training setup: both model should train on the same amount of tokens and the same batch size.
We will compare the training and evaluation performance of an 8x1b moe model and a 1b dense model and hopefully replicate the result from olmoe, where the moe model should not only outperform the dense model with the same number of FLOPs, but also if given the same number of time to train.
Links
Results
- MoE significantly outperforms dense counterpart with the same amount of tokens.
- Small fine-grained experts leads to even worse MFU :(
- When x-axis is time, it seems like there's an intersection point at ~20hr such that moe becomes better than dense even though dense training has a much better MFU.
- The performance difference is also reflected in evaluation.
Description
Many recent frontier models are using MoE architecture (e.g. DeepSeek-V3, OLMoE). We recently implemented MoE architecture in our training framework Levanter and would like to learn more about the effectiveness of this architecture.
OLMoE has done some incredible work analyzing different configurations of MoE and compared with dense models. Their results showed that olmoe (moe) significantly outperforms olmo (dense) in evaluation in many evaluation tasks. However, there are a lot of differences (e.g. tokens trained and batch size) between their moe and dense model. Also, we found that our dense model training has a much higher MFU than theirs, so it may close the gap because it trains much faster than moe training.
Hypothesis or Goal
This experiment aims to replicate the resuls of olmoe but with a more balanced training setup: both model should train on the same amount of tokens and the same batch size.
We will compare the training and evaluation performance of an 8x1b moe model and a 1b dense model and hopefully replicate the result from olmoe, where the moe model should not only outperform the dense model with the same number of FLOPs, but also if given the same number of time to train.
Links
Results