Mixture of Experts

[StellaAthena]

Is the Mixture of Experts model still a work-in-progress as indicated in the README? Which MoE model is being implemented? Is there a reason we are not using @lucidrains's package https://github.com/lucidrains/mixture-of-experts?

Tagging @Mistobaan who asked about this in Discord.

[lucidrains]

@StellaAthena ohh, that package is for pytorch

[lucidrains]

@StellaAthena it's already done, we just need to test it in a run

[StellaAthena]

Thanks! Tomorrow I’ll update the documentation, add testing MoE to the Kanban, and close this issue.

[StellaAthena]

Documentation has been updated. Moved the question of testing the code to #37