DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
-
Updated
May 14, 2024 - Python
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
Run Mixtral-8x7B models in Colab or consumer desktops
Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.
Mixture-of-Experts for Large Vision-Language Models
PyTorch Re-Implementation of "The Sparsely-Gated Mixture-of-Experts Layer" by Noam Shazeer et al. https://arxiv.org/abs/1701.06538
⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
A TensorFlow Keras implementation of "Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts" (KDD 2018)
Tutel MoE: An Optimized Mixture-of-Experts Implementation
A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models
中文Mixtral混合专家大模型(Chinese Mixtral MoE LLMs)
A library for easily merging multiple LLM experts, and efficiently train the merged LLM.
GMoE could be the next backbone model for many kinds of generalization task.
Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch
Implementation of Soft MoE, proposed by Brain's Vision team, in Pytorch
Fast Inference of MoE Models with CPU-GPU Orchestration
Some personal experiments around routing tokens to different autoregressive attention, akin to mixture-of-experts
RealCompo: Dynamic Equilibrium between Realism and Compositionality Improves Text-to-Image Diffusion Models
[SIGIR'24] The official implementation code of MOELoRA.
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
MoEL: Mixture of Empathetic Listeners
Add a description, image, and links to the mixture-of-experts topic page so that developers can more easily learn about it.
To associate your repository with the mixture-of-experts topic, visit your repo's landing page and select "manage topics."