DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
-
Updated
Sep 21, 2024 - Python
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
Run Mixtral-8x7B models in Colab or consumer desktops
A TensorFlow Keras implementation of "Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts" (KDD 2018)
Surrogate Modeling Toolbox
Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.
Mixture-of-Experts for Large Vision-Language Models
PyTorch Re-Implementation of "The Sparsely-Gated Mixture-of-Experts Layer" by Noam Shazeer et al. https://arxiv.org/abs/1701.06538
Tutel MoE: An Optimized Mixture-of-Experts Implementation
Optimizing inference proxy for LLMs
From scratch implementation of a sparse mixture of experts language model inspired by Andrej Karpathy's makemore :)
A Pytorch implementation of Sparsely-Gated Mixture of Experts, for massively increasing the parameter count of language models
⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training (EMNLP 2024)
中文Mixtral混合专家大模型(Chinese Mixtral MoE LLMs)
GMoE could be the next backbone model for many kinds of generalization task.
Implementation of ST-Moe, the latest incarnation of MoE after years of research at Brain, in Pytorch
Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).
A library for easily merging multiple LLM experts, and efficiently train the merged LLM.
Hierarchical Mixture of Experts,Mixture Density Neural Network
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
Fast Inference of MoE Models with CPU-GPU Orchestration
Add a description, image, and links to the mixture-of-experts topic page so that developers can more easily learn about it.
To associate your repository with the mixture-of-experts topic, visit your repo's landing page and select "manage topics."