Unified Efficient Fine-Tuning of 100+ LLMs (ACL 2024)

SGLang is a fast serving framework for large language models and vision language models.

Mixture-of-Experts for Large Vision-Language Models

PyTorch Re-Implementation of "The Sparsely-Gated Mixture-of-Experts Layer" by Noam Shazeer et al. https://arxiv.org/abs/1701.06538

⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training (EMNLP 2024)

A toolkit for inference and evaluation of 'mixtral-8x7b-32kseqlen' from Mistral AI

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

Tutel MoE: An Optimized Mixture-of-Experts Implementation

中文Mixtral混合专家大模型（Chinese Mixtral MoE LLMs）

ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts. We released a collection of ModuleFormer-based Language Models (MoLM) ranging in scale from 4 billion to 8 billion parameters.

MoH: Multi-Head Attention as Mixture-of-Head Attention

MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts

🚀 Easy, open-source LLM finetuning with one-line commands, seamless cloud integration, and popular optimization frameworks. ✨

Implementation of MoE Mamba from the paper: "MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts" in Pytorch and Zeta

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

Batch download high quality videos from https://twist.moe

Implementation of Switch Transformers from the paper: "Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity"

[Preprint] Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models

[ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal, Shiwei Liu, Zhangyang Wang

pytorch open-source library for the paper "AdaTT Adaptive Task-to-Task Fusion Network for Multitask Learning in Recommendations"