my awesome dotfiles

PyTorch implementation of moe, which stands for mixture of experts

Audio conversion module

Network editor for MoE devices

[ICLR 2023] "Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers" by Tianlong Chen*, Zhenyu Zhang*, Ajay Jaiswal, Shiwei Liu, Zhangyang Wang

Meet Moe, a discord bot, written in modern python!

pytorch open-source library for the paper "AdaTT Adaptive Task-to-Task Fusion Network for Multitask Learning in Recommendations"

Batch download high quality videos from https://twist.moe

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

A toolkit for inference and evaluation of 'mixtral-8x7b-32kseqlen' from Mistral AI

⛷️ LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

Anime Themes Quiz for people with taste.

Restoring Spatially-Heterogeneous Distortions using Mixture of Experts Network (ACCV 2020)

ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts. We released a collection of ModuleFormer-based Language Models (MoLM) ranging in scale from 4 billion to 8 billion parameters.

Official repository for paper "MATERobot: Material Recognition in Wearable Robotics for People with Visual Impairments" at ICRA 2024, Best Paper Finalist on Human-Robot Interaction

A simple project that help visualize expert router choices for text generation

PyTorch Re-Implementation of "The Sparsely-Gated Mixture-of-Experts Layer" by Noam Shazeer et al. https://arxiv.org/abs/1701.06538

Community Implementation of the paper: "Multi-Head Mixture-of-Experts" In PyTorch

中文Mixtral混合专家大模型（Chinese Mixtral MoE LLMs）

This is the repo for the MixKABRN Neural Network (Mixture of Kolmogorov-Arnold Bit Retentive Networks), and an attempt at first adapting it for training on text, and later adjust it for other modalities.