Xmixers: A collection of SOTA efficient token/channel mixers

💬 Discord •

Introduction

This repository aims to implement SOTA efficient token/channel mixers. Any technologies related to non-Vanilla Transformer are welcome. If you are interested in this repository, please join our Discord.

Roadmap

• Token Mixers
  • Linear Attention
  • Linear RNN
  • Long Convolution
• Channel Mixers

ToDo

• Add special init.

Model

• LLaMA.

Basic

• Add data type for class and function.

Ops

• long_conv_1d_op.

Token Mixers

• Gtu.