Skip to content

willGuimont/transformers

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

42 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

transformers

Collection of easy to understand transformer-based models in PyTorch. The implementation is heavily commented and should be easy to follow.

Installation

pip install git+https://github.com/willGuimont/transformers

Implemented models

General:

  • Transformer (Vaswani et al., 2017)
  • Parallel Transformer (Dehghani et al., 2023)
  • PerceiverIO (Jaegle et al., 2022)

Positional encoding:

  • Sinusoidal positional encoding
  • Relative positional encoding
  • Learnable positional encoding
  • Learnable Fourier positional encoding (Li, 2021)

Vision:

  • VisionTransformer (Dosovitskiy et al., 2021)

NLP:

  • Simple character-level Transformer language model

Next steps

  • VICReg
  • Rotary positional encoding https://arxiv.org/pdf/2104.09864.pdf
  • Optimizing Deeper Transformers on Small Datasets (Xu et al., 2021)
  • Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau et al., 2016)
  • Universal Transformers Paper
  • Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles (Ryali et al., 2023)
  • Swin Transformer (Liu et al., 2021)
  • DINO: Emerging Properties in Self-Supervised Vision Transformers (Caron et al., 2021)
  • FlashAttention (Dao et al., 2022)
  • DETR (Carion et al., 2020)
  • Unlimiformer: Long-Range Transformers with Unlimited Length Input (Bertsch et al., 2023)
  • PointBERT (Yu et al., 2022)
  • Hydra Attention: Efficient Attention with Many Heads (Bolya et al., 2022)
  • Hyena Hierarchy: Towards Larger Convolutional Language Models (Poli et al., 2023)
  • Thinking Like Transformers (Weiss et al., 2021)
  • Long short-term memory (Schmidhuber, 1997)
  • Rethinking Positional Encoding in Language Pre-training (Ke et al., 2021)

Cite this repository

@software{Guimont-Martin_transformer_flexible_and_2023,
    author = {Guimont-Martin, William},
    month = {2},
    title = {{transformer: flexible and easy to understand transformer models}},
    version = {0.1.0},
    year = {2023}
}