Neural Network Architectures

This project hosts a collection of custom neural network architectures I implemented with PyTorch. I am especially interested in Graph Neural Networks and Transformer architectures. See below for a list of the implementations so far.

Graph Neural Networks

• Graph Attention Networks (GAT) ¹
  • GAT with Paired Nodes Attention [Code]
  • GAT with Masked Self-Attention [Code]
• Graph Convolutional Networks (GCN) ^{2 3}

Gated Neural Networks

• Gated Linear Unit (GLU) ⁴ [Code]
• Gated Residual Network (GRN) ⁵ [Code]

Variable Selection Neural Networks

• Variable Selection Network (VSN) ⁵ [Code]

Other Networks

• Modern Hopfield Network ⁶
• Self-Normalizing Neural Network ⁷ [Code]

References

Footnotes

1. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., & Bengio, Y. (2017). Graph Attention Networks. doi:10.48550/ARXIV.1710.10903 [arxiv] ↩

2. Kipf, T. N., & Welling, M. (2016). Semi-Supervised Classification with Graph Convolutional Networks. doi:10.48550/ARXIV.1609.02907 [arxiv] ↩

3. Defferrard, M., Bresson, X., & Vandergheynst, P. (2016). Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. doi:10.48550/ARXIV.1606.09375 [arxiv] ↩

4. Dauphin, Y. N., Fan, A., Auli, M., & Grangier, D. (2016). Language Modeling with Gated Convolutional Networks. doi:10.48550/ARXIV.1612.08083 [arxiv] ↩

5. Lim, B., Arik, S. O., Loeff, N., & Pfister, T. (2019). Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting. doi:10.48550/ARXIV.1912.09363 [arxiv] ↩ ↩²

6. Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., … Hochreiter, S. (2020). Hopfield Networks is All You Need. doi:10.48550/ARXIV.2008.02217 [arxiv] ↩

7. Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017). Self-Normalizing Neural Networks. doi:10.48550/ARXIV.1706.02515 [arxiv] ↩