Skip to content
Simple transformer implementation from scratch in pytorch.
Branch: master
Clone or download
Type Name Latest commit message Commit time
Failed to load latest commit information.
data Move code over to own repo, tidy up Aug 15, 2019
experiments Cleaning up. Aug 22, 2019
former Fix comment and remove old masking code. Aug 29, 2019
tests Move code over to own repo, tidy up Aug 15, 2019
LICENSE Add MIT license Aug 22, 2019 Update Aug 26, 2019
environment.yml add env Aug 24, 2019 Cleaning up. Aug 22, 2019


Simple transformer implementation from scratch in pytorch. See for an in-depth explanation.


The current models are designed to show the simplicity of transformer models and self-attention. As such they will not scale as far as the bigger transformers. For that you'll need a number of tricks that complicate the code (see the blog post for details).

All models so far are a single stack of transformer blocks (that is, no encoder/decoder structures). It turns out that this simple configuration often works best.


You can clone the code and run the experiments from the root directory. E.g.

python experiments/

Hyperparameters are passed as command line arguments. The defaults should work well. The classification data is automatically downloaded, the wikipedia data is included in the repository.

You should be able to install as a package as well, with

pip install git+

but I haven't tried this. It's probably easier to just copy over the code you need. Let me know if you need this for anything and it doesn't work.


Python 3.6+ is required.

The following should install all requirements pip install torch tb-nightly tqdm numpy torchtext

You may also need pip install future depending on the exact python version.

conda environment

The file environment.yml describes a complete conda environment with all dependencies. After cloning or downloading the project, you create the environment as follows:

conda env create -f environment.yml --name former
conda activate former
You can’t perform that action at this time.