A PyTorch implementation of BERT, pretraining and finetuning, written to understand the architecture and the masked-language-modelling objective from the ground up rather than calling into a library.
model/ the transformer encoder, embeddings and heads
dataset/ MLM masking and next-sentence-prediction data prep
trainer/ the training loop