BERT-on-AGNews

A detailed walk-through of using pytorch-transformers and BERT for text classification.

These notes will show you how to use BERT for text-classification. Some highlights:

• dataloaders in pytorch,
• BERT from pytorch-transformers,
• freezing layers,
• learning rate schedulers, optimizers and gradient clipping,
• mixed precision training,
• logging your training.

Resources:

• a good tutorial from C. McCormick
• the original BERT paper
• the main library we use is pytorch-tranformers from huggingface
• 'Fine-tuning BERT for text classification' by Sun et. al
• a post on compressing and speeding-up large models