Skip to content
Fine-tuning GPT-2 Small for Question Answering
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
gpt2sqa init Jun 1, 2019
LICENSE init Jun 1, 2019 init Jun 1, 2019 Update May 30, 2019 init Jun 1, 2019 init Jun 1, 2019
requirements.txt init Jun 1, 2019 init Jun 1, 2019


This repo includes an experiment of fine-tuning GPT-2 117M for Question Answering (QA). It also runs the model on Stanford Question Answering Dataset 2.0 (SQuAD). It uses Huggingface Inc.'s PyTorch implementation of GPT-2 and adapts from their fine-tuning of BERT for QA.

SQuAD data can be downloaded from:

To train and validate the model:

python --output_dir=output/ --train_file=data/train-v2.0.json --do_train --train_batch_size=32 --predict_file=data/dev-v2.0.json --do_predict

To evaluate:

python data/dev-v2.0.json output/predictions.json

Different fine-tuning experiments will be uploaded soon for GPT-2 345M on datasets that exclusively target commonsense reasoning in an attempt to bring insight to reasoning abilities of GPT-2. Such an insight could potentially improve our ability to improve Natural Language Understanding through language models in semi-supervised settings.

You can’t perform that action at this time.