Skip to content


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?


Failed to load latest commit information.
Latest commit message
Commit time
June 11, 2018 06:32
June 10, 2018 23:04
June 10, 2018 23:04
November 21, 2018 15:50
June 11, 2018 06:32
June 11, 2018 06:32
June 11, 2018 06:32
June 11, 2018 06:32
August 30, 2018 18:41
August 30, 2018 18:41

Status: Archive (code is provided as-is, no updates expected)


Code and model for the paper "Improving Language Understanding by Generative Pre-Training"

Currently this code implements the ROCStories Cloze Test result reported in the paper by running: python --dataset rocstories --desc rocstories --submit --analysis --data_dir [path to data here]

Note: The code is currently non-deterministic due to various GPU ops. The median accuracy of 10 runs with this codebase (using default hyperparameters) is 85.8% - slightly lower than the reported single run of 86.5% from the paper.

The ROCStories dataset can be downloaded from the associated website.