Skip to content
Dataset of GPT-2 outputs for research in detection, biases, and more
Branch: master
Clone or download
Latest commit 6f48b7f May 13, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore first commit, readme and download May 3, 2019
README.md Update README.md May 14, 2019
baseline.py update to gridsearch May 3, 2019
download_dataset.py first commit, readme and download May 3, 2019

README.md

gpt-2-output-dataset

This dataset contains:

  • 250K documents from the WebText test set
  • For each GPT-2 model (trained on the WebText training set), 250K random samples (temperature 1, no truncation) and 250K samples generated with Top-K 40 truncation

We look forward to the research produced using this data!

Download

For each model, we have a training split of 250K generated examples, as well as validation and test splits of 5K examples.

All data is located in Google Cloud Storage, under the directory gs://gpt-2/output-dataset/v1.

There, you will find files:

  • webtext.${split}.jsonl
  • small-117M.${split}.jsonl
  • small-117M-k40.${split}.jsonl
  • medium-345M.${split}.jsonl
  • medium-345M-k40.${split}.jsonl
  • large-762M.${split}.jsonl
  • large-762M-k40.${split}.jsonl
  • xl-1542M.${split}.jsonl
  • xl-1542M-k40.${split}.jsonl

where split is one of train, test, and valid.

We've provided a script to download all of them, in download_dataset.py.

Detectability baselines

We're interested in seeing research in detectability of GPT-2 model family generations.

We've provided a starter baseline which trains a logistic regression detector on TF-IDF unigram and bigram features, in baseline.py. The baseline achieves the following accuracies:

Model Temperature 1 Top-K 40
117M 88.29% 96.79%
345M 88.94% 95.22%
762M 77.16% 94.43%
1542M 74.31% 92.69%

Initial Analysis

Unsurprisingly, shorter documents are harder to detect and performance improves gradually with length. Accuracy of detection of short documents of 500 characters (a long paragraph) is about 15% lower.

Truncated sampling, which is commonly used for high-quality generations from the GPT-2 model family, results in a shift in the part of speech distribution of the generated text compared to real text. A clear example is the underuse of proper nouns and overuse of pronouns which are more generic. This shift contributes to the 8% to 18% higher detection rate of Top-K samples compared to random samples across models.

Data removal requests

If you believe your work is included in WebText and would like us to remove it, please let us know at webtextdata@openai.com.

You can’t perform that action at this time.