Summary Loop CCN/DM Test Set Outputs

Releasing the 11,490 summaries generated by the Summary Loop model (summary_loop_length46.bin) on the CNN/DM test set.
Each summary is released attached with the CNN/DM id.
The following code snippet can be used to evaluate ROUGE scores:

from datasets import load_dataset, load_metric
import json
with open("/home/phillab/data/cnndm_test_summary_loop.json", "r") as f:
    summary_loop_gens = json.load(f)
rouge = load_metric("rouge")
dataset_test = load_dataset("cnn_dailymail", "3.0.0")["test"]
id2summary_loop = {d["id"]: d["summary_loop_gen"] for d in summary_loop_gens}
candidates, references = [], []
for d in dataset_test:
    references.append(d["highlights"])
    candidates.append(id2summary_loop[d["id"]])
print(len(references), len(candidates))
print(rouge.compute(predictions=candidates, references=references))

Notes:
(1) this relies on HuggingFace's datasets repository (https://github.com/huggingface/datasets) to load the CNN/DM dataset, and the ROUGE metric.
(2) The ROUGE metric implementation used in the above example is not the original, PERL-based implementation of ROUGE used for official numbers in the paper. This serves for demonstration purposes to show how to use the file.

Initial models compatible with latest release of transformers library

We release an upgraded set of initial models for the training script that are compatible with transformers==3.1.0 to make it easier to get started.
The original release (0.1) used version 2.8.0 of transformers, and there were some breaking changed introduced since, which leads to some model loading failing.
The requirements.txt in the latest release has been updated with compatible library versions to simplify installation.

Initial Models

These sets of models work using Python 3.6.10, Transformers 3.1.0 and Sklearn 0.22.1:

• keyword_extractor.joblib: An sklearn pipeline that will extract can be used to compute tf-idf scores of words according to the BERT vocabulary, which is used by the Masking Procedure,
• bert_coverage.bin: A bert-base-uncased finetuned model on the task of Coverage for the news domain,
• fluency_news_bs32.bin: A GPT2 (base) model finetuned on a large corpus of news articles, used as the Fluency model,
• gpt2_copier23.bin: A GPT2 (base) model that can be used as an initial point for the Summarizer model.

Final Models

Unfortunately, the three final models (trained summarizers) released in v0.1 do not work anymore in the latest transformers library, and only work in versions 2.8.0 and before. Once we retrain these models, we will reupload them. If this is of interest, feel free to add an issue or contact us.

Models and dataset necessary to run the code

We release models and data needed to run the Summary Loop and use the models we trained.

Initial models

Here are the models needed to run the train_summary_loop.py:

• keyword_extractor.joblib: An sklearn pipeline that will extract can be used to compute tf-idf scores of words according to the BERT vocabulary, which is used by the Masking Procedure,
• bert_coverage.bin: A bert-base-uncased finetuned model on the task of Coverage for the news domain,
• fluency_news_bs32.bin: A GPT2 (base) model finetuned on a large corpus of news articles, used as the Fluency model,
• gpt2_copier23.bin: A GPT2 (base) model that can be used as an initial point for the Summarizer model.

Sample dataset

We release a sample dataset of Wikinews news articles to get researchers started using the Summary Loop: wikinews.db.
We cannot release the full dataset we used for copyright reasons. We note that we do not expect this to be enough to train to best performance, and recommend finding larger datasets (such as Newsroom or CNN/DM) for full-fledged training.

Final models

We release 3 Summarizer models obtained through the Summary Loop procedure for 3 target lengths: summary_loop_length_12.bin, summary_loop_length_27.bin, summary_loop_length_61.bin