No description, website, or topics provided.
Switch branches/tags
Nothing to show
Clone or download
Latest commit a57cca5 May 17, 2018

README.md

WikiReading

This repository contains the three WikiReading datasets as used and described in WikiReading: A Novel Large-scale Language Understanding Task over Wikipedia, Hewlett, et al, ACL 2016 (the English WikiReading dataset) and Byte-level Machine Reading across Morphologically Varied Languages, Kenter et al, AAAI-18 (the Turkish and Russian datasets).

Run get_data.sh to download data the English WikiReading dataset.

Run get_ru_data.sh and get_tr_data.sh to get the Russian and Turkish version of the WikiReading data, respectively.

If you use the data or the results reported in the papers, please feel free to cite them.

@inproceedings {hewlett2016wikireading,
 title = {{WIKIREADING}: A Novel Large-scale Language Understanding Task over {Wikipedia}},
 booktitle = {Proceedings of the The 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016)},
 author = {Daniel Hewlett and Alexandre Lacoste and Llion Jones and Illia Polosukhin and Andrew Fandrianto and Jay Han and Matthew Kelcey and David Berthelot},
 year = {2016}
}

and

@inproceedings{byte-level2018kenter,
  title={Byte-level Machine Reading across Morphologically Varied Languages},
  author={Tom Kenter and Llion Jones and Daniel Hewlett},
  booktitle={Proceedings of the The Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18)},
  year={2018}
}

WikiReading Data

Train, validation, and test datasets are in TFRecord or streamed JSON (one JSON object per line). They are 45GB, 5GB, and 3GB respectively. For example test.tar.gz contains 15 files whose union is the whole test set. We split them to help speed up training/testing by parallelizing reads. Any one of the shards can be opened with a TFRecordReader or with your favorite JSON reader for every line. Download a sample TFRecord shard or a sample JSON shard of the validation set (1/15th) to play around with if disk space is limited.

English

file size description
train 16,039,400 examples TFRecord https://storage.googleapis.com/wikireading/train.tar.gz
JSON https://storage.googleapis.com/wikireading/train.json.tar.gz
validation 1,886,798 examples TFRecord https://storage.googleapis.com/wikireading/validation.tar.gz
JSON https://storage.googleapis.com/wikireading/validation.json.tar.gz
test 941,280 examples TFRecord https://storage.googleapis.com/wikireading/test.tar.gz
JSON https://storage.googleapis.com/wikireading/test.json.tar.gz
document.vocab 176,978 tokens vocabulary for tokens from Wikipedia documents
answer.vocab 10,876 tokens vocabulary for tokens from answers
raw_answer.vocab 1,359,244 tokens vocabulary for whole answers as they appear in WikiData
type.vocab 80 tokens vocabulary for Part of Speech tags
character.vocab 12486 tokens vocabulary for all characters that appear in the string sequences

Russian

file size description
train 4,259,667 examples TFRecord https://storage.googleapis.com/wikireading/ru/train.tar.gz
JSON https://storage.googleapis.com/wikireading/ru/train.json.tar.gz
validation 531,412 examples TFRecord https://storage.googleapis.com/wikireading/ru/valid.tar.gz
JSON https://storage.googleapis.com/wikireading/ru/valid.json.tar.gz
test 533,026 examples TFRecord https://storage.googleapis.com/wikireading/ru/test.tar.gz
JSON https://storage.googleapis.com/wikireading/ru/test.json.tar.gz
document.vocab 965,157 tokens vocabulary for tokens from Wikipedia documents
answer.vocab 57,952 tokens vocabulary for tokens from answers
type.vocab 56 tokens vocabulary for Part of Speech tags
character.vocab 12,205 tokens vocabulary for all characters that appear in the string sequences

Turkish

file size description
train 654,705 examples TFRecord https://storage.googleapis.com/wikireading/tr/train.tar.gz
JSON https://storage.googleapis.com/wikireading/tr/train.json.tar.gz
validation 81,622 examples TFRecord https://storage.googleapis.com/wikireading/tr/valid.tar.gz
JSON https://storage.googleapis.com/wikireading/tr/valid.json.tar.gz
test 82,643 examples TFRecord https://storage.googleapis.com/wikireading/tr/test.tar.gz
JSON https://storage.googleapis.com/wikireading/tr/test.json.tar.gz
document.vocab 215,294 tokens vocabulary for tokens from Wikipedia documents
answer.vocab 11,123 tokens vocabulary for tokens from answers
type.vocab 10 tokens vocabulary for Part of Speech tags
character.vocab 6638 tokens vocabulary for all characters that appear in the string sequences

Features

Each instance contains these features (some features may be empty).

feature name description
answer_breaks Indices into answer_ids and answer_string_sequence.
Used to delimit multiple answers to a question, e.g. a list answer.
answer_ids answer.vocab ID sequence for words in the answer.
answer_location Word indices into the document where any one token in the answer was found.
answer_sequence document.vocab ID sequence for words in the answer.
answer_string_sequence String sequence for the words in the answer.
break_levels One integer [0,4] indicating a break level for each word in the document.
* 0 = no separation between tokens
* 1 = tokens separated by space
* 2 = tokens separated by line break
* 3 = tokens separated by sentence break
* 4 = tokens separated by paragraph break
document_sequence document.vocab ID sequence for words in the document.
full_match_answer_location Word indices into the document where all contiguous tokens in answer were found.
paragraph_breaks Word indices into the document indicating a paragraph boundary.
question_sequence document.vocab ID sequence for words in the question.
question_string_sequence String sequence for the words in the question.
raw_answer_ids raw_answer.vocab ID for the answer.
raw_answers A string containing the raw answer.
sentence_breaks Word indices into the document indicating a sentence boundary.
string_sequence String sequence for the words in the document. character.vocab for char IDs.
type_sequence type.vocab ID sequence for tags (POS, type, etc.) in the document.