Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 

CrowdTruth Corpus for Open Domain Relation Extraction from Sentences

DOI

This repository contains a ground truth corpus for open domain relation extraction from sentences, acquired with crowdsourcing and processed with CrowdTruth metrics that capture ambiguity in annotations by measuring inter-annotator disagreement.

The dataset contains annotations for 4,100 sentences sampled from Angeli et al. (1) and Riedel et al. (2), over 16 relations, with each sentence annotated by 15 workers. The sentences have been pre-processed with Distant Supervision (3) using the Freebase knowledge base, in order to identify the term pairs in each sentence that are likely to express a relation. The crowdsourced data was collected from Figure Eight and Amazon Mechanical Turk.

This corpus has been discussed in the following papers:

Sentence-level data is available in file: |--data/output/aggregated_sentences.csv

Worker-level data is available in file: |--data/output/aggregated_workers.csv

Raw crowdsourcig data is available in folder: |--data/input/

Results of the relation classification model are available in folder: |--data/model_results/

References

(1) Angeli, Gabor, et al. "Combining distant and partial supervision for relation extraction." Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 2014.

(2) Riedel, Sebastian, et al. "Relation extraction with matrix factorization and universal schemas." Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). 2013.

(3) Mintz, Mike, et al. "Distant supervision for relation extraction without labeled data." Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2. Association for Computational Linguistics, 2009.

About

Crowdsourced data for open domain relation classification from sentences

Resources

Packages

No packages published