No description, website, or topics provided.
Switch branches/tags
Nothing to show
Clone or download
Latest commit e7e6479 May 2, 2018

README.md

PICO-data for Participants

This repo contains raw annotaions for PICO dataset used in paper:

Aggregating and Predicting Sequence Labels from Crowd Annotations An Thanh Nguyen, Byron C. Wallace, Junyi Jessy Li, Ani Nenkova and Matthew Lease Association for Computational Linguistics (ACL), 2017.

A SDK and sample codes are provided for retrieving the annotations.

Description

The dataset is in annotations/, it is splited into 4 parts:

  1. train/ contains random selected 3549 abstracts.
  2. dev/ contains random selected 500 abstracts.
  3. test/ contains random selected 500 abstracts.
  4. acl17-test contains 191 abstarcts with annotations by a medical student.

In each folder:

  1. PICO-annos-crowdsourcing.json contains annotations from crowd sourced workers.
  2. PICO-annos-crowdsourcing-agg.json contains aggregated results from crowd sourced annotations. The aggregation methods are described in Aggregating and Predicting Sequence Labels from Crowd Annotations.:
  3. PICO-annos-professional.json for acl17-test only, contains annotations from a medical student.

Environment and dependencies:

cd src
python -m examples.load_annotation