Skip to content
Iterative Rank-Aware Open IE
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
data add test conll file Jun 24, 2019
pretrain/vocabulary init commit May 30, 2019
rerank init commit May 30, 2019
training_config init commit May 30, 2019 modify paper link Jun 3, 2019 update installation & readme May 31, 2019 modify script location & modify readme May 30, 2019 modify script location & modify readme May 30, 2019 modify script location & modify readme May 30, 2019

Iterative Rank-Aware Open IE

This repository contains data and code to reproduce the experiments in:

Improving Open Information Extraction via Iterative Rank-Aware Learning

by Zhengbao Jiang, Pengcheng Yin and Graham Neubig.


Our model implementation is based on AllenNLP, and evaluation is based on supervised-oie.

  1. Prepare the model repository:

    git clone
    cd oie_rank
    conda create -n oie_rank python=3.6
    conda activate oie_rank
    pip install allennlp==0.8.2
    conda deactivate
    cd ..
  2. Prepare the evaluation repository:

    git clone
    cd supervised-oie
    # rollback to the evaluation metrics used in this paper
    git reset --hard ef66de761a5230a7905918b4c7ce87147369bf33
    conda create -n sup_oie python=2.7
    conda activate sup_oie
    pip install docopt==0.6.2 numpy==1.11.2 scikit-learn==0.18 scipy==0.18.1 xtermcolor==1.3 termcolor==1.1.0 nltk==3.2.5 pandas==0.19.0 requests==2.21.0
    conda deactivate
    cd ..

Download pre-trained models

  1. Download pre-trained ELMo (weights and options).

    cd oie_rank
    mkdir pretrain/elmo/
    cd pretrain/elmo/
  2. Download pre-trained RnnOIE (base model in our paper) and its extractions. can be download from here.

    cd oie_rank
    mkdir pretrain/rnnoie/
    cd pretrain/rnnoie/
    # download model model.tar.gz
    # download extractions oie2016.txt

Iterative rank-aware learning

First, create a directory to hold data, models, and extractions during iterative training.


Then, run iterative rank-aware training (3 iterations with beam size of 5).

CONDA_HOME=PATH_TO_YOUR_CONDA_HOME ./ 3 ROOT_DIR training_config/iter.jsonnet 5

The extractions generated at i-th iteration are saved to ROOT_DIR/eval/iteri/tag_model/. The reranking results of extractions generated at i-1-th iteration are saved to ROOT_DIR/eval/iteri/. Code for rank-aware binary classification loss is in rerank/models/, which is really simple and could be easily applied to other models and tasks.

Train and evaluate RnnOIE

In addition to using pretrained RnnOIE model, you can also use the following commands to train from scratch and evaluate.

cd oie_rank
conda activate oie_rank
# train
allennlp train training_config/oie.jsonnet \
    --include-package rerank \
    --serialization-dir MODEL_DIR
# generate extractions
python \
    --model MODEL_DIR/model.tar.gz \
    --inp data/sent/oie2016_test.txt \
    --out RESULT_DIR/oie2016.txt \
# evaluate
cd ../supervised-oie/supervised-oie-benchmark/
conda activate sup_oie
python \
    --gold oie_corpus/test.oie.orig.correct.head \
    --out /dev/null \
    --tabbed RESULT_DIR/oie2016.txt \


    title = {Improving Open Information Extraction via Iterative Rank-Aware Learning},
    author = {Zhengbao Jiang and Pengcheng Yin and Graham Neubig},
    booktitle = {The 57th Annual Meeting of the Association for Computational Linguistics (ACL)},
    address = {Florence, Italy},
    month = {July},
    year = {2019}
You can’t perform that action at this time.