Skip to content


Repository files navigation

Iterative Rank-Aware Open IE

This repository contains data and code to reproduce the experiments in:

Improving Open Information Extraction via Iterative Rank-Aware Learning

by Zhengbao Jiang, Pengcheng Yin and Graham Neubig.


Our model implementation is based on AllenNLP, and evaluation is based on supervised-oie.

  1. Prepare the model repository:

    git clone
    cd oie_rank
    conda create -n oie_rank python=3.6
    conda activate oie_rank
    pip install allennlp==0.8.2
    conda deactivate
    cd ..
  2. Prepare the evaluation repository:

    git clone
    cd supervised-oie
    # rollback to the evaluation metrics used in this paper
    git reset --hard ef66de761a5230a7905918b4c7ce87147369bf33
    conda create -n sup_oie python=2.7
    conda activate sup_oie
    pip install docopt==0.6.2 numpy==1.11.2 scikit-learn==0.18 scipy==0.18.1 xtermcolor==1.3 termcolor==1.1.0 nltk==3.2.5 pandas==0.19.0 requests==2.21.0
    conda deactivate
    cd ..

Download pre-trained models

  1. Download pre-trained ELMo (weights and options).

    cd oie_rank
    mkdir pretrain/elmo/
    cd pretrain/elmo/
  2. Download pre-trained RnnOIE (base model in our paper) and its extractions. can be download from here.

    cd oie_rank
    mkdir pretrain/rnnoie/
    cd pretrain/rnnoie/
    # download model model.tar.gz
    # download extractions oie2016.txt

Iterative rank-aware learning

First, create a directory to hold data, models, and extractions during iterative training.


Then, run iterative rank-aware training (3 iterations with beam size of 5).

CONDA_HOME=PATH_TO_YOUR_CONDA_HOME ./ 3 ROOT_DIR training_config/iter.jsonnet 5

The extractions generated at i-th iteration are saved to ROOT_DIR/eval/iteri/tag_model/. The reranking results of extractions generated at i-1-th iteration are saved to ROOT_DIR/eval/iteri/. Code for rank-aware binary classification loss is in rerank/models/, which is really simple and could be easily applied to other models and tasks.

Train and evaluate RnnOIE

In addition to using pretrained RnnOIE model, you can also use the following commands to train from scratch and evaluate.

cd oie_rank
conda activate oie_rank
# train
allennlp train training_config/oie.jsonnet \
    --include-package rerank \
    --serialization-dir MODEL_DIR
# generate extractions
python \
    --model MODEL_DIR/model.tar.gz \
    --inp data/sent/oie2016_test.txt \
    --out RESULT_DIR/oie2016.txt \
# evaluate
cd ../supervised-oie/supervised-oie-benchmark/
conda activate sup_oie
python \
    --gold oie_corpus/test.oie.orig.correct.head \
    --out /dev/null \
    --tabbed RESULT_DIR/oie2016.txt \


    title = {Improving Open Information Extraction via Iterative Rank-Aware Learning},
    author = {Zhengbao Jiang and Pengcheng Yin and Graham Neubig},
    booktitle = {The 57th Annual Meeting of the Association for Computational Linguistics (ACL)},
    address = {Florence, Italy},
    month = {July},
    year = {2019}


Iterative Rank-Aware Open IE






No releases published


No packages published