Skip to content
Go to file

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time


Source code for ACL 2019 paper "DIAG-NRE: A Neural Pattern Diagnosis Framework for Distantly Supervised Neural Relation Extraction".


DIAG-NRE aims to establish a bridge between Distant Supervision (DS) and Weak Label Fusion (WLF) for Neural Relation Extraction (NRE), which generally contains two key stages:

  • Pattern Extraction, generating patterns automatically based on reinforcement learning (RL);
  • Pattern Refinement, refining patterns with the help of human annotations on a small set of actively selected instances.

Advantages of DIAG-NRE include:

  • denoising distantly supervised labels with reduced human skill requirements;
  • enabling quicker generalization to new relation types;
  • interpreting which features neural models have learned.
  • interpreting from what kinds of noises the target relation type suffers.


If you find our work interesting, you can cite the paper as

               title={DIAG-NRE: A Neural Pattern Diagnosis Framework for Distantly Supervised Neural Relation Extraction},
               author={Zheng, Shun and Han, Xu and Lin, Yankai and Yu, Peilin and Chen, Lu and Huang, Ling and Liu, Zhiyuan and Xu, Wei},


  1. Run conda env create --file=environment.yml to prepare a basic python environment.

  2. Then, when starting a fresh shell, please run the following codes first.

# activate the conda env
source activate diag-nre
# set proper environment variables, including 'PATH', 'PYTHONPATH', 'WORK_DIR', etc.
source ./shell/ will help to set proper environment variables, and create logs and model directories when needed.

Data Preparation

We have preprocessed the NYT dataset and the UW as mentioned in the paper.

  1. Download our preprocessed data collection associated with human diagnostic annotations,, into the current directory.

  2. Run the following codes to prepare well-formatted data for subsequent model training

cd base_data
# 'WORK_DIR' will be set properly after executing 'source ./shell/'
python $WORK_DIR


0. Training NRE Models

Run python, then the program will automatically bind idle gpus and train NRE models for all 14 relations.

1. Pattern Extraction

Run python, then the program will train pattern-extraction agents for all 14 relations by interacting with well-trained NRE models.

2. Pattern Refinement + Weak Label Fusion (WLF)

Run python, then the program will

  • conduct the pattern refinement based on human diagnostic annotations;
  • produce denoised training labels by fusing multiple weak supervision sources;

Here, we already provide our annotations based on the actively selecting process described in the paper.

For users who need to apply DIAG-NRE to other data, you can extract instances to be annotated from the pattern hierarchy file, named as 'rule_hierarchy_model{}.pkl'.

Example codes to check the pattern hierarchy

# load hierarchy
import pickle
with open('rule_hierarchy_file_path', 'rb') as fin:
    pattern_hierarchy = pickle.load(fin)

# print hierarchy
from rule_helpers import print_rule_hierarchy

Note that, in this code, the term 'rule' shares the same notion with the term 'pattern' used in the paper.

3. Retraining & Evaluation

Run python to retrain multiple NRE models with different random seeds for all kinds of labels.

Run python to aggregate all evaluation results.

File Specifications


  • log.relXXX, the log for NRE model training
  • log.agentXXX, the log for agent network training and pattern generation
  • log.diagXXX, the log for the pattern refinement
  • log.eval_totalXXX, the log for the final total evaluation


  • erasure_policy_modelXXX, the checkpoint of the agent network;
  • rel_modelXXX, the checkpoint of the NRE model;
  • train_erasure_decision_modelXXX, the pickled file for agent's decisions;
  • rule_info_modelXXX, the pickled file storing pattern information derived from certain decision files;
  • rule_hierarchy_modelXXX, the pickled file storing the constructed pattern hierarchy merged from multiple pattern information sources.


Source code for ACL 2019 paper "DIAG-NRE: A Neural Pattern Diagnosis Framework for Distantly Supervised Neural Relation Extraction".




No releases published


No packages published