Skip to content
No description, website, or topics provided.
Java PostScript Makefile C HTML Roff Other
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
SL_models
config
data
lib
logs
out/production/main
output
scripts
serialized_data
src/main
target
.gitignore
README.md
pom.xml

README.md

Prerequisites

  • Have maven installed in your system
  • Run sh scripts/mvn_install.sh in your terminal to install the jar dependencies
  • Have Gurobi6.5.2 installed in your system and have env var GUROBI_HOME setup in your path. Please contact the author (qning2@illinois.edu) if you cannot find the official package for version 6.5.2. Note that if you only need the system output files, you can move forward without it.

Reproduce EMNLP Results

All the following cmd should be run from the root dir of the project, i.e., StructTempRel-EMNLP17/.

  • sh scripts/emnlp17_table3_line5.sh to generate the numbers on line 5 (SP+ILP), table 3.
    • The numbers on line 5 can be found at the end of logs/evaluation_process_20_0.01_0.0.txt (here it's slightly better than the reported values in the paper due to some minor changes).
      === Temporal Awareness Score ===
      Temporal Score  F1  P R
      67.5557 69.7597 65.4867 
      Overall Temporal Awareness Score (F1 score): 67.5557
      
    • The resulting .tml files can be found in output/pred_20_0.01_0.0/. The numbers above were generated by comparing these .tml files to the gold .tml files in data/TempEval3/Evaluation/te3-platinum/ using the TempEval-3 official evaluation script.
    • To get exactly the same results as in the paper, we have included the original system .tml files that generated those values, which you can find in output/pred_table3_SP+ILP. To evaluate it, run sh scripts/evaluate_test.sh table3_SP+ILP and go to the end of logs/evaluation_process_table3_SP+ILP, where you should see:
      === Temporal Awareness Score ===
      Temporal Score  F1  P R
      67.2406 69.0909 65.4867 
      Overall Temporal Awareness Score (F1 score): 67.240
      
  • sh scripts/emnlp17_table4_line6.sh to generate the numbers on line 6, table 4.
    • The numbers on line 6 can be found at the end of logs/evaluation_process_0.6_1.4_1.txt.
      === Temporal Awareness Score ===
      Temporal Score  F1  P R
      40.2624 35.5236 46.4602 
      Overall Temporal Awareness Score (F1 score): 40.2624
      
    • The system output .tml files can be found in output/pred_0.6_1.4_1/. The numbers above were generated by comparing these .tml files to the gold .tml files in data/TempEval3/Evaluation/te3-platinum/ using the TempEval-3 official evaluation script. If you simply want to evaluate it, run
      sh scripts/evaluate_test.sh 0.6_1.4_1
      
    • Note: What emnlp17_table4_line6.sh does is to (1) (re)generate those .tml files in output/pred_0.6_1.4_1/ and (2) run evaluate_test.sh to evaluate it. Therefore, you can also skip the generation step and run sh scripts/evaluate_test.sh 0.6_1.4_1 directly. This situation is the same as "line 13" below.
  • sh scripts/emnlp17_table4_line13.sh to generate the numbers on line 13, table 4.
    • The numbers on line 13 can be found at the end of logs/evaluation_process_chambers_codl_tdtest_0.3_1.4_1.txt.
      === Temporal Awareness Score ===
      Temporal Score  F1  P R
      48.5253 45.567  51.8944 
      Overall Temporal Awareness Score (F1 score): 48.5253
      
    • The resulting .tml files can be found in output/Chambers/codl/TD-Test/0.3_1.4_1/. The numbers above were generated by comparing these .tml files to the gold .tml files in output/Chambers/gold/ using the TempEval-3 official evaluation script. If you simply want to evaluate it, run
      sh scripts/evaluate_general.sh ./output/Chambers/gold ./output/Chambers/codl/TD-Test/0.3_1.4_1/ chambers_codl_tdtest_0.3_1.4_1
      

Citation

Qiang Ning, Zhili Feng, Dan Roth, "A structured learning approach to temporal relation extraction", EMNLP 2017 (paper, slides (pdf), slides (ppt))

@inproceedings{NingFeRo17,
    author = {Qiang Ning and Zhili Feng and Dan Roth},
    title = {A Structured Learning Approach to Temporal Relation Extraction},
    booktitle = {EMNLP},
    month = {9},
    year = {2017},
    address = {Copenhagen, Denmark},
    url = "http://cogcomp.org/papers/NingFeRo17.pdf",
    funding = {AI2, IBM-ILLINOIS C3SR, DARPA, ARL},
}
You can’t perform that action at this time.