TAR Stopping Point Processes

Repository for the code of the paper "Stopping Methods for Technology Assisted Reviews based on Point Processes" by Mark Stevenson and Reem Bin-Hezam

Content

Notebooks for the following:

• Run Point Processes
• Run Point Processes for Multiple Runs & Visulaization
• Baselines vs Point Process Across Multiple Datasets Visualization
• Per Topic Visualization
• T-Test Analysis

Data

Ranking and Relevance files of:

• CLEF (2017, 2018, 2019)

• TREC (TR , Legal)

• Link to complete datasets ranking inlcuding TREC TR & Legal: download (~305MB).

  • add them to data/rankings sub-folder

• Links to CLEF qrels for Baselines Comparison: CLEF (2017, 2018, 2019)

  • merge each dataset topics qrels into one file and add them to sub-folder: data/qrels

• Links to CLEF qrels for for Multiple Runs: CLEF (2017, 2018)

  • for CLEF2017 , add qrel_abs_test.txt to sub-folder: /RunMultipleRankings/CLEFData/clef_runs/2017/relevance
  • for CLEF2018 , add full.test.abs.2018.qrels to sub-folder: /RunMultipleRankings/CLEFData/clef_runs/2018/relevance

• Links to TREC qrels (needs access permission): TR qrels , Legal qrels.

  • add them to sub-folder: data/qrels
  • guidlines on how to get access and usage agreements: TR , Legal