Skip to content

Code and Supplementary Material to the Paper: Pairwise Learning to Rank by Neural Networks Revisited: Reconstruction, Theoretical Analysis and Practical Performance

Notifications You must be signed in to change notification settings

makoeppel/direct-ranker

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Loading data

For loading the data go into the folders data and run:

source get_data.sh

For MQ2007/8 we remove all the zeros in the queries with the script delete_zeros.py.

Virtualenv & Compile AdaRank

# install all the python requirements
pip install virtualenv
virtualenv DirectRanker
source DirectRanker/bin/activate
pip install -r /path/to/requirements.txt

# install the Rust libary for AdaRank
# first install Rust: https://www.rust-lang.org/learn/get-started
pip install maturin
# install it in the virtualenv
cd rustlib
maturin develop -r
# install it in your global python env
maturin build
pip install target/wheels/*.whl

Test the code

./test_all.sh

Figure 4

For generating the plots from Figure 2 one should run the following scripts in the folder fig4:

  1. Generating the synthetic data:
python gen_fig4a.py  # for Fig. 4 a)
python gen_fig4b.py  # for Fig. 4 b)
python gen_fig4c.py  # for Fig. 4 c)
python gen_fig4d.py  # for Fig. 4 d)

Figure 5

For generating the plots from Figure 5 one should run the following scripts in the folder fig5:

  1. Run the model tests for the synthetic data:
python training_label.py
python training_size.py
  1. Run for the MSLR-WEB10K data:
python training_label_mslr.py
python training_size_mslr.py
  1. Run the plot script:
python plot_fig5.py

Figure 6

For generating the plots from Figure 6 one should run the following scripts in the folder fig6:

  1. Generating the synthetic data:
ln -s ../Rankers .
python plot_sorted_list.py

Table 1 and Figure 2/3

For generating table 1 one should run the following scripts in the folder table1:

  1. Running the ranklib models
python create_runscripts.py --data ../data/MQ2007 --jarpath .. --resultsdir results_ranklib --datalabel MQ2007 # create run scripts for ranklib and MQ2007
python create_runscripts.py --data ../data/MQ2008 --jarpath .. --resultsdir results_ranklib --datalabel MQ2008 # create run scripts for ranklib and MQ2008
python create_runscripts.py --data ../data/MSLR-WEB10K --jarpath .. --resultsdir results_ranklib --datalabel MSLR-WEB10K # create run scripts for ranklib and MSLR-WEB10K
python run_jobs.py # run the ranklib models
  1. Running the DirectRankerV1 model
ln -s ../Rankers .
python gridsearch.py --path ../data/MQ2007/ --model DirectRankerV1 --jobs 5 --data MQ2007
python gridsearch.py --path ../data/MQ2008/ --model DirectRankerV1 --jobs 5 --data MQ2008
python gridsearch.py --path ../data/MSLR-WEB10K/ --model DirectRankerV1 --jobs 5 --data MSLR10K
  1. Running the tensorflow V2 models
python gridsearch.py --path ../data/MQ2007/ --model RankNet --jobs 5 --data MQ2007 --ttest 1 --binary 0
python gridsearch.py --path ../data/MQ2007/ --model RankNet --jobs 5 --data MQ2007 --ttest 1 --binary 1
python gridsearch.py --path ../data/MQ2008/ --model RankNet --jobs 5 --data MQ2008 --ttest 1 --binary 0
python gridsearch.py --path ../data/MQ2008/ --model RankNet --jobs 5 --data MQ2008 --ttest 1 --binary 1
python gridsearch.py --path ../data/MSLR-WEB10K/ --model RankNet --jobs 5 --data MSLR10K --ttest 1 --binary 0
python gridsearch.py --path ../data/MSLR-WEB10K/ --model RankNet --jobs 5 --data MSLR10K --ttest 1 --binary 1

python gridsearch.py --path ../data/MQ2007/ --model ListNet --jobs 5 --data MQ2007 --ttest 1 --binary 0
python gridsearch.py --path ../data/MQ2007/ --model ListNetOri --jobs 5 --data MQ2007 --ttest 1 --binary 0
python gridsearch.py --path ../data/MQ2007/ --model ListNet --jobs 5 --data MQ2007 --ttest 1 --binary 1
python gridsearch.py --path ../data/MQ2007/ --model ListNetOri --jobs 5 --data MQ2007 --ttest 1 --binary 1
python gridsearch.py --path ../data/MQ2008/ --model ListNet --jobs 5 --data MQ2008 --ttest 1 --binary 0
python gridsearch.py --path ../data/MQ2008/ --model ListNetOri --jobs 5 --data MQ2008 --ttest 1 --binary 0
python gridsearch.py --path ../data/MQ2008/ --model ListNet --jobs 5 --data MQ2008 --ttest 1 --binary 1
python gridsearch.py --path ../data/MQ2008/ --model ListNetOri --jobs 5 --data MQ2008 --ttest 1 --binary 1
python gridsearch.py --path ../data/MSLR-WEB10K/ --model ListNet --jobs 5 --data MSLR10K --ttest 1 --binary 0
python gridsearch.py --path ../data/MSLR-WEB10K/ --model ListNetOri --jobs 5 --data MSLR10K --ttest 1 --binary 0
python gridsearch.py --path ../data/MSLR-WEB10K/ --model ListNet --jobs 5 --data MSLR10K --ttest 1 --binary 1
python gridsearch.py --path ../data/MSLR-WEB10K/ --model ListNetOri --jobs 5 --data MSLR10K --ttest 1 --binary 1

python gridsearch.py --path ../data/MQ2007/ --model DirectRanker --jobs 5 --data MQ2007 --ttest 1 --binary 0
python gridsearch.py --path ../data/MQ2007/ --model DirectRanker --jobs 5 --data MQ2007 --ttest 1 --binary 1
python gridsearch.py --path ../data/MQ2008/ --model DirectRanker --jobs 5 --data MQ2008 --ttest 1 --binary 0
python gridsearch.py --path ../data/MQ2008/ --model DirectRanker --jobs 5 --data MQ2008 --ttest 1 --binary 1
python gridsearch.py --path ../data/MSLR-WEB10K/ --model DirectRanker --jobs 5 --data MSLR10K --ttest 1 --binary 0
python gridsearch.py --path ../data/MSLR-WEB10K/ --model DirectRanker --jobs 5 --data MSLR10K --ttest 1 --binary 1

python gridsearch.py --path ../data/MQ2007/ --model AdaRank --jobs 1 --data MQ2007 --ttest 1 --binary 0
python gridsearch.py --path ../data/MQ2007/ --model AdaRank --jobs 1 --data MQ2007 --ttest 1 --binary 1
python gridsearch.py --path ../data/MQ2008/ --model AdaRank --jobs 1 --data MQ2008 --ttest 1 --binary 0
python gridsearch.py --path ../data/MQ2008/ --model AdaRank --jobs 1 --data MQ2008 --ttest 1 --binary 1
python gridsearch.py --path ../data/MSLR-WEB10K/ --model AdaRank --jobs 1 --data MSLR10K --ttest 1 --binary 0
python gridsearch.py --path ../data/MSLR-WEB10K/ --model AdaRank --jobs 1 --data MSLR10K --ttest 1 --binary 1

python gridsearch.py --path ../data/MQ2007/ --model LambdaMart --jobs 5 --data MQ2007 --ttest 1 --binary 0
python gridsearch.py --path ../data/MQ2007/ --model LambdaMart --jobs 5 --data MQ2007 --ttest 1 --binary 1
python gridsearch.py --path ../data/MQ2008/ --model LambdaMart --jobs 5 --data MQ2008 --ttest 1 --binary 0
python gridsearch.py --path ../data/MQ2008/ --model LambdaMart --jobs 5 --data MQ2008 --ttest 1 --binary 1
python gridsearch.py --path ../data/MSLR-WEB10K/ --model LambdaMart --jobs 5 --data MSLR10K --ttest 1 --binary 0
python gridsearch.py --path ../data/MSLR-WEB10K/ --model LambdaMart --jobs 5 --data MSLR10K --ttest 1 --binary 1
  1. Generating the Table / Friedman Test
python get_results.py

Table 2

For generating table 2 one should run the following scripts in the folder table2:

./run_ranknet_eval.sh
python ranknet_eval.py --plot 1

Table B1

For generating table B1 one should run the following scripts in the folder tableB1:

./run_time_ana.sh

The different time results for RankLib are printed in the terminal while the results for the DirectRanker and RankNet are stored in an output folder.

About

Code and Supplementary Material to the Paper: Pairwise Learning to Rank by Neural Networks Revisited: Reconstruction, Theoretical Analysis and Practical Performance

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 90.5%
  • Shell 7.7%
  • Rust 1.8%