Poor results of Learning to Rank model

[Gu-Youngfeng]

ReConfig utilizes the library RankLib.jar to re-rank the original predicted ranking list outputed by the rank-based method.
However, the results shows that learning to rank model cannot improve the accuracy at all.

General Usage of RankLib
https://sourceforge.net/p/lemur/wiki/RankLib%20How%20to%20use/#ranking.

The corresponding code snippet is,

def build_l2r_model(ranker):    ### Line 736 in reconfig.py ###
    ...
    cmd_line = "java -jar RankLib.jar -train " + txtfile + " -ranker " + str(ranker) + \
    " -save " + folderpath + "/mymodel_" + name_index + ".txt"
    os.system(cmd_line)
    ...

Here are 9 rankers provided by RankLib.jar.

Ranker ID	Ranker Name
0	MART
1	RankNet
2	RankBoost
3	AdaRank
4	Coordinate Ascent
6	LambdaMART
7	ListNet
8	Random Forests

Should we tune some parameters or try different learning to rank rankers? For example, we can change the variable ranker from 0 to 6 (default is 2), or we can add an argument such as -metric2t to set metric to optimize on the training data (default is ERR@10).

[Gu-Youngfeng]

According to the follow-up experimental results, I found that the method classification performed well among comparative methods. Here is a results on 18 datasets using 4 comparative methods.

• M1: ReConfig
• M2: Rank-based
• M3: Classification
• M4: Random Deletion

TABLE I. RDTie on 18 datasets using four comparative methos

Datasets	M1 M2 M3 M4	M1 M2 M3 M4	M1 M2 M3 M4	M1 M2 M3 M4
	Top-1	Top-3	Top-5	Top-10
Apache	8.0 8.9 8.5 9.6	4.0 3.8 3.3 4.2	2.7 2.1 1.6 2.3	1.5 0.8 0.6 1.2
BDBJ	11.3 10.9 10.7 10.5	6.3 5.8 5.8 5.6	4.1 3.6 3.8 3.5	2.1 1.4 1.7 1.5
clasp	27.7 31.1 20.9 29.5	10.9 12.5 9.1 10.2	7.7 7.8 5.1 7.1	6.4 4.3 3.9 5.6
lrzip	24.1 24.4 18.6 25.0	10.0 10.4 7.8 9.7	6.1 6.0 4.7 6.7	4.1 2.9 2.8 5.0
noc-obj1	11.4 12.2 9.9 14.2	5.2 4.5 3.5 4.9	3.5 2.4 1.9 2.9	2.8 1.0 1.2 1.9
noc-obj2	10.6 10.3 6.7 9.3	4.2 4.5 3.0 4.4	3.2 2.8 2.2 3.4	2.3 1.4 1.3 2.8
snw-obj1	7.4 7.5 6.6 7.3	3.3 3.2 2.8 3.2	2.5 2.3 2.0 2.3	1.2 0.9 0.8 0.9
snw-obj2	6.9 6.8 6.9 6.6	2.2 2.2 2.2 2.1	1.0 1.0 1.0 1.0	0.4 0.5 0.3 0.4
sqlite	183.8 202.5 217.7 196.3	95.4 100.4 97.8 103.1	67.3 66.8 65.2 71.1	40.3 36.3 42.1 41.6
wc+rs-3d-c4-obj1	16.7 16.7 15.7 16.1	6.3 6.4 5.8 5.8	4.4 4.3 4.0 4.0	2.2 1.8 1.7 1.8
wc+rs-3d-c4-obj2	4.7 5.0 4.4 4.7	2.0 2.2 1.9 2.1	1.4 1.4 1.3 1.5	0.7 0.5 0.6 0.8
wc+sol-3d-c4-obj1	18.8 18.6 18.3 19.0	7.4 7.9 7.3 7.9	5.2 5.3 5.1 5.4	2.9 2.6 2.7 3.2
wc+sol-3d-c4-obj2	4.7 4.7 4.7 4.6	1.9 2.0 1.8 1.9	1.2 1.2 1.0 1.2	0.7 0.5 0.5 0.6
wc+wc-3d-c4-obj1	14.8 15.5 13.7 15.3	3.7 4.2 4.5 4.7	2.0 2.1 2.7 3.0	1.1 0.7 1.5 1.6
wc+wc-3d-c4-obj2	6.0 6.1 5.8 6.4	2.6 2.5 2.5 2.8	1.7 1.5 1.7 1.9	1.1 0.5 0.9 1.3
wc-3d-c4-obj1	37.3 38.8 28.0 37.0	11.7 15.4 9.5 13.3	5.7 9.4 4.9 7.8	3.2 4.8 2.9 4.8
wc-3d-c4-obj2	23.3 24.4 17.1 25.8	10.2 12.4 6.9 12.5	7.2 8.6 4.1 8.7	5.1 5.4 2.1 5.7
WGet	27.1 25.9 27.0 25.1	11.9 11.2 11.9 11.8	7.1 6.7 7.1 7.3	3.6 3.5 3.3 3.5

Throught the Table 1, we can find that M3 can achieve the lowest measurement (i.e., RDTie) in the most of datasets under the condition of Top-1, Top-3, Top-5, Top-10. Can we draw a conclusion that Classification is the best choose to solve the tie issue problem in configurable systems?