Issue with Precision and Recall

[vinceRicchiuti]

Hi all, i don't know if i can use this post in this way to solve my problem, but i'm using rival in my thesis work to evaluate the predictions of multiple recommender systems; my issue is then when i try to compute precision and recall of my results, recall is always set to 1.0 and precision is not correct (i made compute by hand and the results are diffents) i try 2 different configurations of test, the former with binarized score and the latter with prediction confidence score, but noone give me the correct results; i'll write the contents of the two files i use to do the test; i think i'm loosing something so the meaning of the test change in my case.

The file are in the format : idUser (tab) idItem (tab) score
Left side: predicitions file | Right side: test file

First Configuration
2 13 1 | 2 13 1
2 19 1 | 2 19 0
2 50 1 | 2 50 1
2 251 1 | 2 251 1
2 257 1 | 2 257 1
2 279 0 | 2 279 1
2 280 0 | 2 280 0
2 281 0 | 2 281 0
2 290 0 | 2 290 0
2 292 1 | 2 292 1
2 297 1 | 2 297 1
2 298 1 | 2 298 0
2 299 0 | 2 299 1
2 301 1 | 2 301 1
2 303 1 | 2 303 1
2 307 1 | 2 307 0
2 308 0 | 2 308 0
2 312 0 | 2 312 0
2 313 1 | 2 313 1
2 314 0 | 2 314 0
2 315 1 | 2 315 0
2 316 1 | 2 316 1

Precision: 0.5454545454545454
Recall: 1.0

Second Configuration

2 13 0.5328112138740229 | 2 13 1
2 19 0.8095414650648188 | 2 19 0
2 50 0.9888651730555137 | 2 50 1
2 251 0.8796398906715018 | 2 251 1
2 257 0.8994154288885712 | 2 257 1
2 279 0.2031462463186414 | 2 279 1
2 280 0.137729028181314 | 2 280 0
2 281 0.12677275323108753 | 2 281 0
2 290 0.12267777345041282 | 2 290 0
2 292 0.6708907647085832 | 2 292 1
2 297 0.769468001849272 | 2 297 1
2 298 0.878495414483143 | 2 298 0
2 299 0.09460751859396674 | 2 299 1
2 301 0.5983346178722854 | 2 301 1
2 303 0.8094825203837621 | 2 303 1
2 307 0.7223299632360657 | 2 307 0
2 308 0.11311986767160687 | 2 308 0
2 312 0.1279286169284483 | 2 312 0
2 313 0.9889272842775845 | 2 313 1
2 314 0.0026441894503800604 | 2 314 0
2 315 0.955807550363639 | 2 315 0
2 316 0.965191057772059 | 2 316 1

Precision: 0.5454545454545454
Recall: 1.0

The prediction has a confusion matrix of

TP=10
FP=4
FN=2
TN=6

so metrics should be
precision: 0.714
recall: 0.833

As you can see they are very different. Can u tell me if i'm doing something wrong with the use of your framework? Thanks in advance, Vincenzo.

[abellogin]

Hi Vincenzo,

I will take a look at it as soon as possible. Can you tell us how you are calling the precision and recall metrics? If you are using your code, it will be useful to know how you are creating and printing the metrics. If you are using a script, it would also be helpful to know the commands you are using.

Thanks,
Alejandro

[vinceRicchiuti]

yes sure, This is my code:

....
SimpleParser sp = new SimpleParser();
DataModel dm = sp.parseData(new File("path_of_my_predictions_file"));
DataModel dmtest = sp.parseData(new File("path_of_my_test_file"));

Precision pr = new Precision(dm, dmtest);
pr.compute();
System.out.println(pr.getValue());
Recall rc = new Recall(dm, dmtest);
rc.compute();
System.out.println(rc.getValue());
....

[abellogin]

Hi Vincenzo, sorry it took so long, but I thought it was going to be something related with how you called the metrics, so I needed to prepare a test case. In the end, the problem seems to be with your definition of precision (and recall): instead of using the definition from the classification literature, you should look at the one used in information retrieval (here), where the total number of retrieved documents are considered in the denominator. In your case, this means: 12 (positive) / 22 (retrieved) = 0.54.

Let me know if you have any comments,
Alejandro