Unable to reproduce the result given in the paper

[yijunCai]

Hi Zhang,
I was trying to reproduce the result on Caltech test set, using the pretrained model given for the demo . I made a few modefications on script_rpn_pedestrian_VGG16_caltech.m, simply skip the training part so that I can use the pretrained model to do evaluation directly. I got only 10.4% MR on Caltech test set, compared to 9.6% in your paper. Is there any difference between the pretrained model given for the demo and the one you used in the paper? Or any other explanation for my result?
Thanks a lot!

[zhangliliang]

Hi,

It is reasonable to get a ~10% MR.

[yijunCai]

Thanks for your reply! Would you mind my asking another question here?
When I first run the evaluation, I use DeepTest_otf_trans_ratio（） for scoring, and that gave me 10.10% MR. Later I change to dbEval_RPNBF for scoring, resulting a 10.42% MR. Although the difference between those two results are small, does it indicate that one of the evaluation script is not that accurate? I notice that the former one is from Pitor's toolbox, and the latter from the evaluation code given by Caltech, so both of them can serve as official evaluation scripts, and the results should be exactly the same?

[zhangliliang]

You are right. The evaluation code from the dollar's toolbox and the official evaluation code from the Caltech website are slightly different, since they use different parameters and settings.

[yijunCai]

I see. Thanks again!