Replicating paper's results

[bartmch]

Hi, I was just trying to replicate the ROC's you presented in your paper using the test_set you described in prepare_data.py. I'm executing trainer.py with your proposed settings:

                "--mode","test",
                "--checkpoint","path_to_CovidAID_transfered.pth.tar",
                "--cm_path","path_to_plots/cm_best",
                "--roc_path","path_to_plots/roc_best",
                "--combine_pneumonia"

I see that default batch size is 64 which brings the total inputs.shape to [640,3,224,224] (default 10 crops). This doesn't fit on GPU's memory. I was wondering what other settings you used to validate on the test set? E.g.: did you added a specific batch size "--bs","1" or did you use the distributed processing - if so can you explain to me how to use this in your code?
Thanks!

[arpanmangal]

batch_size at test time does not matter. You can use any size which fits in your GPU memory. For training, see sec. 3.3 of our paper for hyper-parameters.

[bartmch]

I just lowered the batch size and it runs fine (testing bs shouldn't matter - you're right).
A more urgent question: I created the ROCs and CMs for the data split in the code (which reflects the paper). When I want to replicate the results using:
["--mode","test","--freeze","--checkpoint","path_to/CovidAID_transfered.pth.tar","--bs","8","--cm_path","path_to/plots/cm_best","--roc_path","path_to/plots/roc_best","--combine_pneumonia"]
I get the following ROC and CM:

[arpanmangal]

Which pre-trained model did you use? Did you use the correct model?

[bartmch]

Which pre-trained model did you use? Did you use the correct model?

I followed your steps. The only difference is that I had to make the model torch>1.0 compatible but I didn't change any weights or so, only the names of the layers - that shouldn't really matter right?
FYI: test.txt

[bartmch]

Apologies - I just figured out I was pointing to the wrong trained network! Everything looks good now.