Reproducing source-only results on DomainNet

[robbiejones96]

Hi SENTRY authors, congrats on the impressive results from your recent paper and thank you for sharing your code with us! As a starting point I am trying to reproduce the DomainNet source-only Sketch -> Real results, which in your paper is listed as 77% average per-class accuracy. I'm currently running this command with your repo to attempt to reproduce this performance:

python train.py --id source-only-domainnet --source sketch --target real --img_dir <local path to DomainNet> --LDS_type natural --use_cuda True --cnn ResNet50FS --optimizer SGD --lr 1e-2 --wd 5e-4 --num_epochs 100 --batch_size 16 --temperature 0.05 --l2_normalize True

All the hyperparameters other than num_epochs I believe I have taken from your paper, but I may have misread/missed something. The current target average per-class accuracy I'm getting is 66%. Would you be able to share the exact command/hyperparameters you used to obtain the 77% transfer accuracy from sketch to real? Thank you so much!

[tarun005]

Using similar hyperparameters and software versions given in code, I was able to reproduce the result.

[virajprabhu]

Great, glad you could reproduce it!

[robbiejones96]

Cool! @tarun005 Would you be able to paste the command you ran here, and/or compare it to the one that I posted in my original issue? Thanks for sharing your update!

[robbiejones96]

Nevermind, it was silly of me to manually input the hyperparameters instead of running from the sentry.yml config. This has made all the results reproducible. Cheers!