Question about reproducing results on CIFAR

[shaoyijia]

Hi,

Thank you so much for opening the source code!

I’m trying to reproduce your results on CIFAR using ResNet18 according to your paper. Since there is no checkpoints in the repo, I trained a ResNet18 on CIFAR10 by myself and its IND acc is 93.31%. I used this model to do OOD detection on CIFAR10 - SVHN, CIFAR10 - LSUN Crop, CIFAR10 - LSUN Resize, but failed to reproduce the results in your paper.

Specifically, I ran the following commands to do the experiment,

python compute_threshold.py --in-dataset CIFAR-10 --model-arch resnet-18
python eval.py --threshold {result computed by compute_threshold.py} --in-dataset CIFAR-10 --out-datasets SVHN LSUN LSUN_resize

I thought the problem may lie in the process of training ResNet18, so I retrained the model strictly following the experimental details in the paper, i.e. ,

For both CIFAR-10 and CIFAR-100, the models are trained for 100 epochs. The start learning rate is 0.1 and decays by a factor of 10 at epochs 50, 75, and 90.

In my training, I set batch_size=1024 and used Adam as the optimizer.

The ResNet18 I got on CIFAR10 has an IND accuracy of 90.47% but the ResNet18 trained on CIFAR100 failed to converge which only has an IND accuracy of 1% (I think lr=0.1 at the beginning is too large for CIFAR100 training, people usually choose a smaller lr to train it). Also, using the new ResNet18 trained on CIFAR10, compute_threshold.py returns 0.0 which makes me even more confused.

I'm wondering if I did anything wrong or misunderstood something. It'll be extremely helpful if you could give me some guidance on how to reproduce the results on CIFAR.

Thanks for your time and help!

[sunyiyou]

Hi, thanks for your interest in our work.

I have updated the code repo which may support running the command you showed.

Here is the link for the checkpoint of CIFAR you may want to use.
https://drive.google.com/drive/folders/1XsHLo-qS92kmLpRzPf7UyINL5uL8nwg3?usp=sharing