Train and evaluate vanilla SimCLR

[hyun78]

I want to reproduce the experiment with the vanilla SimCLR in the paper. (Table 7)
I followed the instructions in the main page;

I run the below command to train vanilla SimCLR. (Since I have trouble in installing torchlars package, I changed the optimizer adam)
CUDA_VISIBLE_DEVICES=0,1,2,3 python3 -m torch.distributed.launch --nproc_per_node=4 train.py --dataset cifar10 --model resnet18 --mode simclr --shift_trans_type rotation --batch_size 32 --one_class_idx 0 --optimizer adam

After that, I tried the evaluation command.

python3 eval.py --mode ood_pre --dataset cifar10 --model resnet18 --ood_score simclr --shift_trans_type rotation --print_score --ood_samples 10 --resize_factor 0.54 --resize_fix --one_class_idx 0 --load_path {LOAD_PATH}

But I got errors.
Here are detailed logs.

Pre-compute global statistics...
axis size: 5000 5000 5000 5000
weight_sim: 1.0000
weight_shi: 0.0000
Pre-compute features...
Compute OOD scores... (score: simclr)
Traceback (most recent call last):
File "eval.py", line 23, in
train_loader=train_loader, simclr_aug=simclr_aug)
File "/home/hyun78/aya/CSI/evals/ood_pre.py", line 84, in eval_ood_detection
scores_id = get_scores(P, feats_id, ood_score).numpy()
File "/home/hyun78/aya/CSI/evals/ood_pre.py", line 121, in get_scores
score += (f_sim[shi] * P.axis[shi]).sum(dim=1).max().item() * P.weight_sim[shi]
IndexError: list index out of range

+) I think the vanilla version of SimCLR should be the same as the original SimCLR paper, but your training codes include shift layers. I don't make sure it is okay to include the additional backward steps (line74~80 in training/unsup/simclr.py) and shift layers(line100 in common/train.py).

[jihoontack]

Hi, thank you for your interest in our work!

I think you should run the SimCLR code with --shift_trans_type none and also test with the same option (since as you already mentioned, SimCLR does not require shifting transformation classification layer).

If you don't want to train the model again, maybe forcing the P.K_Shift=1 (after get_shift_module function) might resolve the problem (not sure about this...).

And by the way, I recommend lars optimizer since I had never tried with adam and I am not sure about the performance. When I used sgd rather than lars optimizer, the overall performance had dropped around 1~2%. Also, If you are training SimCLR, you should set the batch size to 128 (then it will make 4 * 128 in total, for CSI it was 32 since we multiply with the number of rotation transformation).

Thank you again for your interest.

[hyun78]

Thanks a lot!
I will try --shift_trans_type none with batch size 128
CUDA_VISIBLE_DEVICES=0,1,2,3 python3 -m torch.distributed.launch --nproc_per_node=4 train.py --dataset cifar10 --model resnet18 --mode simclr --shift_trans_type none --batch_size 128 --one_class_idx 0 --optimizer adam --suffix new

[jihoontack]

If you have any problems, feel free to reopen the issue.
Thank you.

[hyun78]

Hi, I run the following code.
for training:
CUDA_VISIBLE_DEVICES=0,1,2,3 python3 -m torch.distributed.launch --nproc_per_node=4 train.py --dataset cifar10 --model resnet18 --mode simclr --shift_trans_type none --batch_size 128 --one_class_idx 0 --optimizer adam --suffix new
for evaluation:
python3 eval.py --mode ood_pre --dataset cifar10 --model resnet18 --ood_score simclr --shift_trans_type none --print_score --ood_samples 10 --resize_factor 0.54 --resize_fix --one_class_idx 0 --load_path <load_path>

And I got a different result.

I think it should be 87.9 (in the paper, Table 7a). It may be lower due to the optimizer, but it is too low.

1. Is there any problem with my command?
2. Could you provide sample codes to reproduce the Table 7a result?

Thanks