Batchnormtrack Flag setting for cifar10

[jizongFox]

Hi @xu-ji,
Thanks for this wonderful work. I am rerunning your code here and noticed that in the commands.txt, the setting with cifar10 is without --batchnorm_track, while most other commands are with this flag. I can understand that freezing the BN can be used in a finetuning setting but here apparently not the case. Can you tell me why the BN has been freezed for this particular setting for training Cifar10 from scratch?
Thanks for your help in advance.

[xu-ji]

Setting --batchnorm_track means each batchnorm module in the network has track_running_stats as True. So after calling net.eval(), while performing inference on the evaluation data, the batchnorm statistics are taken from training. If False the batchnorm statistics are taken from the evaluation data on the fly (docs). There's no particular reason to go with one or the other. It makes a negligible difference, e.g. 0.7% for CIFAR10, because training and test sets are the same for most of the datasets in the fully unsupervised setting.

The exception is STL, where the full training set contains a lot of distractor classes in the unlabelled images not present in test. This is why when setting --batchnorm_track for STL it makes sense to set --double_eval too, which makes an additional run through the test data (= training data for main output head) without any training on the IIC loss. This updates the batchnorm stats to the main output head's (= test) data prior to use in the evaluation.

[jizongFox]

Hi,
Thanks for the reply.
Here I would like to ask you for another details.

IIC/code/scripts/cluster/cluster_greyscale_twohead.py

Line 344 in 90f262e

assert (all_imgs.requires_grad and all_imgs_tf.requires_grad)

in this line, you let the input image to be differentiable, while in the cluster_greyscale.py, you haven't. Could you explain why here the gradient is required?

[xu-ji]

Hi, that's redundant, thanks for pointing it out. I've removed it.

[jizongFox]

Hi @xu-ji
See attached a training summary of mnist using your provided code.
Is that normal?

[xu-ji]

Yes, it is.

[jizongFox]

Is that normal to have an average acc of 84%?

[xu-ji]

Sorry, I skimmed over that second graph. It’s ok but not as good as my reported model. My trained model:

(By the way if you download the models you can see the plots and records.)

As you can see in that graph the average is 98.4. Other MNIST models I've trained have averaged at 96.6, 92.0, 92.5, 95.9.

[jizongFox]

If I understand correctly, you may run several experiments and choose to show the best one.

[xu-ji]

Yes, I ran a few experiments and show the best model.

[primecai]

Can we say that as long as the distribution of train and test set stays the same，setting this flag or not should not make too much difference？For example，if we split CIFAR10 into a 7:3 train-test partition，train on the train set and test on the unseen test set，using a batch size of 660 this flag should not affect much of the performance？

[xu-ji]

If the test batches' statistics are representative of the training batches' statistics (same class distribution, same input distribution, same size) and training is given enough time for batchnorm stats to reflect the latest features, then yes theoretically there should not be a material difference between taking the test time batchnorm statistics from training batches or test batches. In practice, there may be a small difference.