Settings for full ImageNet (not subset)

[mgwillia]

Basically the title: I see the settings for SCAN for the subsets. I notice they are all the same. Shall I use the same settings (batch size, LR, entropy weight, etc.) for regular ImageNet in order to reproduce your results? And which MoCo backbone did you originally use- I'm having trouble getting above ~33% with the backbone I found (SimCLR, not MoCo, so I wanted to try yours).

[wvangansbeke]

Hi @mgwillia,

We used a ResNet-50 with MoCo pretrained weights (800 epochs). Then we ran SCAN with multiple heads (10) and a very large batchsize. In order to make it easier, you can only train the last linear layer and keep the SimCLR augmentation as in configs/scan/imagenet_eval.yml. Try a few settings: LR=5, batch size=4k, num_neighbors=50, entropy_weight=5. You can play around with it but you should get close to 40%. (Getting self-labeling to work requires very large batch sizes, ideally more than 4k).

[mgwillia]

Ok. I think the SimCLR I'm using is actually comparable to that MoCo then (it's also trained for 800 epochs). The batch size you suggest is about 4x larger than my assumption though, so I'll change that first and work from there. Thanks for the reply!

Also, wouldn't accuracy need to be significantly >50% in order for self-labeling to work? I thought it relied on fairly confident predictions. Did you ever do a run where self-labeling was helpful?

[wvangansbeke]

Yes you're correct. It seems difficult to get self-labeling working for the 1000 ImageNet classes. However, overclustering makes sense in this case (+4k clusters). This will make the clusters more fine-grained, which should result in a higher purity per cluster.

[mgwillia]

Have you run that experiment (4k+ batch size and 4k+ num clusters for imagenet)? If so, would you mind sharing results?

[wvangansbeke]

No extra numbers yet. I just believe that should work since I've seen other papers use these settings. 4k+ clusters requires a lot of memory and I haven't been able to fully experiment with it. We were only able to finish 1 experiment with 1000 clusters for the paper.