Guarin lig 3062 add mocov2 imagenet benchmark

[guarin]

Changes

• Add MoCoV2 to imagenet benchmarks
• Add gather_distributed option to MemoryBankModule
• Move memory_bank.py from lightly/loss to lightly/modules/models
  • This makes conceptually more sense as the memory bank is its own module, independent from loss functions
  • It was required to avoid circular imports (we import MemoryBankModule in lightly/modules/models/nn_memory_bank.py)

Closes #1288
Closes #1018

Results

Ran a benchmark using the code from #1009:

max val_online_cls_top1: 0.5466600060462952
max val_online_cls_top5: 0.7995399832725525
max linear val_top1: 0.6080999970436096
max linear val_top5: 0.835860013961792
max finetune val_top1: 0.739300012588501
max finetune val_top5: 0.9166200160980225
knn val_top1: 0.4095799922943115
knn val_top5: 0.711139976978302

The results are slightly worse than expected. I would expect similar performance as SimCLR which has 63.2% linear top1 accuracy while mocov2 has only 60.8 here. I figured out that the transform parameters are slightly different in the original code and made an additional PR to update them: #1441. Will run a new benchmark with those parameters.

New results using this branch:

max val_online_cls_top1: 0.5602999925613403
max val_online_cls_top5: 0.808139979839325
max linear val_top1: 0.6151000261306763
max linear val_top5: 0.8411200046539307
max finetune val_top1: 0.7437400221824646
max finetune val_top5: 0.9182599782943726
knn val_top1: 0.41255998611450195
knn val_top5: 0.7188000082969666

It is still slightly worse than SimCLR which has 63.2% linear top 1, while MoCov2 has 61.5%. I think we can merge it like that, I can't find any differences atm. A possible reason might be that we use the SimCLR version with optimized lr for smaller batch sizes.

Review

When reviewing please only consider these files:

• benchmarks/imagenet/resnet50/mocov2.py
• README.md
• benchmarks.rst

The other files have already been reviewed when their respective PRs were merged into this branch.

[codecov]

Codecov Report

Attention: 10 lines in your changes are missing coverage. Please review.

Comparison is base (caf72c5) 85.24% compared to head (bc8adad) 85.33%.
Report is 5 commits behind head on master.

Files	Patch %	Lines
lightly/models/utils.py	0.00%	7 Missing ⚠️
lightly/loss/memory_bank.py	0.00%	1 Missing ⚠️
lightly/models/modules/memory_bank.py	98.03%	1 Missing ⚠️
lightly/utils/benchmarking/metric_callback.py	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1291      +/-   ##
==========================================
+ Coverage   85.24%   85.33%   +0.09%     
==========================================
  Files         130      131       +1     
  Lines        5517     5566      +49     
==========================================
+ Hits         4703     4750      +47     
- Misses        814      816       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.