Add example with Swin training

[MeshchaninovViacheslav]

Project goals:

• add an exemplar code for imagenet classification, e.g. examples/imagenet
• figure out the best way to scale imagenet training via hivemind

Current status:

• choose base code
  • currently building on https://github.com/pytorch/examples/blob/main/imagenet/main.py
• choose the model
  • review ViT, DEIT, CAIT
  • optional: SWIN V1 and SWIN V2
    • note: Cristoph says that SWIN-V2 is currently high priority for TIMM , we could reuse their code or hacks
  • optional: MPViT ( suggested by @christophschuhmann )
  • optional: CoAtNet (suggested by @avecplezir )
• reproduce base model w/o hivemind
  • use hyperparams from papers
  • match val accuracy
  • figure out how to train quickly_enough to iterate
• training with hivemind.Optimizer
  • at least make sure the loss goes down
  • figure out how we log things
• compare variations in training regimen, at least the following
  • LAMB with large batch size, params from this paper
  • parameter averaging, params from this paper
  • any other suggestions? :)
• prep hivemind/examples/imagenet
  • add README with basic instructions
  • try to reduce the code size (for readability)
  • include some tool for logging and checkpointing

[codecov]

Codecov Report

Merging #459 (7a72754) into master (762f116) will decrease coverage by 0.87%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #459      +/-   ##
==========================================
- Coverage   84.01%   83.14%   -0.88%     
==========================================
  Files          78       81       +3     
  Lines        7932     8061     +129     
==========================================
+ Hits         6664     6702      +38     
- Misses       1268     1359      +91     

Impacted Files	Coverage Δ
hivemind/averaging/matchmaking.py	83.63% <0.00%> (-4.77%)	⬇️
hivemind/averaging/allreduce.py	87.21% <0.00%> (-4.57%)	⬇️
hivemind/optim/grad_averager.py	92.15% <0.00%> (-1.66%)	⬇️
hivemind/averaging/partition.py	97.25% <0.00%> (-1.10%)	⬇️
hivemind/optim/progress_tracker.py	97.80% <0.00%> (-1.10%)	⬇️
hivemind/compression/__init__.py	100.00% <0.00%> (ø)
hivemind/optim/power_sgd_averager.py	37.77% <0.00%> (ø)
hivemind/utils/math.py	44.44% <0.00%> (ø)
hivemind/compression/serialization.py	100.00% <0.00%> (ø)
hivemind/optim/optimizer.py	62.42% <0.00%> (+0.14%)	⬆️
... and 4 more