[FEATURE] Support EfficientViT

[Randl]

Add models from
EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention https://arxiv.org/abs/2305.07027
https://github.com/microsoft/Cream/tree/main/EfficientViT

Looks like fast and high-quality models, would be nice to have them in timm

[rwightman]

@Randl I noticed that one, also related are their mini/tiny vit. Look like reasonable arch, blend of LeViT / EfficientFormer w/ Swin and other ideas. BUT, they all need fairly extensive refactoring w/ checkpoint mapping to fit timm, get feat extraction working, etc so not currently something I have bandwidth for.

If anyone wants to tackle this or the others, criteria for accepting:

• refactored to stages, downsample always at beginning of stage! See this relevant examples
  • https://github.com/huggingface/pytorch-image-models/blob/main/timm/models/levit.py (vs original https://github.com/facebookresearch/LeViT)
  • https://github.com/huggingface/pytorch-image-models/blob/main/timm/models/efficientformer_v2.py (vs original https://github.com/snap-research/EfficientFormer)
  • https://github.com/huggingface/pytorch-image-models/blob/main/timm/models/efficientformer.py
• Residuals inline with block code
• Add feature_info and feature extraction support if model has NCHW or NHWC layout at end of stages
  • see https://github.com/huggingface/pytorch-image-models/blob/main/timm/models/efficientformer_v2.py
• Rename some of the modules like Conv_BN to match above examples, remove _
• Extract stem as module as per levit/efficientformer

[seefun]

I tried to implement EfficientViT (MSRA) in here: #1894

There is another job with the same name before this EfficientViT, which looks good and has also been added.

[youssefadr]

Hello @rwightman, is there a model I can work on? I would be very happy to contribute!

[rwightman]

@youssefadr this one is done, I currently have MobileOne, FastViT and Inception_neXt (and a prototype underway... but #1842 (FasterViT) hasn't been tackled, although it looks like it's an easy adaptation, I'd want the downsamples moved, and some other aspects cleaned up, so that can be a bit of fun...

[youssefadr]

Thank you for the answer, I will take a look at FasterViT soon 👍

[Kaschi14]

In the EfficientVit https://github.com/mit-han-lab/efficientvit there are weights for higher res models which are currently not supported in timm. Is it planned to include the higher res (1024x2048) Models in Timm as well? :)

[seefun]

@Kaschi14 I noticed the origin repo added the L-series efficientvit weights and the SAM distillation weights, but the higher res (1024x2048) models is from cityscape segmentation models.