Swin V2

[ChristophReich1996]

Hi, first of all, thanks for all the amazing work. I just reimplemented a version of the Swin Transformer V2 (no official implementation currently available) and I would like to push it to timm, making it easier accessible for others. What should I be aware of before opening a pull request (would be my first timm pull request)? My code is available here. Thanks in advance!

[rwightman]

@ChristophReich1996 I looked at your impl, it's nice and clean. You added the defmorable variant which is nice. I'm open to a PR with a few notes

• I'm sure the official models will be released at some point, I want to reserve the swin_...._v2 names for that as they will come with a full set of pretrained weights. Perhaps swin_large..._v2_cr or cr_swin_large...v2 (your initials, to differentiate)
• To include a full architecture here I want to have at least one good set of in1k pretrained weights, I gather you don't have any?
• Some mentions re other guidlines for models is there some template for creating my own model ? #1075

So, next steps would be:

• You create a PR, try to match timm style and registration, create model helpers, interface
• I'll help clean up
• I can try to train one of the small-medium sized variants, if it's good we merge, otherwise figure out what's going on
• Announce the inclusion of weights / model impl, hopefully before official v2 is released

Re style, yours is clean as mentioned, the most significant thing I'd end up changing is indent alignment, I generally don't like to align multiple lines of args, etc from opening brackets, but align next line on standard indent as I find it much easier to maintain over long term (with diffs, etc).

[ChristophReich1996]

@rwightman thanks for your detailed description! Yes, unfortunately, I don't have any ImageNet1k weights (due to limited computing resources), however, I did some smaller tests on the CIFAR10 and Places365 (256x256 variant) dataset which went quite well, so I'm optimistic about getting some decent results from the model. I think I should be able to get the code ready in one or two days and will open a pull request :)