Add PoolFormer Model

[tanaymeh]

🌟 New model addition

I would like to add the recently announced PoolFormer model to the Transformers library.

Model description

PoolFormer model was proposed in the paper, “MetaFormer is Actually What You Need for Vision” by Sea AI Lab and the main argument behind this is that performance of transformer/MLP-like models primarily comes from the general architecture instead of the specific token mixers they are using (like Attention for example).

To show this, they have added a basic non-parametric Pooling operator which does basic token mixing. This suggested model outperforms DeiT and ResMLP.

Link to the Model Repo: PoolFormer
Link to the Paper: MetaFormer is Actually What You Need for Vision

Open-source status

• the model implementation is available: In PyTorch on PoolFormer repository
• the model weights are available: In the same repository, link
• who are the authors: Sea AI Lab