handle non square patch sizes

[FilipAndersson245]

Hello, would it be difficult to allow for nonsquare patches to be used, I'm examining the use of Vit on some non-image data where it cannot be represented in a square format.
Preferably by having a tuple as an input to select width and height. Is there anything currently that would block this from being done?

[FilipAndersson245]

I added initial functionality for this inside the vanilla Vit class,
See #105 (comment)

[lucidrains]

@FilipAndersson245 hey! it actually already works with non-square patches, as long as you set the image size to be that of max(height, width), and your height and width is divisible by the patch size

[FilipAndersson245]

@lucidrains Hey! do you have an example of how you do that?
This is the code I currently use with my pull request,

from vit_pytorch import vit

v = vit.ViT(
    image_size = (25,60),
    patch_size = (25,1),
    num_classes = 30,
    dim = 1024,
    depth = 6,
    heads = 16,
    mlp_dim = 2048,
    dropout = 0.1,
    emb_dropout = 0.1
)

img = torch.randn(1, 3, 25, 60)

preds = v(img)

[lucidrains]

@FilipAndersson245 yup, just try

import torch
from vit_pytorch import ViT

v = ViT(
    image_size = 256,
    patch_size = 32,
    num_classes = 1000,
    dim = 1024,
    depth = 6,
    heads = 16,
    mlp_dim = 2048,
    dropout = 0.1,
    emb_dropout = 0.1
)

img = torch.randn(1, 3, 256, 128) # not a square

preds = v(img) # (1, 1000)

[lucidrains]

@FilipAndersson245 updated the readme too :) this is a frequently asked question

[FilipAndersson245]

@lucidrains Okay, but in your example, the patch size is still square 32x32 correct?
Thanks for the quick response!

[lucidrains]

Ohh gotcha, you want nonsquare patches! 🤦‍♂️ Ok I'll get this built :)

[FilipAndersson245]

Did a pull request implementing it if you want to give it a look :)

[FilipAndersson245]

my pull request was only for VIT but it would probably be good to do the same to all variants.

[lucidrains]

@FilipAndersson245 6549522

[Fodark]

Are there plans to include this modification to the other variants of ViT? e.g. the efficient one?