You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, I'm training using the example script on the cats and dogs dataset, and noticed the validation accuracy is consistently higher than training accuracy throughout training. This occurs both with and without dropout and dropout_emb. Is this typical for a ViT? Below is a plot of training and validation accuracy for SmallDatasetVit compared to a CNN for reference.
The model and parameters used are as follows:
from vit_pytorch.vit_for_small_dataset import ViT
batch_size = 64
model = ViT(
image_size = 224,
patch_size = 16,
num_classes = 2,
dim = 1024,
depth = 6,
heads = 16,
mlp_dim = 2048,
dropout = 0.1,
emb_dropout = 0.1)
The text was updated successfully, but these errors were encountered:
Update: if this is occurring for anyone else, it's to do with torch's random image augmentation on the train data. I got these results using RandomResizedCrop and RandomHorizontalFlip like in the example code. This also occurs when using the same transforms on a torch CNN (the example above in red was keras). Without image augmentation, the training accuracy far exceeds validation accuracy, but performance is diminished. It's nothing VIT-specific, just torch.
Hi, I'm training using the example script on the cats and dogs dataset, and noticed the validation accuracy is consistently higher than training accuracy throughout training. This occurs both with and without dropout and dropout_emb. Is this typical for a ViT? Below is a plot of training and validation accuracy for SmallDatasetVit compared to a CNN for reference.
The model and parameters used are as follows:
from vit_pytorch.vit_for_small_dataset import ViT
batch_size = 64
model = ViT(
image_size = 224,
patch_size = 16,
num_classes = 2,
dim = 1024,
depth = 6,
heads = 16,
mlp_dim = 2048,
dropout = 0.1,
emb_dropout = 0.1)
The text was updated successfully, but these errors were encountered: