Learnable group tokens fine-tuning on out-of-domain datasets

[AhmedBourouis]

Hi! Thank you for the great work and neat implementation.

In section C.4 of the paper you reported results from testing GroupViT pretrained model on COCO dataset, which were quit impressive but not as good as the ones on PASCAL VOC. This is probably due to the domain shift between PASCAL VOC and COCO images, classes/text descriptions.

I was wondering if it's possible to fine-tune GroupViT on COCO dataset (and out-of-domain datasets in general) by freezing the model's weights and training only the learnable group tokens on the new datasets in a few-shot manner.

If it's the case, I'm ready to implement this with you're guidance.

[xvjiarui]

Hi @AhmedBourouis

Thanks for your interest in our work.

Regarding domain shift, the GroupViT is trained on neither PASCAL VOC nor COCO datasets. So the domain shift to Pascal and COCO should be roughly the same.

Nevertheless, it is totally possible to fine-tuning the group token only. You may need to freeze the text encoder as well.