On the evaluation under zero/few-shot setting with ITM+MLM

[matthewdm0816]

Hi,
Thanks for the open-source code provided in this repository for the paper, and I'm very interested in the zero/few-shot ability as shown in your great work, but as I investigate to evaluate the model under zero-shot MLM+ITM setting in run_gqa_prompt_zero_few.py, I found that the ITM head (i.e. model.cls_ans.seq_relationship) is newly initialized (and of different shape from original pretrained ITM head) and actually not used in the evaluation. Am I wrong on the usage of this code or is ITM head not participating in the evaluation?
Thanks a lot for your time and attention.

[LYuhang]

@matthewdm0816 Thanks for your attention. Actually, we have initialized the cls_ans.seq_relationship in the code (refer to

DPT/VinVL/Oscar/oscar/run_gqa_prompt_zero_few.py

Line 50 in b6b4835

def init_weights_with_pretrained_parameters(self, config):

)