smrl_cfvqa_rubi is TOO slow to train

[Mike4Ellis]

I can train all the other versions except smrl_cfvqa_rubi with batch_size = 256, I have to change it to 64 preventing CUDA out of memory. But the training is too slow, which takes a day to train a epoch with three 3090.

I wonder what is the difference between smrl_cfvqa_rubi and other versions making it so speical, and is it normal to train so slowly? or is it because I did something wrong?

[yuleiniu]

It is wired. Could you also try smrl_cfvqasimple_rubi as it has the same function as smrl_cfvqa_rubi?

BTW, I will delete smrl_cfvqa_rubi to avoid confusion as it is redundant.

[Mike4Ellis]

It is wired. Could you also try smrl_cfvqasimple_rubi as it has the same function as smrl_cfvqa_rubi?

BTW, I will delete smrl_cfvqa_rubi to avoid confusion as it is redundant.

I have trained smrl_cfvqasimple_rubi, and it works well.

I realized the only difference between smrl_cfvqasimple_rubi and smrl_cfvqa_rubi is q_single = True in smrl_cfvqa_rubi while False in smrl_cfvqasimple_rubi.
Also, smrl_cfvqa_rubi has mlp_v, but may not work because fusion_mode is rubi.

Is it because q_single? What dose this parameter really mean? And why do we need a single text encoder?

[yuleiniu]

The q_single option uses a separate text encoder for the QA branch. In our implementation, the main branch and QA branch share the same text encoder, following RUBi, \ie, q_single=False. For mlp_v, you are right, this option does not work when we implement the 'cfvqasimple' option. You can use smrl_cfvqasimple_rubi for applying RUBi+CF-VQA.

[Mike4Ellis]

Okay! I got it.

Thank you very much for your patient reply!