training V-L and L-P projection layer

[remvanthull]

Hi!

I have a question about training the projection layers; is this allowed by the code by default, or do you need to specify model arguments/make changes in the code to do this? I was getting quite confused when trying to figure this out.

Thanks so much in advance!
Rachel

[remvanthull]

@hanoonaR I noticed in train_ft.py that the mm_projector weights were set to requires_grad = False, should this be set to True to finetune the V-L projection layer?
https://github.com/mbzuai-oryx/groundingLMM/blob/0c8b252efc5f7b8310bf2f83c3441d78862eb750/train_ft.py#L214C1-L215C32

Or should you set --tune_mm_mlp_adapter to True?

Thanks,
Rachel