How could I train ImageBind completely from scratch?

[ChloeL19]

Thank you for the very cool work!
I'm having trouble finding your implementation of NCE loss, however. I know @fabawi has implemented a version of this for his LoRA fine-tuning version (kudos). However, if I wanted to train the original ImageBind model completely from scratch how would I do this?

[fabawi]

you can train the model without LoRA using ImageBind-LoRA. Simply remove the --lora argument when calling train.py and set --full_model_checkpointing. I don't have the resources to fine-tune it but it should work in theory. Try it out on the toy dataset provided in the repo, and later you can try to implement dataloaders for the original datasets.

On a side note, LoRA adaptation was shown to outperform full model fine-tuning in many instances and for different applications --- this is not specific to ImageBind-LoRA but a general statement about low-rank adaptation. If you would like to see the outcomes of fine-tuning on LoRA, check out the example. The weights are in the repo itself. Just install the requirements and follow the instructions in the README.

[Wolfwjs]

Hi @ChloeL19 ， May I ask if you have been able to train the model from scratch？

[wendellgithub0206]

I want to know how to train and how to apply the model after training is complete?
How to present the results?

[K-M-Ibrahim-Khalilullah]

you can train the model without LoRA using ImageBind-LoRA. Simply remove the --lora argument when calling train.py and set --full_model_checkpointing. I don't have the resources to fine-tune it but it should work in theory. Try it out on the toy dataset provided in the repo, and later you can try to implement dataloaders for the original datasets.

On a side note, LoRA adaptation was shown to outperform full model fine-tuning in many instances and for different applications --- this is not specific to ImageBind-LoRA but a general statement about low-rank adaptation. If you would like to see the outcomes of fine-tuning on LoRA, check out the example. The weights are in the repo itself. Just install the requirements and follow the instructions in the README.

Why use LoRA? Did ImageBind paper use it?