How to evaluate

[xuyeliu]

how to evaluate ASVR? do you have any evaluation instruction to use cambrian to evaluate asvr? Thanks

[AlenjandroWang]

Thanks for issue. You can directly use the code from LLaVA or Cambrian to evaluate the ASVR-trained model checkpoints, since the evaluation process for ASVR does not require the vision head. Therefore, the model architecture is consistent with LLaVALlamaForCausalLM, and the parameters can be loaded without issue.

[xuyeliu]

Thanks for your reply. Could you please provide the steps to evaluate it? I notice the Camrian may need the model bin file to evaluate. Currently, based on the readme, we only have config.json and checkpoint which is not whole model file. So I can only pretrain and fine-tune again.

I will appreciate If you can provide a specific evaluation process.
Thanks

[xuyeliu]

I tried to follow cambrian evaluation step to evaluate the fine-tuned asvr model(use finetune.sh to generate). But it seems cambrian cannot directly load asvr model. Could you please also include the evaluation scripts to help us evaluate it?

Thanks