Whether the visual encoder participates in training

[LoverLost]

In the 3rd stage, following your paper 'Finally, we further perform instruction tuning of the pre-trained model on visual language instruction datasets,' I'm wondering if siglip is also unfrozen and involved in the training during this stage. Or is it only the projector and LLM?

[LoverLost]

If possible, I would also like to know which version of llama is used for the LLM of the 3b model?

[Lyken17]

ViT was during vila training.

For 3b, we used shearedllama from princeton.

[Lyken17]

https://huggingface.co/princeton-nlp/Sheared-LLaMA-2.7B