Multimodal example

[MehreenMehreen]

Hello,

Thanks for your great work. I am trying out the multimodal example for qwen2.5vl. Would the code work? Dense.py uses Qwen2_5OmniThinkerForConditionalGeneration but for qwen2.5vl model it should be Qwen2_5VLForConditionalGeneration. Would the dataset.py files also need changing? It is using utils from omni package not the vl package.

Also, in dataset.py, the line: image = document_info.get('image', None). Should this be replaced by: image = document_info.get('document_image', None)?

Thanks in advance for your help,
Mehreen

[ArvinZhuang]

Hi @MehreenMehreen , if you want to use qwen2.5vl as the backbone, change Qwen2_5OmniThinkerForConditionalGeneration to Qwen2_5VLForConditionalGeneration, qwen_omni_utils to qwen_vl_utils, and process_mm_info to process_vision_info in collator.py.

We only use image as the name of the feature for query and docs.

[MehreenMehreen]

Thanks so much for the detailed answer. Got it to work with your suggested changes and some minor ones in dataset.py for my particular installation.

[MehreenMehreen]

Do you have any recommendations for mining hard negatives for my particular dataset? I have Arabic newspapers but the queries and answers are all in English and so none of your models is trained on this and hence, I do not have a 'results' file.