Unable to reproduce SQA results for llava-1.5

[clairez-cerebras]

I was attempting to reproduce llava-1.5's results in ScienceQA but was not able to reproduce the results reported.
Command:

python -m accelerate.commands.launch --num_processes=1 -m lmms_eval --config ./configs/eval_scienceqa_llava1.5.yaml

Config:

- model: llava
  model_args: pretrained=liuhaotian/llava-v1.5-7b,use_flash_attention_2=False,model_name=llava
  tasks: scienceqa_full
  batch_size: 1
  log_samples: true
  log_samples_suffix: llava1.5_sqa
  output_path: "./logs/"

The results I got:

|     Tasks      |Version|Filter|n-shot|  Metric   |Value |   |Stderr|
|----------------|-------|------|-----:|-----------|-----:|---|-----:|
|scienceqa_full  |N/A    |none  |     0|exact_match|0.3699|±  |0.0097|
| - scienceqa    |Yaml   |none  |     0|exact_match|0.3744|±  |0.0074|
| - scienceqa_img|Yaml   |none  |     0|exact_match|0.3604|±  |0.0107|

|    Groups    |Version|Filter|n-shot|  Metric   |Value |   |Stderr|
|--------------|-------|------|-----:|-----------|-----:|---|-----:|
|scienceqa_full|N/A    |none  |     0|exact_match|0.3699|±  |0.0097|

which is far from what's reported in the paper, for example, SQA-IMG is reported to have 71.6 in the llava-1.5 paper and SQA in general is reported to be around 70.4 in the excel sheet
What could be wrong?

[kcz358]

Thank you for reporting the issue. I will try to look into this error later.

[GoGoJoestar]

I I encountered the same problem when reproduce llava-1.6-mistral-7b results in ScienceQA. I found the reason maybe the following lines in models/llava.py.

lmms-eval/lmms_eval/models/llava.py

Lines 361 to 371 in efb5295

	# The above for loop has bugs. When there is no visuals, e.g. pure text,
	# there will be no for loop execute resulting in an empty question_input (because no visuals)
	# Scenario 1 won't even be execute
	if len(flattened_visuals) == 0:
	for context in contexts:
	question = context
	conv = conv_templates[self.conv_template].copy()
	conv.append_message(conv.roles[0], question)
	conv.append_message(conv.roles[1], None)
	prompt_question = conv.get_prompt()
	question_input.append(prompt_question)

Although the annotation says “The above for loop has bug” when input has no visuals, but actualy, the above loop run normally and add a prompt_question to the question_input list, and then these line add a prompt_question again. As the result, these no visual inputs generate 2 answers, leading to order mismatch of questions and answers.

After remove these line code, the scienceqa-full result changes from 36.3 to 76.8.

[kcz358]

Hi @GoGoJoestar , I think your fix is correct. We previously use flattened visuals instead of batched visuals in the previous loop, resulting error when handling none visuals. I will remove these lines