[Usage] Missing File in Release LLaVA-v1.6 and RuntimeError in model_vqa_qbench.py

[X1AOX1A]

Describe the issue

Issue:

The file llava/eval/model_vqa_qbench.py is missing in the Release LLaVA-v1.6. Could you provide some insight into why this file was removed?

Additionally, I encountered a bug while using the file llava/eval/model_vqa_qbench.py from commit ac89962.

Command:

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/qbench.sh dev

Log:

Traceback (most recent call last):
  File "/root/anaconda3/envs/feat_align/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/anaconda3/envs/feat_align/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/data/root/Documents/CODES/FeatAlign/llava/eval/model_vqa_qbench.py", line 122, in <module>
    eval_model(args)
  File "/data/root/Documents/CODES/FeatAlign/llava/eval/model_vqa_qbench.py", line 99, in eval_model
    n_diff_input_output = (input_ids != output_ids[:, :input_token_len]).sum().item()
RuntimeError: The size of tensor a (77) must match the size of tensor b (5) at non-singleton dimension 1

Related issue: Q-Bench Evaluation Scripts for LLaVA-v1.5. #581

Thank you!

[TGLTommy]

Describe the issue

Issue:

The file llava/eval/model_vqa_qbench.py is missing in the Release LLaVA-v1.6. Could you provide some insight into why this file was removed?

Additionally, I encountered a bug while using the file llava/eval/model_vqa_qbench.py from commit ac89962.

Command:

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/qbench.sh dev

Log:

Traceback (most recent call last):
  File "/root/anaconda3/envs/feat_align/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/anaconda3/envs/feat_align/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/data/root/Documents/CODES/FeatAlign/llava/eval/model_vqa_qbench.py", line 122, in <module>
    eval_model(args)
  File "/data/root/Documents/CODES/FeatAlign/llava/eval/model_vqa_qbench.py", line 99, in eval_model
    n_diff_input_output = (input_ids != output_ids[:, :input_token_len]).sum().item()
RuntimeError: The size of tensor a (77) must match the size of tensor b (5) at non-singleton dimension 1

Related issue: Q-Bench Evaluation Scripts for LLaVA-v1.5. #581

Thank you!

you can check it here : https://github.com/haotian-liu/LLaVA/pull/581/files#diff-24aa88b077252259843630f0a32315e0c252f04dfd7ede7bcee31e1ffe4c883b

[X1AOX1A]

Thanks for your reply. Is it compatible with LLaVA with commit c121f04? I copied the file you mentioned, but I still encounter the same problem during evaluation:

Command:

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/qbench.sh dev

Log:

Traceback (most recent call last):
  File "/root/anaconda3/envs/feat_align/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/anaconda3/envs/feat_align/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/data/root/Documents/CODES/FeatAlign/llava/eval/model_vqa_qbench.py", line 122, in <module>
    eval_model(args)
  File "/data/root/Documents/CODES/FeatAlign/llava/eval/model_vqa_qbench.py", line 99, in eval_model
    n_diff_input_output = (input_ids != output_ids[:, :input_token_len]).sum().item()
RuntimeError: The size of tensor a (77) must match the size of tensor b (5) at non-singleton dimension 1