About evaluation result

[zhangliang-04]

Hi @mmaaz60,
Thanks for your great work and open sourcing!
I am trying to evaluate PALO-7B (loaded from transformers) on the multilingual-llava-in-the-wild, but I find the performance is much lower than the reported numbers. Here are the results I got:

Model	English	Chinese
PALO-7B (paper)	64.2	55.7
PALO-7B (my results)	54.0	43.0

Here are the generated content files:
PALO-7B_English_content.json
PALO-7B_Chinese_content.json

Here are the evaluation files with scores:
PALO-7B_English.json
PALO-7B_Chinese.json

Summaries produced by palo/eval/summarize_gpt_review.py

PALO-7B_English                                                                                                                              
all 54.0 85.2 46.0                                                                                                                           
llava_bench_complex 62.8 82.5 51.8                                                                                                           
llava_bench_conv 52.4 86.5 45.3                                                                                                              
llava_bench_detail 40.6 88.7 36.0

PALO-7B_Chinese                                                                                                                    
all 43.0 86.0 37.0
llava_bench_complex 55.6 82.9 46.1
llava_bench_conv 27.2 88.8 24.1
llava_bench_detail 39.1 88.7 34.7

Is there a significant discrepancy between the content I generated and yours, or there are issues in evaluation? Do you have any idea about this, or share the generated result files with me?

[zhangliang-04]

By the way, I find the path of some language in the evaluation scripts contains corrected, is there an updated version of the multilingual-llava-in-the-wild benchmark?