ChartBench: A Benchmark for Complex Visual Reasoning in Charts

Introduction

We propose the challenging ChartBench to evaluate the chart recognition of MLLMs.

We improve the Acc+ metric to avoid the randomly guessing situations.

We collect a larger set of unlabeled charts to emphasize the MLLM's ability to interpret visual information without the aid of annotated data points.

Todo

• Open source all data of ChartBench.
• Open source the evaluate scripts.
• Open source the inference scripts.
• Open source the demo data (10%).

Setup

Please follow the official repository instructions below to set up the local environment.

• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 

Inference

1. Complete the basic environment setup.
2. Set task_name in ./Repos/myprompt.py, such as test or BLIP2_Style.
3. Select or set the desired system prompt in ./Repos/myprompt.py.
4. Modify the default path of CKPT_PATH in ./Repos/{MODEL_NAME}/run.py.
5. Run run.py following the command format in ./Scripts/inference.sh.
6. The results are saved by default in ./Eval/{task_name}/{MODEL_NAME}.
7. Set the parameters in ./Scripts/stat_acc_plus.py and the statistical results are saved in ./Eval/{task_name}/Eval_Result.

Ranking

Citation

@article{ChartBench,
    title={ChartBench: A Benchmark for Complex Visual Reasoning in Charts},
    author={Zhengzhuo Xu and Sinan Du and Yiyan Qi and Chengjin Xu and Chun Yuan and Jian Guo},
    journal={ArXiv},
    year={2023},
    volume={abs/2312.15915},
    url={https://api.semanticscholar.org/CorpusID:266550948}
}