FinRAGBench-V: A Benchmark for Visual RAG in the Financial Domain

FinRAGBench-V is a comprehensive benchmark for visual retrieval-augmented generation (RAG) in finance, addressing the challenge that most existing financial RAG research focuses predominantly on text while overlooking rich visual content in financial documents. By integrating multimodal data and providing visual citation, FinRAGBench-V ensures traceability and supports robust evaluation of Multimodal Large Language Models (MLLMs).

🤗 Dataset 📄 Paper

Benchmark: FinRAGBench-V

• 📊 Multimodal Retrieval Corpus: We construct a multimodal financial corpus by collecting documents from various real-world financial sources, including research reports, financial statements, prospectuses, academic papers, financial magazines, and financial news. The corpus contains 60,780 Chinese pages and 51,219 English pages from 1,104 Chinese and 1,105 English documents.

• 📝 High-Quality QA Dataset: We construct a high-quality, human-annotated question-answering (QA) dataset spanning heterogeneous data types (charts, tables, and texts) and seven question categories, including time-sensitive, numerical calculations, comparison and sorting, and multi-page queries.

Baseline: RGenCite

We provide RGenCite, a multimodal RAG baseline that seamlessly integrates:

• 🔍 Retrieval: Efficiently retrieves relevant textual and visual information from the dataset.
• ✍️ Generation: Produces high-quality answers grounded in retrieved content.
• 📌 Fine-Grained Visual Citation: Provides precise visual evidence (page- and block-level) to support answers.

Automatic Citation Evaluation

We propose an automatic evaluation method for visual citation, which:

• Measures precision and recall at multiple citation levels.
• Uses box-bounding and image-cropping techniques to assess the alignment of cited visual evidence.

Installation

git clone https://github.com/zhaosuifeng/FinRAGBench-V.git
conda create --name finragbench python=3.11 -y
conda activate finragbench
pip install -r requirements.txt
pip install -e .

Dataset Download and Preprocessing

First, download all dataset files (including corpus, queries, and qrels) from zhaosuifeng/FinRAGBench-V and place them into the corresponding folders under ./data/.

The corpus is provided in multiple split parts (part_*.tar.gz). Please merge them before extraction.

Example (English corpus)

# Create folder
mkdir -p ./data/corpus/en
cd ./data/corpus/en

# Merge all parts into a single tar.gz
cat part_*.tar.gz > corpus_en.tar.gz

# Extract
tar -xzvf corpus_en.tar.gz

# Back to prepare_data
cd ../../../prepare_data

# Convert corpus and queries into parquet format
python generate_parquet.py

Retrieval:

cd ./FinRAGBench-V/retrieval
python eval_mm_retriever.py

use encode_config.json for encoding and retrieve_config.json for retrieval.

Generation:

cd ./FinRAGBench-V/generation
python generate.py

Evaluation:

cd ./FinRAGBench-V/eval
python eval_generation.py
python eval_citation.py

Other Related Projects

• OpenMatch

Citation

If you find this work useful, please cite:

@article{zhao2025finragbench,
  title={FinRAGBench-V: A Benchmark for Multimodal RAG with Visual Citation in the Financial Domain},
  author={Zhao, Suifeng and Jin, Zhuoran and Li, Sujian and Gao, Jun},
  journal={arXiv preprint arXiv:2505.17471},
  year={2025}
}