[CVPR 2026] EvoGraph-R1: Self-Evolving Multimodal Knowledge Hypergraphs for Agentic Retrieval

Official implementation of EvoGraph-R1: Self-Evolving Multimodal Knowledge Hypergraphs for Agentic Retrieval. A GraphRAG framework that models retrieval as an MDP over a dynamic hypergraph, enabling an agent to query, expand, edit, and answer through closed-loop graph evolution.

Get Started

Create the environment:

conda create -n evograph-r1 python=3.11
conda activate evograph-r1

pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt
pip install flash-attn==2.6.3 --no-build-isolation
pip install -e . --no-deps

Set API keys:

cp .env.example .env

export OPENAI_API_KEY=...
export OPENAI_BASE_URL=...
export OPENAI_MODEL=gpt-4o-mini
export JINA_API_KEY=...

Data

Raw datasets and generated graph artifacts are not included. Keep local files under ignored directories:

datasets/      Text QA data
datasets_mm/   Multimodal VQA, KB, and image data
expr/          Text hypergraphs and retrieval indexes
expr_mm/       Multimodal graph and index artifacts

Text datasets used in the paper:

Dataset	Source
2WikiMultiHopQA	Official repository
HotpotQA	Official website
Natural Questions	Official website

Multimodal datasets and assets:

Resource	Source
EchoSight assets	EchoSight repository
Encyclopedic VQA / E-VQA	Google Research release
InfoSeek	Project page
OK-VQA	Dataset page
OVEN images	Hugging Face
Google Landmarks v2	Dataset page
iNaturalist 2021	Dataset page

Text EvoGraph-R1 Construction

Prepare data in the text EvoGraph-R1 layout:

datasets/<DATA_SOURCE>/raw/qa_train.json
datasets/<DATA_SOURCE>/raw/qa_dev.json
datasets/<DATA_SOURCE>/raw/qa_test.json

Preprocess and build the knowledge hypergraph:

export DATA_SOURCE=<text-dataset-name>

python script_process.py --data_source ${DATA_SOURCE}
python script_build.py --data_source ${DATA_SOURCE}

Multimodal Graph Construction

For EchoSight-style E-VQA and InfoSeek assets, generate a local placement plan:

export MM_DATA_ROOT=/path/to/evograph-mm-data
export MM_DATASET=<E-VQA-or-InfoSeek>
export MM_SUBSET=<subset-name>

python script_prepare_echosight_mm.py --root ${MM_DATA_ROOT} --dataset ${MM_DATASET} --write-plans --required-only

After placing assets, validate, preprocess, and build the multimodal graph:

python script_validate_echosight_mm.py --root ${MM_DATA_ROOT} --dataset ${MM_DATASET}
python script_process_mm.py --root ${MM_DATA_ROOT} --dataset ${MM_DATASET} --subset ${MM_SUBSET} --metadata-only --format parquet
python script_build_mm.py --root ${MM_DATA_ROOT} --dataset ${MM_DATASET} --subset ${MM_SUBSET} --output-root ${MM_DATA_ROOT}/expr_mm --embedding-backend ${MM_EMBEDDING_BACKEND} --model ${MM_EMBED_MODEL_PATH}

For OK-VQA or custom multimodal datasets, prepare the same processed parquet and graph/index layout under ${MM_DATA_ROOT}/datasets_mm/${MM_DATASET}/ and ${MM_DATA_ROOT}/expr_mm/${MM_DATASET}/.

Training

Text EvoGraph-R1

Start the retrieval API and run GRPO:

export DATA_SOURCE=<text-dataset-name>

python script_api.py --data_source ${DATA_SOURCE} --port 8001
bash run_grpo.sh -p Qwen/Qwen2.5-7B-Instruct -m Qwen2.5-7B-Instruct -d ${DATA_SOURCE}

Other text RL launchers:

bash run_rpp.sh -p Qwen/Qwen2.5-7B-Instruct -m Qwen2.5-7B-Instruct -d ${DATA_SOURCE}
bash run_ppo.sh -p Qwen/Qwen2.5-7B-Instruct -m Qwen2.5-7B-Instruct -d ${DATA_SOURCE}

Multimodal EvoGraph-R1

Start the text and multimodal APIs:

export TEXT_DATA_SOURCE=<text-dataset-name>
export MM_DATASET=<multimodal-dataset-name>
export MM_SUBSET=<subset-name>
export MM_KB_DIR=${MM_DATA_ROOT}/expr_mm/${MM_DATASET}
export MM_EMBED_DEVICE=cpu

python script_api.py --data_source ${TEXT_DATA_SOURCE} --working_dir expr/${TEXT_DATA_SOURCE} --port 8001
python script_api_mm.py --dataset ${MM_DATASET} --subset ${MM_SUBSET} --working_dir ${MM_KB_DIR} --embedding-backend ${MM_EMBEDDING_BACKEND} --model ${MM_EMBED_MODEL_PATH} --port 8003

Run multimodal GRPO:

export ACTOR_LR=5e-7

bash run_mm_grpo.sh \
  -p Qwen/Qwen2.5-VL-7B-Instruct \
  -m Qwen2.5-VL-7B-Instruct \
  -d ${MM_DATASET} \
  -s ${MM_SUBSET}

The 3B variants can be used as lower-resource alternatives by replacing the model path and name.

Use environment variables such as ACTOR_LR, N_GPUS, TRAIN_BATCH_SIZE, VAL_BATCH_SIZE, and TOTAL_EPOCHS to tune and scale training.

Citation

@inproceedings{lin2026evograph,
  title={EvoGraph-R1: Self-Evolving Multimodal Knowledge Hypergraphs for Agentic Retrieval},
  author={Lin, Jiashi and Jiang, Changhong and Lin, Xiangru and Zhang, Ruifei and Zhu, Xinyi and Liu, Jiyao and Tang, Cheng and Du, Ye and Gao, Shujian and Ning, Junzhi and others},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={756--765},
  year={2026}
}

Acknowledgements

This repository builds on Graph-R1, EchoSight, Search-R1, and verl. We thank the authors for open-sourcing their code and resources.