Fine-grained Adaptive Visual Prompt for Generative Medical Visual Question Answering

Setup & Data Preparation

Prepare the environment

git clone https://github.com/OpenMICG/FAVP.git
cd FAVP
conda env create -f environment.yml
conda activate favp

Data Preparation

ROCO-Dataset: Download from here
PMC-VQA: Download from here
SLAKE: Download from here
VQA-RAD: Download from here
DME：Download from here

Model Preparation

Vicuna V0 7B: Download from here. Then, set the variable llama_model in the model config file to the LLM path here

Pretraining

Stage1

The weights of the first stage are saved in output_dir of train_configs/stage1_pretrain.yaml, and you can change it to your own directory

cd run_scripts
sh stage1_pretrain.yaml

Stage2

The weights of the second stage are saved in output_dir of train_configs/stage2_pretrain.yaml, and you can change it to your own directory

cd run_scripts
sh stage2_pretrain.yaml

Finetuning

cd run_scripts
# SLAKE
sh train_slake.yaml
# VQA-RAD
sh train_rad.yaml

If you don't want to go through the above training process, you can download VQA-RAD checkpoint from huggingface

Test

cd run_scripts
# VQA-RAD
sh test_rad.yaml

Acknowledgement

The implementation of FAVP relies on Minigpt-V and SAM-Med2D. We thank the original authors for their work and open source code.

Citation

If you find FAVP useful for your research and applications, please cite using this BibTeX:

@inproceedings{yu2025fine,
  title={Fine-grained Adaptive Visual Prompt for Generative Medical Visual Question Answering},
  author={Yu, Ting and Tong, Zixuan and Yu, Jun and Zhang, Ke},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={39},
  number={9},
  pages={9662--9670},
  year={2025}
}