CoQAH

This repository is the implementation of Generalizing Visual Question Answering from Synthetic to Human-Written Questions via a Chain of QA with a Large Language Model

Implementation for CLEVR

1. Clone MDETR and follow instructions to setup environment.
2. Copy ipynb files in the mdetr folder of this repo to the cloned mdetr folder.
3. Download mdetr checkpoint trained on CLEVR from here clevr_checkpoint.pth and place it into the mdetr folder. Information about the checkpoint is available here
4. Download CLEVR-Human dataset (json files) and correspoing images. Note that you need to configure the path to the dataset in the code to match your setup.

Implementation for Chest X-rays

1. Clone OFA and follow instructions to setup environment.
2. Train finetune OFA model with MIMIC-DIFF-VQA dataset
3. Copy ipynb files in the ofa folder of this repo to the cloned OFA folder.
4. Download PLEURAL checkpoint trained on MIMIC-DIFF-VQA from here and place it into the mdetr folder.
5. Download VQA-RAD and SLAKE. Note that you need to configure the path to the dataset in the code to match your setup.