DeepGrounder: Generalized Reasoning for MLLM-based Remote Sensing Visual Grounding

🛠️ Setup

conda create -n deepgrounder python=3.10
conda activate deepgrounder
bash setup.sh

📊 Inference

cd ./src/eval

python infer_allbox.py

💪🏻 Training

📚 SFT

We use LLaMA-Factory to train the SFT model.

1. Clone the LLaMA-Factory repository and install the dependencies.

git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics]"

2. Use the mllm_rec_json.json that we provided, nodify the dataset_info.json and qwen2_5_vl_full_sft.yaml. Put the json files in the LLaMA-Factory/data directory and the yaml file in the LLaMA-Factory/examples/train_full directory.
3. Run the following command to train the SFT model.

llamafactory-cli train examples/train_full/qwen2_5_vl_full_sft.yaml

📚 GRPO

1. Download the COCO Train2014 image and unzip it, and we refer to the image dir as <your_image_root>.
2. Download the RefCOCO/+/g and LISA-Grounding Annotation files and unzip it (LISA-Grounding is used for out-of-domain evaluation).
3. Change the data_paths and image_folders in the run_scripts/run_grpo_rec.sh file.

# These jsonl files are included in the annotation files at step 2.
# Note: please use jsonl files instead of json files.
data_paths="path/to/refcoco_train.jsonl:path/to/refcocop_train.jsonl:path/to/refcocog_train.jsonl"
image_folders="path/to/coco:path/to/coco:path/to/coco"

4. bash src/open-r1-multimodal/run_scripts/run_grpo_rec.sh

🤝 Acknowledgements

We would like to express our sincere gratitude to DeepSeek, Open-R1, QwenVL, Open-R1-Multimodal, VLM-R1, LLaMA-Factoryfor providing open-source resources that contributed to the development of this project.