VLM-R1-Seg

Update 2025.04.25

Training Data and Checkpints

• Qwen-VL2.5-7B@refCOCOg_9k_840_mask
• /training/zilun/Seg-Zero_best_ckpt/rewardmodelsam_segformat_thinkingformat_bboxiou_masktrim_cliphiger_nokl_562

ReasonSeg

Best GIOU: 0.6113

Best CIOU: 0.5497

Refcocog-test

Best GIOU: 0.7440

Best CIOU: 0.7392

How to Make Refcocog val and test set

conda activate /training/zilun/conda/vlm-r1
cd /training/zilun/dataset
python /training/zilun/vlm-r1seg/codebase/dataset_tools/refcocog_val_test_convert.py

# Dataset pushed to: https://huggingface.co/datasets/Zilun/RefCOCOg_test
# 5023 items for test. Change key words "val" and "test" to switch split
# Uncomment line 102-109 to double the test set with different answers.

How to Make ReasonSeg

# Download refCOCOg_9k_840

conda activate /training/zilun/conda/vlm-r1
cd /training/zilun/vlm-r1seg/codebase

# run dataset_tools/reasoningseg_dataset_transform.py to generate masks and vlmr1_refcocog_9k_840.json
python dataset_tools/reasoningseg_dataset_transform.py

# run dataset_tools/dataset_transform_refcocog9k_mask.py to make refCOCOg_9k_840_mask hf dataset
python dataset_tools/dataset_transform_refcocog9k_mask.py

How to evaluate

conda activate /training/zilun/conda/seg_zero
cd /training/zilun/vlm-r1seg/codebase/Seg_zero

# Eval all checkpoints or single checkpoints
bash evaluation_scripts/eval_reasonseg_group.sh

# Eval single checkpoint and visualize
bash evaluation_scripts/eval_reasonseg.sh

How to train

conda activate /training/zilun/conda/seg_zero

cd /training/zilun/vlm-r1seg/codebase/Seg-Zero

bash training_scripts/run_qwen2_5_7b_refCOCOg.sh

bash merge_ckpt.sh

bash rm_pt.sh workdir/run_qwen2_5_7b_refCOCOg

# sudo chmod 777 -R first
wandb sync /training/zilun/Seg-Zero/wandb/offline-run-20250424_230653-mg5vqmv2/