Re-purposing SAM into Efficient Visual Projectors for MLLM-Based Referring Image Segmentation

This repo is a implementation of the Re-purposing SAM into Efficient Visual Projectors for MLLM-Based Referring Image Segmentation.

Introduction

Recently, Referring Image Segmentation (RIS) frameworks that pair the Multimodal Large Language Model (MLLM) with the Segment Anything Model (SAM) have achieved impressive results. However, adapting MLLM to segmentation is computationally intensive, primarily due to visual token redundancy. We observe that traditional patch-wise visual projectors struggle to strike a balance between reducing the number of visual tokens and preserving semantic clarity, often retaining overly long token sequences to avoid performance drops. Inspired by text tokenizers, we propose a novel semantic visual projector that leverages semantic superpixels generated by SAM to identify "visual words" in an image. By compressing and projecting semantic superpixels as visual tokens, our approach adaptively shortens the token sequence according to scene complexity while minimizing semantic loss in compression. To mitigate loss of information, we propose a semantic superpixel positional embedding to strengthen MLLM's awareness of superpixel geometry and position, alongside a semantic superpixel aggregator to preserve both fine-grained details inside superpixels and global context outside. Experiments show that our method cuts visual tokens by 93% without compromising performance, notably speeding up MLLM training and inference, and outperforming existing compressive visual projectors on RIS.

Preparing Project

Datasets

Prepare datasets following GLaMM. The downloaded datasets should be placed under PROJECT_ROOT/datasets.

The directory structure of datasets would like:

datasets
├── Refer_Segm
│   ├── refcoco
│   ├── refcoco+
│   ├── refcocog
│   ├── refclef
├── coco_2014
│   ├── train2014
│   │   ├── COCO_train2014_000000000009.jpg
│   │   ├── COCO_train2014_000000000025.jpg

Pre-trained Models

Download GLaMM-GranD-Pretrained and place it under PROJECT_ROOT/checkpoints.

The directory structure of checkpoints would like:

checkpoints
├── GLaMM-GranD-Pretrained
│   ├── config.json

Semantic Superpixels

The semantic superpixels are generated by FMA-WSSS. Download semantic superpixels HERE and place it under PROJECT_ROOT/experiments.

The directory structure of experiments would like:

experiments
├── SAM_superpixels
│   ├── coco_2014
│   │   ├── train2014
│   │   │   ├── sps
│   │   │   │   ├── COCO_train2014_000000000009.pkl
│   │   │   │   ├── COCO_train2014_000000000025.pkl
│   │   │   ├── viz

Usage

Run following commands for training and evaluation of SVP on RIS.

cd PROJECT_ROOT

bash scripts/train/SVP_train_eval.sh

Result

Please refer to our paper for detailed results. Our checkpoints and logs are available HERE.

Acknowledgement

This project is build upon GLaMM. Thanks for their wonderful works.

Pay the highest attribute to AlchemyCat, a powerful and user-friendly deep learning toolkit.

License

This repo is released under the Apache License.

Citation

If you find this project helpful for your research, please consider citing the following BibTeX entry.

@misc{yang2025repurposingsamefficientvisual,
      title={Re-purposing SAM into Efficient Visual Projectors for MLLM-Based Referring Image Segmentation},
      author={Xiaobo Yang and Xiaojin Gong},
      year={2025},
      eprint={2509.13676},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2509.13676},
}