✨X-SAM✨

From Segment Anything to Any Segmentation

Hao Wang^1,2, Limeng Qiao³, Zequn Jie³, Zhijian Huang¹, Chengjian Feng³,

Qingfang Zheng², Lin Ma³, Xiangyuan Lan^2📧, Xiaodan Liang^1📧

¹ Sun Yat-sen University, ² Peng Cheng Laboratory, ³ Meituan Inc.

^📧 Corresponding author

👀 Notice

Note: X-SAM is under active development, and we will continue to update the code and documentation. Please check TODO to get our development schedule.

Reproducing AAAI 2026 results: To reproduce the results reported in our AAAI 2026 submission, please switch to the AAAI26_Archive branch:

git clone -b AAAI26_Archive https://github.com/wanghao9610/X-SAM.git

We strongly recommend that everyone uses English to communicate in issues. This helps developers from around the world discuss, share experiences, and answer questions together.

If you have any questions or would like to collaborate, please feel free to open an issue or reach out to me at wanghao9610@gmail.com.

💥 Updates

• 2026-07-03: ✨ We update the codebase to align with X2SAM, along with a new training recipe based on Qwen3-VL.
• 2026-06-18: 🎉🎉🎉 Congratulations! 🎉🎉🎉 X2SAM has been accepted by ECCV 2026!
• 2026-04-28: We release X2SAM, a new project for any segmentation in images and videos. X-SAM training configs are also supported in X2SAM — welcome to try it!
• 2026-01-29: We update the camera-ready version of our paper and uploaded the official paper poster for AAAI 2026.
• 2025-11-21: We release the code for X-SAM with Qwen3-4B-Instruct-2507 and Qwen3-1.7B. We will release the weights soon.
• 2025-11-19: We release the code for Training X-SAM. Welcome to try it! If you have any questions, please feel free to open an issue.
• 2025-11-08: 🎉🎉🎉 Congratulations! 🎉🎉🎉 X-SAM has been accepted by AAAI 2026! We will release all the code in the coming week!
• 2025-09-28: We update the Local Demo inference script — you can run local inference instead of on the Web Demo.
• 2025-08-11: Thanks for your great attention to our work! We have deployed another Online Demo2. You can also try it if Online Demo1 is not available.
• 2025-08-11: We released the effective code for Evaluation on Segmentation Benchmarks. We have updated all code except for Training X-SAM.
• 2025-08-10: We released the detailed instructions for Demo Deployment.
• 2025-08-08: We released the simple code for Evaluation on VLM Benchmarks.
• 2025-08-06: We are excited to publish the Technical Report, please check it out for more technical details.
• 2025-08-05: We provided the Model Weights on HuggingFace🤗.
• 2025-07-26: We deployed the Online Demo, you can try it now!

🚀 Highlights

This repository provides the official PyTorch implementation, pre-trained models, training, evaluation, visualization, and demo code of X-SAM:

• X-SAM introduces a unified multimodal large language model (MLLM) framework, extending the segmentation paradigm from segment anything to any segmentation, thereby enhancing pixel-level perceptual understanding.

• X-SAM proposes a novel Visual GrounDed (VGD) segmentation task, which segments all instance objects using interactive visual prompts, empowering the model with visually grounded, pixel-wise interpretative capabilities.

• X-SAM presents a unified training strategy that enables co-training across multiple datasets. Experimental results demonstrate that X-SAM achieves state-of-the-art performance on various image segmentation benchmarks, highlighting its efficiency in multimodal, pixel-level visual understanding.

✨ This repository provides unified and effective code for training, evaluation, and visualization of segmentation MLLMs. We hope this repository will promote further research on MLLMs. For image-and-video any-segmentation, please also check out our follow-up project X2SAM.

📖 Table of Contents

• Abstract
• Overview
• Benchmarks
• Quickstart
• Demo
• TODO
• Acknowledge
• Citation

🔖 Abstract

Large Language Models (LLMs) demonstrate strong capabilities in broad knowledge representation, yet they are inherently deficient in pixel-level perceptual understanding. Although the Segment Anything Model (SAM) represents a significant advancement in visual-prompt-driven image segmentation, it exhibits notable limitations in multi-mask prediction and category-specific segmentation tasks, and it cannot integrate all segmentation tasks within a unified model architecture. To address these limitations, we present X-SAM, a streamlined Multimodal Large Language Model (MLLM) framework that extends the segmentation paradigm from segment anything to any segmentation. Specifically, we introduce a novel unified framework that enables more advanced pixel-level perceptual comprehension for MLLMs. Furthermore, we propose a new segmentation task, termed Visual GrounDed (VGD) segmentation, which segments all instance objects with interactive visual prompts and empowers MLLMs with visual grounded, pixel-wise interpretative capabilities. To enable effective training on diverse data sources, we present a unified training strategy that supports co-training across multiple datasets. Experimental results demonstrate that X-SAM achieves state-of-the-art performance on a wide range of image segmentation benchmarks, highlighting its efficiency for multimodal, pixel-level visual understanding.

🔍 Overview

Figure 1: Overview of X-SAM. The Vision Encoder extracts global visual representations, while the Mask Encoder captures fine-grained visual features. The Large Language Model generates the language response and produces the latent condition embedding, which guides the Mask Decoder in generating the segmentation mask.

📊 Benchmarks

Table 1: Comparison of state-of-the-art segmentation methods across image segmentation benchmarks.

👉 More benchmark results can be found in benchmarks.md.

👉 To reproduce these AAAI 2026 submission results, use the AAAI26_Archive branch (see Notice).

🏁 Quickstart

1. Structure

We provide a detailed project structure for X-SAM. Please follow this structure to organize the project.

📁 Project

X-SAM
├── datas
│   ├── img_chat
│   ├── img_gcgseg
│   ├── img_genseg
│   ├── img_intseg
│   ├── img_ovseg
│   ├── img_reaseg
│   ├── img_refseg
│   ├── img_vgdseg
│   └── LMUData
├── inits
│   ├── huggingface
│   ├── mask2former-swin-large-coco-panoptic
│   ├── Phi-3-mini-4k-instruct
│   ├── sam-vit-large
│   └── X-SAM
├── xsam
│   ├── requirements
│   └── xsam
│       ├── configs
│       ├── dataset
│       ├── demo
│       ├── engine
│       ├── evaluation
│       ├── model
│       ├── structures
│       ├── tools
│       └── utils
├── wkdrs
│   ├── s1_train
│   │   └── ...
│   ├── s2_train
│   │   └── ...
│   ├── s3_train
│   │   └── ...
│   └── ...
...

2. Environment

Basic setup

# 1) Clone X-SAM and enter project home directory
git clone https://github.com/wanghao9610/X-SAM.git
cd X-SAM
export PROJ_HOME="$(realpath ./)"
export PYTHONPATH="$PROJ_HOME/xsam:$PYTHONPATH"

# 2) Create and activate conda environment
conda create -n xsam python=3.10 -y
conda activate xsam

# 3) Install X-SAM dependencies
cd "$PROJ_HOME/xsam"
pip install -r requirements/runtime.txt
pip install -r requirements/deepspeed.txt
pip install -r requirements/xsam.txt

# 4) Compile Deformable-Attention
cd "$PROJ_HOME/xsam/xsam/model/ops"
bash make.sh

# 5) Setup .env (machine-local paths for tooling; .env is gitignored)
cd "$PROJ_HOME"
cp -n .env.example .env
# CONDA_HOME  — conda installation root
# PYTHON_HOME — active env root (xsam); basename is the env name
sed -i "s|YOUR_CONDA_HOME|$(conda info --base)|; s|YOUR_PYTHON_HOME|${CONDA_PREFIX}|" .env

Optional: set CUDA_HOME

export CUDA_HOME="your_cuda12.4_path"
export PATH="$CUDA_HOME/bin:$PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib64:$LD_LIBRARY_PATH"
echo -e "CUDA version:\n$(nvcc -V)"

Optional: install gcc 11

conda install gcc=11 gxx=11 -c conda-forge -y

Optional: install VLMEvalKit

cd "$PROJ_HOME"
git clone -b v0.3rc1 https://github.com/open-compass/VLMEvalKit.git
cd VLMEvalKit
pip install -e .

3. Dataset

Please refer to datasets.md for detailed instructions on data preparation.

4. Model

Please refer to models.md for detailed instructions on model preparation.

5. Training

We provide a comprehensive script that covers the entire pipeline, including training, evaluation, and visualization. For detailed instructions, please refer to gpu_run.sh.

cd "$PROJ_HOME"

# Distributed training across multiple nodes.
# Set NODE_RANK to specify the rank (ID) of each node in distributed training.
# MASTER_ADDR and MASTER_PORT should be set to the IP address and port of your master node.
# Execute the following commands on every machine, updating NODE_RANK for each node accordingly.

# 1) Stage 1: Segmentor Fine-tuning
NUM_NODES=1 NODE_RANK=0 GPU_PER_NODE=8 MASTER_ADDR=127.0.0.1 MASTER_PORT=29510 \
  bash runs/gpu_run.sh \
  xsam/xsam/configs/xsam/s1_train/xsam_sam_vit_large_m2f_e36_gpu16.py \
  "train"

# 2) Stage 2: Alignment Pre-training (Phi-3 example)
NUM_NODES=1 NODE_RANK=0 GPU_PER_NODE=8 MASTER_ADDR=127.0.0.1 MASTER_PORT=29510 \
  bash runs/gpu_run.sh \
  xsam/xsam/configs/xsam/s2_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_e1_gpu16.py \
  "train"

# 3) Stage 3: Mixed Fine-tuning (Phi-3 example)
NUM_NODES=1 NODE_RANK=0 GPU_PER_NODE=8 MASTER_ADDR=127.0.0.1 MASTER_PORT=29510 \
  bash runs/gpu_run.sh \
  xsam/xsam/configs/xsam/s3_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py \
  "train segeval vlmeval visualize"

More Stage-3 configs

Backbone	Config
Phi-3-mini-4k-instruct	xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py
Qwen3-4B-Instruct-2507	xsam_qwen3_4b_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py
Qwen3-1.7B	xsam_qwen3_1x7b_wothinking_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py
Qwen3-VL-4B-Instruct (LoRA)	xsam_qwen3_vl_4b_instruct_sam_vit_large_m2f_e1_gpu16_lora.py
Qwen3-VL-2B-Instruct (LoRA)	xsam_qwen3_vl_2b_instruct_sam_vit_large_m2f_e1_gpu16_lora.py

6. Evaluation

Segmentation Benchmarks

cd "$PROJ_HOME"
bash runs/gpu_run.sh \
  xsam/xsam/configs/xsam/s3_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py \
  "segeval"

VLM Benchmarks

cd "$PROJ_HOME"
bash runs/gpu_run.sh \
  xsam/xsam/configs/xsam/s3_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py \
  "vlmeval"

7. Visualization

cd "$PROJ_HOME"
bash runs/gpu_run.sh \
  xsam/xsam/configs/xsam/s3_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py \
  "visualize"

8. Tools

Dataset Exploration

We provide a tool for dataset exploration. You can use it to explore the dataset and get visualizations of the samples.

cd "$PROJ_HOME"
python xsam/xsam/tools/explore.py \
  xsam/xsam/configs/xsam/s3_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py \
  --output-dir "wkdrs/dataset_exploration" \
  --subset train \
  --max-samples 100

Model Conversion

We provide a tool for model conversion. You can use it to convert the checkpoint to the Hugging Face format.

cd "$PROJ_HOME"
python xsam/xsam/tools/model_tools/pth_to_hf.py \
  xsam/xsam/configs/xsam/s3_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py \
  "wkdrs/s3_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16" \
  --pth_model latest

💻 Demo

Local Demo

🏞️ Inference

cd "$PROJ_HOME"
python xsam/xsam/demo/demo.py \
  xsam/xsam/configs/xsam/s3_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py \
  --pth_model "inits/X-SAM/s3_mixed_finetune/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_large_m2f_gpu16_mixed_finetune/pytorch_model.bin" \
  --task-name TASK_NAME \
  --image INPUT_IMAGE/INPUT_DIR \
  --prompt INPUT_PROMPT \
  --vprompt-masks INPUT_VPROMPT_MASKS

🏞️ Examples

# Example: img_chat
python xsam/xsam/demo/demo.py \
  xsam/xsam/configs/xsam/s3_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py \
  --pth_model "inits/X-SAM/s3_mixed_finetune/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_large_m2f_gpu16_mixed_finetune/pytorch_model.bin" \
  --task-name img_chat \
  --image xsam/xsam/demo/sample.jpg \
  --prompt "What is unusal about this image?"

# Example: img_genseg
python xsam/xsam/demo/demo.py \
  xsam/xsam/configs/xsam/s3_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py \
  --pth_model "inits/X-SAM/s3_mixed_finetune/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_large_m2f_gpu16_mixed_finetune/pytorch_model.bin" \
  --task-name img_genseg \
  --image xsam/xsam/demo/sample.jpg \
  --output-dir "wkdrs/demo_outputs" \
  --prompt "ins: person, bird, boat; sem: water, sky"

# Example: img_refseg
python xsam/xsam/demo/demo.py \
  xsam/xsam/configs/xsam/s3_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py \
  --pth_model "inits/X-SAM/s3_mixed_finetune/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_large_m2f_gpu16_mixed_finetune/pytorch_model.bin" \
  --task-name img_refseg \
  --image xsam/xsam/demo/sample.jpg \
  --output-dir "wkdrs/demo_outputs" \
  --prompt "the ironing man"

# Example: img_reaseg
python xsam/xsam/demo/demo.py \
  xsam/xsam/configs/xsam/s3_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py \
  --pth_model "inits/X-SAM/s3_mixed_finetune/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_large_m2f_gpu16_mixed_finetune/pytorch_model.bin" \
  --task-name img_reaseg \
  --image xsam/xsam/demo/sample.jpg \
  --output-dir "wkdrs/demo_outputs" \
  --prompt "What can be used to warm clothes?"

# Example: img_gcgseg
python xsam/xsam/demo/demo.py \
  xsam/xsam/configs/xsam/s3_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py \
  --pth_model "inits/X-SAM/s3_mixed_finetune/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_large_m2f_gpu16_mixed_finetune/pytorch_model.bin" \
  --task-name img_gcgseg \
  --image xsam/xsam/demo/sample.jpg \
  --output-dir "wkdrs/demo_outputs"

# Example: img_intseg (requires visual prompt masks)
python xsam/xsam/demo/demo.py \
  xsam/xsam/configs/xsam/s3_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py \
  --pth_model "inits/X-SAM/s3_mixed_finetune/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_large_m2f_gpu16_mixed_finetune/pytorch_model.bin" \
  --task-name img_intseg \
  --image xsam/xsam/demo/sample.jpg \
  --output-dir "wkdrs/demo_outputs" \
  --vprompt-masks PATH_TO_VPROMPT_MASK.png

# Example: img_vgdseg (requires visual prompt masks)
python xsam/xsam/demo/demo.py \
  xsam/xsam/configs/xsam/s3_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py \
  --pth_model "inits/X-SAM/s3_mixed_finetune/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_large_m2f_gpu16_mixed_finetune/pytorch_model.bin" \
  --task-name img_vgdseg \
  --image xsam/xsam/demo/sample.jpg \
  --output-dir "wkdrs/demo_outputs" \
  --vprompt-masks PATH_TO_VPROMPT_MASK1.png PATH_TO_VPROMPT_MASK2.png

Web Demo

🛠️ Deployment

cd "$PROJ_HOME"
python xsam/xsam/demo/app.py \
  xsam/xsam/configs/xsam/s3_train/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_vit_large_m2f_e1_gpu16.py \
  --pth_model "inits/X-SAM/s3_mixed_finetune/xsam_phi3_mini_4k_instruct_siglip2_so400m_p14_384_sam_large_m2f_gpu16_mixed_finetune/pytorch_model.bin" \
  --log-dir "wkdrs/app_logs" \
  --seed 0 \
  --port 7860

Then, you can access the demo website at http://localhost:7860.

✅ TODO

• Release the Online Demo.
• Release the Model Weights.
• Release the Technical Report.
• Release the code for Evaluation on VLM Benchmarks.
• Release the code for Demo Deployment.
• Release the code for Evaluation on Segmentation Benchmarks.
• Release the code for Training X-SAM.
• Release the code and weight for X-SAM with Qwen3.
• Release the code and weight for X-SAM with Qwen3-VL.
• Release the inference and demo code supporting transformers.
• Release the code and instructions for training with Ascend NPU.

🙏 Acknowledge

This project has referenced some excellent open-sourced repos (xtuner, VLMEvalKit, Sa2VA). Thanks for their wonderful works and contributions to the community!

📌 Citation

If you find X-SAM and X2SAM are helpful for your research or applications, please consider giving us a star 🌟 and citing the following papers by the following BibTex entry.

@inproceedings{wang2026xsam,
  title={X-SAM: From segment anything to any segmentation},
  author={Wang, Hao and Qiao, Limeng and Jie, Zequn and Huang, Zhijian and Feng, Chengjian and Zheng, Qingfang and Ma, Lin and Lan, Xiangyuan and Liang, Xiaodan},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={40},
  number={31},
  pages={26187--26196},
  year={2026}
}

@article{wang2026x2sam,
  title={X2SAM: Any Segmentation in Images and Videos},
  author={Wang, Hao and Qiao, Limeng and Zhang, Chi and Wan, Guanglu and Ma, Lin and Lan, Xiangyuan and Liang, Xiaodan},
  journal={arXiv preprint arXiv:2605.00891},
  year={2026}
}