🏥 Medical-SAM3

A Foundation Model for Universal Prompt-Driven Medical Image Segmentation

  

📰 News

• [2026-06]: 🎓 Medical SAM 3 V2 (3D) training & evaluation code released under medical/, pretrained weights released in huggingface.
• [2026-01-20]: 🚀 Pretrained weights for Medical-SAM3 are released!
• [2026-01-15]: 📄 Paper is available on arXiv.

⚡ Inference & evaluation (2D medical benchmarks)

Toolkit for 2D datasets (CHASE_DB1, STARE, CVC-ClinicDB, etc.) — box/text prompts, baseline comparison, visualization.

cd inference
python run_medsam3_evaluation.py --checkpoint /path/to/checkpoint.pt --model-name medsam3

SAM3 is bundled in this repo (sam3/); no separate clone required.

🎬 Training & 3D evaluation

Fine-tune SAM3 on 3D annotations and run held-out 3D / JSON eval (train-aligned protocol).

export MEDSAM3_ROOT=$(pwd)
export MEDSAM3_DATA_ROOT=/path/to/project
export MEDSAM3_CVPR_ROOT=$MEDSAM3_DATA_ROOT/converted_cvpr_biomedsegfm

pip install -e ".[train]"
bash medical/training/prepare_config_paths.sh   # once, if yaml paths differ

# Train
python sam3/train/train.py -c configs/medsam3_stage1_train_all_unified --num-gpus 4

# Eval 
python medical/evaluation/3d_val.py \
  --ckpt experiments/medsam3_stage1_train_all_unified/checkpoints/checkpoint.pt \
  --out-dir experiments/results/3d_val/unified \
  --eval-all-volumes --text-only --use-train-prompts

Path	Role
inference/	2D image inference & public benchmark eval
medical/	3D training + 3D/JSON evaluation
sam3/	SAM3 model, trainer, Hydra config
assets/	BPE vocabulary
experiments/	Checkpoints & eval outputs (runtime)

📅 Todo List

Feature	Status	Description
Demo	🚧 Doing	Online interactive demo.
Data Scaling	🚧 Doing	Expand training corpus and benchmarks.
3D Training	✅ Released	medical/ + medsam3_stage1_train_all_unified config.
Medical-SAM3 Agent	📅 Planned	LLM agentic segmentation.

📝 Citation

@article{jiang2026medicalsam3,
  title={Medical SAM3: A Foundation Model for Universal Prompt-Driven Medical Image Segmentation},
  author={Jiang, Chongcong and Ding, Tianxingjian and Song, Chuhan and Tu, Jiachen and Yan, Ziyang and Shao, Yihua and Wang, Zhenyi and Shang, Yuzhang and Han, Tianyu and Tian, Yu},
  journal={arXiv preprint arXiv:2601.10880},
  year={2026},
  url={https://arxiv.org/abs/2601.10880}
}