PromptPilot: Game-Theoretic Multi-Agent Prompt Optimization for Segment Anything

News | Description | Highlights | Framework | Repository Structure | Installation | Data Preparation | Running | Results | Citation | Acknowledgement

PromptPilot is a hierarchical multi-agent reinforcement learning framework for point prompt optimization with frozen DINOv2 and Segment Anything Model (SAM). It formulates automated prompt construction as a cooperative game, where specialized agents balance semantic consistency, spatial coverage, and prompt-level credit assignment without updating the foundation models during inference.

News

• 2026.05: PromptPilot is accepted by ICML 2026.
• 2026.06: Code is released.
• 2026.07: Paper is available on OpenReview: https://openreview.net/forum?id=H6T8ECJafn.

Description

Few-shot segmentation with vision foundation models relies on high-quality prompts to segment unseen categories from limited support annotations. Existing prompt construction strategies often depend on test-time adaptation, fixed heuristic sampling, or monolithic reinforcement learning. These designs make it difficult to jointly optimize semantic consistency, spatial coverage, and prompt-level credit assignment.

PromptPilot addresses this limitation by decomposing prompt optimization into complementary decision spaces. A Feature Agent improves semantic discriminability in the DINOv2 feature space, a Physical Agent improves spatial coverage in the image plane, and a centralized Manager Agent selects actions according to SAM feedback and local marginal contribution. The framework operates as an inference-time prompt optimizer and keeps DINOv2 and SAM frozen.

Highlights

• Game-theoretic prompt optimization: Prompt construction is formulated as sequential decision making over a cooperative prompt game.
• Hierarchical multi-agent policy: Feature and Physical agents propose complementary prompt modifications, while a Manager agent arbitrates the executed action.
• SAM-aligned feedback: Global DSC feedback from SAM aligns prompt optimization with the final segmentation objective.
• Prompt credit assignment: Leave-one-out marginal contribution with EMA smoothing estimates the utility of individual prompts in the current prompt coalition.

Framework

Repository Structure

The public repository contains the main training, optimization, and evaluation entry points:

PromptPilot/
+-- agents/                 # Feature, Physical, and Manager agents
+-- feature_matching/       # DINOv2-based matching and prompt initialization
+-- segmenter/              # SAM interface and mask prediction utilities
+-- arg_game_multi.py       # Multi-agent prompt optimization
+-- train_game.py           # Agent training
+-- evaluate_all.py         # Benchmark evaluation
+-- arg_run.sh              # Training/optimization launch script
+-- evaluate_run.sh         # Evaluation launch script
+-- requirements.txt        # Python dependencies
+-- utils.py                # Shared utilities

Installation

Clone the repository and install dependencies:

git clone https://github.com/L-AILab/PromptPilot.git
cd PromptPilot
pip install -r requirements.txt

Prepare the required frozen backbones:

• DINOv2 for patch-level feature extraction.
• SAM for promptable mask decoding.

Place downloaded checkpoints in the paths expected by your local configuration or scripts.

Data Preparation

PromptPilot is evaluated on natural image, medical image, and video object segmentation datasets:

• PASCAL VOC
• COCO
• ISIC
• Kvasir
• GBM
• JSRT
• DAVIS 2016
• DAVIS 2017

Organize datasets according to the paths used in arg_run.sh, evaluate_run.sh, or your local configuration. A common layout is:

Dataset/
+-- ClassName/
    +-- reference_images/
    +-- reference_masks/
    +-- target_images/
    +-- target_masks/

Running

Train the PromptPilot agents:

bash arg_run.sh

Run multi-agent prompt optimization directly:

python arg_game_multi.py

Evaluate trained policies:

bash evaluate_run.sh

or run the evaluation entry point:

python evaluate_all.py

For custom paths, checkpoints, or benchmark splits, edit the corresponding shell scripts and configuration arguments in the repository.

Results

Natural and Medical Image Segmentation

PromptPilot achieves state-of-the-art automated prompt optimization performance across natural and medical segmentation benchmarks. Gray rows in the paper correspond to manual-prompt or adaptation-based reference settings.

Method	VOC		COCO		ISIC		Kvasir		GBM		JSRT
	DSC	mIoU	DSC	mIoU	DSC	mIoU	DSC	mIoU	DSC	mIoU	DSC	mIoU
SAM (Point)	50.4	41.2	52.2	42.8	69.3	60.1	72.4	63.6	65.4	52.5	59.6	43.1
SAM (Box)	80.1	72.8	72.3	64.5	84.6	75.9	78.6	72.0	55.8	41.7	60.8	45.9
SAM-Adapter	--	--	--	--	74.7	65.4	86.3	79.1	51.9	41.6	90.0	82.2
SAM-LoRA	--	--	--	--	91.0	84.4	83.4	75.8	67.5	55.5	87.3	81.6
PerSAM	55.3	49.3	26.5	22.7	63.9	54.5	29.3	19.9	43.6	33.2	58.2	41.5
PerSAM-f	54.5	48.0	23.8	20.2	60.6	51.4	33.3	25.0	45.2	35.2	57.3	40.8
VRP-SAM	48.8	40.8	25.2	19.2	64.5	55.5	28.0	17.9	19.7	12.3	49.8	33.4
Matcher	68.9	59.9	45.0	38.4	74.7	66.3	39.4	29.5	58.8	45.3	89.5	81.3
GBMSeg	55.3	47.7	32.9	26.9	59.2	48.3	40.2	28.9	59.1	45.0	79.6	66.9
FM-PPO	61.3	52.8	36.7	30.4	72.4	62.7	44.9	33.9	66.0	53.1	87.7	78.4
PromptPilot (Ours)	69.3	61.3	54.4	47.1	78.6	69.0	49.3	40.1	72.5	60.6	89.8	81.7

Video Object Segmentation

Method	DAVIS 2016		DAVIS 2017
	J	F	J	F
PerSAM	68.2	69.7	61.7	67.7
PerSAM-f	69.3	70.6	64.9	70.3
VRP-SAM	47.4	47.4	35.1	39.0
Matcher	78.5	81.7	65.1	73.0
GBMSeg	76.5	74.6	56.1	54.9
FM-PPO	75.9	76.0	61.5	62.4
PromptPilot (Ours)	80.1	81.8	69.3	71.1

Ablation Study

On four medical benchmarks, the full PromptPilot design reaches 72.6 DSC and 62.9 mIoU on average. Using both actor agents with Manager supervision improves the average mIoU to 61.3, and adding leave-one-out marginal contribution further improves it by 1.6 points.

Actor Agents		Manager Agent		ISIC		Kvasir		GBM		JSRT		Average
Feature	Physical	DSC	LOO	DSC	mIoU	DSC	mIoU	DSC	mIoU	DSC	mIoU	DSC	mIoU
yes	no	no	no	51.4	41.1	22.4	15.8	40.9	31.4	63.9	47.5	44.7	34.0
no	yes	no	no	52.3	42.2	22.6	15.9	39.9	30.5	65.5	49.2	45.1	34.5
yes	yes	no	no	50.3	39.9	23.2	16.2	41.4	31.6	65.6	49.2	45.1	34.2
yes	no	yes	no	74.4	65.3	36.3	27.4	61.1	48.5	79.3	67.7	62.8	52.2
no	yes	yes	no	74.8	65.7	36.8	27.8	64.0	51.4	79.2	67.0	63.7	53.0
yes	yes	yes	no	77.9	68.3	47.7	38.4	71.1	58.9	88.3	79.6	71.3	61.3
yes	yes	yes	yes	78.6	69.0	49.3	40.1	72.5	60.6	89.8	81.7	72.6	62.9

Plug-and-Play Prompt Optimization

Setting	Method	ISIC		Kvasir		GBM		JSRT		Average
		DSC	mIoU	DSC	mIoU	DSC	mIoU	DSC	mIoU	DSC	mIoU
Feature Matching	Baseline	71.6	61.5	36.9	28.8	49.9	38.9	73.2	60.5	57.9	47.4
Feature Matching	FM-PPO	72.4	62.7	44.9	33.9	66.0	53.1	87.7	78.4	67.8	57.0
Feature Matching	FM-PromptPilot	78.6	69.0	49.3	40.1	72.5	60.6	89.8	81.7	72.6	62.9
Coarse Segmentation	Baseline	46.0	36.0	20.4	14.5	25.4	18.3	44.9	31.2	34.2	25.0
Coarse Segmentation	CS-PPO	51.2	41.1	22.4	16.2	21.2	14.7	58.5	42.3	38.3	28.6
Coarse Segmentation	CS-PromptPilot	78.5	69.0	45.9	36.4	71.2	58.4	89.3	81.1	71.2	61.2

Citation

If you find this project useful, please cite:

@inproceedings{
shi2026promptpilot,
title={PromptPilot: Game-Theoretic Multi-Agent Prompt Optimization for Segment Anything},
author={Guangze Shi and Yingjie Mi and Jia Shen and Feixue Shao and Jiarui Cao and Yexin Lai and Xueyu Liu and Rui Wang and Yongfei Wu and Mingqiang Wei},
booktitle={Forty-third International Conference on Machine Learning},
year={2026},
url={https://openreview.net/forum?id=H6T8ECJafn}
}

Acknowledgement

This work builds on DINOv2, SAM, and Plug-and-Play PPO. We thank the authors of these projects for their open research contributions.

This work was supported by the National Natural Science Foundation of China, the Fundamental Research Program of Shanxi Province, and the Key Research and Development Program of Shanxi Province.