FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models

FVG-PT: Adaptive Foreground View-Guided Prompt Tuning for Vision-Language Models
Haoyang Li^1,2, Liang Wang^1,2, Siyu Zhou¹, Jiacheng Sun², Jing Jiang¹, Chao Wang², Guodong Long¹ and Yan Peng².
¹Australian Artificial Intelligence Institute   ²Shanghai University

Arxiv Link: https://arxiv.org/abs/2603.08708

---

📜 Abstract

CLIP-based prompt tuning enables pretrained Vision-Language Models (VLMs) to efficiently adapt to downstream tasks. Although existing studies have made significant progress, they pay limited attention to changes in the internal attention representations of VLMs during the tuning process. In this paper, we attribute the failure modes of prompt tuning predictions to shifts in foreground attention of the visual encoder, and propose Foreground View-Guided Prompt Tuning (FVG-PT), an adaptive plug-and-play foreground attention guidance module, to alleviate the shifts. Concretely, FVG-PT introduces a learnable Foreground Reliability Gate to automatically enhance the foreground view quality, applies a Foreground Distillation Compensation module to guide visual attention toward the foreground, and further introduces a Prior Calibration module to mitigate generalization degradation caused by excessive focus on the foreground. Experiments on multiple backbone models and datasets show the effectiveness and compatibility of FVG-PT.

🔍 Framework

Figure 1. Overview of our proposed FVG-PT. As a plug-and-play method, in (a) tuning stage, FVG-PT obtains the foreground view x^fg of image x and fine-tunes the learnable (b) Foreground Reliability Gate to learn an adaptive foreground trust score r. Meanwhile, Foreground Distillation Compensation module inserts adapters after image-text alignment of frozen backbone model to guide visual attention toward the foreground. In parallel, independent Prior Calibration fine-tunes the (c) Backbone Reliability Gate on the decoupled new branch (indicated by dashed lines) to balance the tuned model and the CLIP prior.

💡 Our previous work on prompt tuning

• [CVPR 25] DPC: Dual-Prompt Collaboration for Tuning Vision-Language Models
  Haoyang Li, Liang Wang, Chao Wang, Jing Jiang, Yan Peng and Guodong Long.
  [Paper] [Project Page] [Poster]

• [ICME 25] MAO: Efficient Model-Agnostic Optimization of Prompt Tuning for Vision-Language Models
  Haoyang Li, Siyu Zhou, Liang Wang and Guodong Long.
  [Paper] [Project Page] [Poster]

• [arxiv] Raw Data Matters: Enhancing Prompt Tuning by Internal Augmentation on Vision-Language Models
  Haoyang Li, Liang Wang, Chao Wang, Siyu Zhou, Jing Jiang, Yan Peng and Guodong Long
  [Paper] [Project Page]

---

⚙️ Running

1. Create the environment and install Dassl.pytorch library. Please follow the instructions detailed in INSTALL.md.
2. Prepare the dataset. We release 11 prompt-tuning datasets with foreground views on [🤗HuggingFace], just use them directly.
   These foreground views are generated by SEEM. They are put in the ./mask folder in each dataset.
   Details of data preparation can be found in DATASETS.md.
3. Run fine-tuning script on the backbone models first (e.g., CoOp):
   

   python run_tuning.py --dataset caltech101 --trainer CoOp --seed_list 1 --sub_class base
   python run_tuning.py --dataset caltech101 --trainer CoOp --seed_list 1 --sub_class new

4. Run FVG-PT fine-tuning based on the pre-tuned backbone models:
   

    python run_tuning.py --dataset caltech101 --trainer FVGPT_CoOp --seed_list 1 --sub_class base --lambda_base 10.0
    python run_tuning.py --dataset caltech101 --trainer FVGPT_CoOp --seed_list 1 --sub_class new --lambda_base 10.0

Code Statement

We are still working on organizing the code repository.

✅ One-key fine-tuning pipeline for backbone models (run_tuning.py)

✅ FVG-PT foreground datasets and corresponding dassl DataLoader

📅 (To-do) FVG-PT Trainers

We promise to release our full code implementation in the future.