Key Words: Visual Reprogramming, Closed-loop Correction Reprogramming (CCR), PID, Proportional Adjustment Controller (PAC), Fine-grained, Machine Learnin
Abstract: Visual Reprogramming (VR) adapts pre-trained models to new tasks through pixel-level attention modulation without parameter modification. While existing methods achieve competent performance on basic classification, they dispersed attention in critical discriminative regions for fine-grained tasks. Inspired by PID control theory, we propose Closed-loop Correction Reprogramming (CCR) that integrates proportional feedback. Concretely, the framework comprises dual streams: a Foundation Flow for initial attention patterns and a Correction Flow that iteratively refines them with residual feedback, alternating between both. A Proportional Adjustment Controller (PAC) dynamically calibrates perturbation intensity via learnable error mapping--enhancing the correction flow's contribution in response to increased foundational stream errors, otherwise maintaining the foundation's dependable attributes. Experiments on 11 benchmarks demonstrate CCR achieves up to 10.8% accuracy gain with only 0.64% parameter increase, attaining 8.62% average improvement on five challenging fine-grained datasets (GTSRB, FLOWERS102, DTD, UCF101, FOOD101). The framework offers enhanced visual cues that improve discrimination in fine-grained classification.
Method: The Closed-loop Correction Reprogramming (CCR) model enhances pre-trained models for fine-grained classification tasks using Visual Reprogramming (VR). It employs a dual-stream architecture consisting of Foundation Flow and Correction Flow to dynamically refine attention maps.
- Foundation Flow generates initial perturbation patterns for coarse adaptation.
- Correction Flow iteratively refines these patterns based on residual feedback from the Foundation Flow, improving the attention distribution for fine-grained tasks.
A key component, the Proportional Adjustment Controller (PAC), estimates the perturbation magnitude during testing without needing ground-truth labels. PAC dynamically adjusts the influence of the two flows: it strengthens the Correction Flow when the Foundation Flow’s errors exceed a threshold, and preserves reliable features when errors are small.
- For CIFAR10, CIFAR100, GTSRB, SVHN, simply run our code and use TorchVision to download them automatically.
- For other datasets, follow this paper to prepare them.
Then put all the download datasets in /dataset/
-
Python (3.10.0)
-
PyTorch (2.5.1)+cu124
-
TorchVision (0.20.1)+cu124
pip install -r requirements.txt
To train the model, simply run the CCR.ipynb notebook. Modify the following section in the notebook to adjust the model and dataset settings:
class Args:
network = "resnet18" # Choose from ["resnet18", "resnet50", "ViT_B32"]
dataset = "flowers102" # Choose from ["cifar10", "cifar100", "gtsrb", "svhn", "food101", "eurosat", "sun397", "UCF101", "flowers102", "DTD", "oxfordpets"]The model_pth directory should contain the following pre-trained models:
- resnet18 https://drive.google.com/file/d/1zSCBxYnEOu7rOZo8BL3GxsjSSFmjzl5o/view?usp=drive_link
- resnet50 https://drive.google.com/file/d/17mLBH3g3L1lkfIopaxJJdcL7b6yij5sI/view?usp=drive_link
- ViT_B32
The final training results will be saved in the results folder, allowing easy access to evaluate and analyze the outcomes of your training.
Visual results of trained VR on the Flowers102 dataset.ResNet-18 is used as the pre-trained model as an example.
This repo is built upon these previous works: