CloakDiff

Official implementation of “Imperceptible and Reversible Adversarial Examples against Vision-Language Models for Privacy Protection”, accepted by ACM Multimedia 2026 (MM ’26).

CloakDiff protects shared images from text-based privacy inference by vision-language models (VLMs). It first generates a high-fidelity adversarial image through diffusion-based editing, then uses an invertible neural network (INN) to embed the original image into the adversarial image. The resulting reversible adversarial example (RAE) is designed to disrupt VLM interpretation while supporting recovery of the original image in a trusted setting.

Highlights

• Cross-model and cross-prompt protection: adversarial guidance acts in both the pixel and latent spaces to improve transferability across VLM architectures and prompts.
• High visual fidelity: self-attention retention constrains structural changes during diffusion sampling.
• Reversibility: an INN combines the original and adversarial images into an RAE and reconstructs the original image through inverse propagation.
• EDM-Heuristic Sampling (EHS): a nonlinear diffusion schedule allocates sampling steps across noise levels through a single parameter, rho.

The paper evaluates CloakDiff on the NIPS 2017 adversarial competition dataset and MS-COCO, covering nine VLMs: BLIP, BLIP-2, InstructBLIP, Flamingo, UniDiffuser, MiniGPT-4, LLaVA, Qwen2.5-VL, and InternVL.

Method Overview

CloakDiff consists of two stages:

1. Adversarial example generation. Stable Diffusion edits an input image using CLIP feature guidance and latent cross-attention guidance. A self-attention objective retains the image’s global structure, while EHS controls the diffusion trajectory.
2. Reversible adversarial example generation. The original image is treated as the secret image and the adversarial example as the cover image. The INN produces the final RAE and recovers the original image by running the network in reverse.

EDM-Heuristic Sampling

Figure 1. Sampling positions for EHS with 20 steps. Different values of rho redistribute the selected diffusion timesteps across noise levels; increasing rho shifts the schedule toward lower selected cumulative-alpha values.

Qualitative Results

Figure 2. Reversible adversarial examples evaluated from left to right on BLIP, UniDiffuser, Flamingo, and LLaVA. x_orig and x_rae show the model outputs for the original and reversible adversarial images, respectively.

Repository Structure

.
├── adv_generate.py          # Diffusion-based adversarial image generation
├── attentionControl.py      # Cross/self-attention control
├── rae_generate.py          # RAE generation, recovery, and quality metrics
├── model.py                 # INN model wrapper
├── hinet.py                 # HiNet architecture
├── invblock.py              # Invertible blocks
├── modules/                 # Wavelet and network utilities
├── config.py                # INN architecture/training configuration
├── environment.yaml         # Conda environment
└── Img/                     # README figures

Installation

A CUDA-capable GPU is required by the current implementation. Create the provided Conda environment with:

conda env create -f environment.yaml
conda activate cloakdiff

The environment specifies Python 3.10.19, PyTorch 2.5.1 with CUDA 12.1, Diffusers 0.30.3, and Transformers 4.30.2. On first use, adv_generate.py downloads stabilityai/stable-diffusion-2-base unless a local model path is supplied.

Usage

1. Prepare the adversarial-generation inputs

adv_generate.py currently reads the following fixed directory layout:

clean/
├── Img/                     # Original images
└── caption.txt              # One caption per original image
target/
├── Img/                     # Target images used as feature-shift anchors
└── caption.txt              # One caption per target image

Images are naturally sorted and paired by position. Keep the number and order of image files consistent with the corresponding caption files; each line in caption.txt represents one image caption.

2. Generate adversarial examples

python adv_generate.py \
  --device cuda:0 \
  --save_dir outputs/adv \
  --rho 3

The command saves images as outputs/adv/0000_adv_image.png, outputs/adv/0001_adv_image.png, and so on. It also writes the aggregate FID result to fid.txt in the working directory.

Important options:

Option	Default	Description
--pretrained_diffusion_path	stabilityai/stable-diffusion-2-base	Hugging Face model ID or local model directory
--diffusion_steps	20	Number of diffusion sampling steps
--start_step	15	Sampling step at which adversarial optimization begins
--iterations	50	Number of adversarial optimization iterations
--res	224	Processing resolution; must be at least 96 and divisible by 32
--guidance	2.5	Classifier-free guidance scale
--rho	2	EHS schedule parameter; the paper uses 3
--device	cuda:3	PyTorch device; set this explicitly for your machine

Although --images_root and --label_path are present in the command-line interface, the current script reads clean/ and target/ directly.

3. Generate reversible adversarial examples

The INN checkpoint is not included in this repository. Supply a checkpoint compatible with the HiNet architecture in model.py; the checkpoint must contain the model weights under the net key.

python rae_generate.py \
  --cover_path outputs/adv \
  --secret_path clean/Img \
  --steg_path outputs/rae \
  --rev_path outputs/recovered \
  --metric_path outputs/metrics.tsv \
  --model_path /path/to/inn_checkpoint.pt \
  --gpu_id 0

Here, cover_path contains adversarial images and secret_path contains the corresponding original images. Files are lexicographically sorted and paired by position, then resized to 224 × 224. The script produces:

• reversible adversarial examples in steg_path;
• reconstructed original images in rev_path;
• per-image and mean PSNR, SSIM, and LPIPS measurements in metric_path.

Use matching filenames or otherwise verify the sorted order before generation. If the two directories contain different numbers of supported images (.png, .jpg, or .jpeg), only the smaller paired subset is processed.

Acknowledgements

The evaluation data follows CroPA. The invertible-network implementation is based on HiNet.

Citation

If this work is useful in your research, please cite:

@article{lu2026imperceptible,
  title={Imperceptible and Reversible Adversarial Examples against Vision-Language Models for Privacy Protection},
  author={Lu, Qi and Zhou, Ziqi and Song, Yufei and Li, Zijing and Xue, Lulu and Li, Minghui and Hu, Shengshan and Zhang, Leo Yu},
  journal={arXiv preprint arXiv:2607.10329},
  year={2026}
}