This repository provides the official PyTorch implementation of the following paper:

StyleGuard: Preventing Text-to-Image-Model-based Style Mimicry Attacks by Style Perturbations

link: https://arxiv.org/abs/2505.18766

Yanjie Li, Wenxuan Zhang, Xinqi LYU, Yihao LIU, Bin Xiao

Hong Kong Polytechnic University

NeurIPS 2025 poster

🛡️

TL;DR: StyleGuard defends against unauthorized style mimicry in diffusion models (e.g., DreamBooth/Textual Inversion) via style-aware adversarial perturbations robust to purification attacks.

Key Innovations

• 🎨 Style Loss: Latent-space optimization disrupts style transfer while maintaining model-agnostic transferability
• ⚔️ Upscale Loss: Ensemble purification resistance via adversarial training with upscalers/purifiers
• 🏆 SOTA Robustness: Outperforms Glaze/Anti-DreamBooth against:
  • Random transformations (blur, JPEG, etc.)
  • Diffusion purification (DiffPure, Noise Upscaling)

Example Output Images

Original Artwork

Vincent van Gogh - "Orchard in Blossom Bordered by Cypresses" (1888)

Protected Version

Style-protected version with StyleGuard

---

DreamBooth Generation Results

Clean Output (Unprotected)

Prompt: "A sks painting of a mountain landscape with a blue sky"

Adversarially Protected Output

Prompt: "A sks painting of a mountain landscape with a blue sky"
(StyleGuard-protected generation)

Table of contents

1. Environment setup
2. Dataset
3. How to run
4. Contacts
5. Acknowledgement
6. Citation

Environment setup

Install dependencies:

conda env create -f environment.yml
conda activate styleguard

Logging into HuggingFace Hub:

huggingface-cli login

or set the huggingface token in the environment variable:

export huggingface_token=<your huggingface token>

You can save the token in the token.txt file.

Pretrained checkpoints of different Stable Diffusion versions can be downloaded from provided links in the table below:

Version	Link
2.1	stable-diffusion-2-1-base
1.5	stable-diffusion-v1-5
1.4	stable-diffusion-v1-4
stabilityai/stable-diffusion-x4-upscaler	stable-diffusion-x4-upscaler
stabilityai/sd-x2-latent-upscaler	stabilityai/sd-x2-latent-upscaler
SinSR	SinSR_v1.pth

The DiffPure Model can be downloaded from here.Weu use the Guided Diffusion for ImageNet.

Please download the pretrain weights and define "$MODEL_PATH" in the script.

GPU allocation: We use eight 3090 GPUs to train and test.

Dataset

We use the WikiArt dataset for training and testing. Example images are put into data/wikiart.

There are two face datasets: VGGFace2 and CelebA-HQ which are provided at here (from Aiti-dreambooth paper).

For convenient testing, we have provided a split set of one subject in CelebA-HQ at ./data/CelebA-HQ/103 as the Anti-dreambooth does.

How to run

The below script is designed to demonstrate the entire process of using StyleGuard for protecting artistic styles against style mimicry attacks. It includes steps to generate protective noises using StyleGuard, using Noise Upscaler to try to remove the protective noises, and then fine-tuning a model (DreamBooth) on these "denoised" examples. The final step involves generating test images to verify the effectiveness of the defense.

To defense Stable Diffusion version 1.4 (default), you can run

bash scripts/attack_ensemble_aspl_style.sh

It is supposed to take about an hour if run on 8 gpus.

Our dreambooth code is from the diffusers library. Please refer to the diffusers for more details.

Inference: generates examples with prompt

python infer.py \
  --model_path $DREAMBOOTH_OUTPUT_DIR \
  --output_dir $DREAMBOOTH_OUTPUT_DIR/checkpoint-1000-test-infer \
  --prompt "an sks painting including a house"

Evaluation

Suppose the reference folder is $REF_FOLDER, where the images are generated from a SD model finetuned on the clean images.

First, generate FID reference stats

# https://github.com/mseitzer/pytorch-fid
pip install pytorch-fid
export REF_FOLDER=outputs/clean
python -m pytorch_fid --save-stats $REF_FOLDER evaluate/fid_ref.npz

Compute FID and Precision

python evaluate/eval_fid_new.py \
      --input_folder outputs/style/wikiart/vangogh_StyleGuard_style_loss_upscaling/checkpoint-1000-test-infer \
      --refer evaluate/fid_ref.npz \
      --output_folder outputs

CUDA_VISIBLE_DEVICES=-1 python evaluate/eval_precision_new.py \
      --reference_folder outputs/clean \
      --gen_folder outputs/style/wikiart/vangogh_StyleGuard_style_loss_upscaling/checkpoint-1000-test-infer \
      --output_folder outputs

To compute CMMD distance, it is necessary to install scenic.

$ git clone https://github.com/google-research/scenic.git
$ cd scenic
$ pip install .
$ export PYTHONPATH=$PYTHONPATH:/home/.../StyleGuard/scenic

Then compute CMMD distance:

conda run -n cmmd python -m cmmd.main --ref_folder=$REF_FOLDER \
  --gen_folder=$DREAMBOOTH_OUTPUT_DIR/checkpoint-$FINETUNE_STEP-test-infer/an_sks_painting_including_a_house \
  --max_num=12 \
  --batch_size=$NUM_GPUS \
  --output_folder $DREAMBOOTH_OUTPUT_DIR

run baselines (Anti-DreamBooth, SimAC)

bash attack_aspl.sh
bash attack_simac.sh

Test on the Official implementation of Noise Upscaling:

git clone https://github.com/ethz-spylab/robust-style-mimicry.git

python robust-style-mimicry/noise.py --in_dir /home/yjli/AIGC/diffusers/StyleGuard/outputs/style/wikiart/vangogh_StyleGuard_style_loss_upscaling/noise-ckpt/50 --out_dir /home/yjli/AIGC/diffusers/StyleGuard/outputs/style/wikiart/vangogh_StyleGuard_style_loss_upscaling/noise_upscale_official --gaussian_noise 0.05

python robust-style-mimicry/upscale.py --in_dir outputs/style/wikiart/vangogh_StyleGuard_style_loss_upscaling/noise_upscale_official --out_dir outputs/style/wikiart/vangogh_StyleGuard_style_loss_upscaling/noise_upscale_official

We also implement an alternative Noise Upscaling that contains two different upscale models:

python Noisy_Upscaling.py \
  --input_folder="outputs/style/wikiart/$EXPERIMENT_NAME/noise-ckpt/50" \
  --output_folder=$INSTANCE_DIR \
  --upscaler="x2"

The upsclar can be x2 or x4.

Citation

Details of algorithms and experimental results can be found in our following paper:

@inproceedings{li2025styleguard,
  title={StyleGuard: Preventing Text-to-Image-Model-based Style Mimicry Attacks by Style Perturbations},
  author={Li, Yanjie and Zhang, Wenxuan and Lyu, Xinqi and Liu, Yihao and Xiao, Bin},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2025}
}

Acknowledgement

This repo is based on Anti-DB and SimAC. Thanks for their impressive works!

Contacts

If you have any problems, please open an issue in this repository or send an email to yanjie.li@connect.polyu.hk.