Att-Adapter

[ICCV 2025] Att-Adapter: A Robust and Precise Domain-Specific Multi-Attributes T2I Diffusion Adapter via Conditional Variational Autoencoder

Att-Adapter: A Robust and Precise Domain-Specific Multi-Attributes T2I Diffusion Adapter via Conditional Variational Autoencoder
Wonwoong Cho¹, Yan-Ying Chen² Matthew Klenk² David Inouye¹ Yanxia Zhang²,
¹ Elmore Family School of Electrical and Computer Engineering, Purdue University
² Toyota Research Institute

Key Contributions

• Fine-grained, continuous control over multiple attributes at once (e.g., exact eye-openness and car width) on top of any pretrained diffusion model.

• A decoupled cross-attention layer that cleanly fuses text prompts with numeric or categorical attribute inputs, plus a Conditional VAE that can prevent over-fitting and keeps the outputs diverse.

• No paired data needed: it learns straight from unpaired real images in domains like faces (FFHQ) and cars (EVOX), rather than learns from paired synthetic images.

• Scales to 20 + attributes in one model—no extra fine-tuning is required.

News

• 2025-07-22: ⭐️ Training and sampling codes of Att-Adapter with RealisticVision have been released.

Quick Start

Setup

1. Environment setup

conda create -n att-adapter python=3.10
conda activate att-adapter

2. Requirements installation

git clone https://github.com/WonwoongCho/Att-Adapter.git
cd Att-Adapter
pip install -r requirements.txt
conda install -c conda-forge onnxruntime # for inference with ipadapter
conda install -c conda-forge dlib # for inference with ipadapter

Training

Setup

1. Download pretrained models ($pwd should be under Att-Adapter)

mkdir models
cd models
wget -O ip-adapter-faceid_sd15.bin https://huggingface.co/h94/IP-Adapter-FaceID/resolve/main/ip-adapter-faceid_sd15.bin?download=true
cd ..

2. Data

You need FFHQ dataset, preprocessed facial ID embedding, caption and attributes for each image.

• 2-1. FFHQ data

Download FFHQ data and put them under ~/data/ffhq-dataset.

The structure needs to be like ~/data/ffhq-dataset/images1024x1024/00000/xxxxx.png, ~/data/ffhq-dataset/images1024x1024/01000/xxxxx.png, ..., ~/data/ffhq-dataset/images1024x1024/69000/xxxxx.png.

• 2-2. Facial embedding

You need to preprocess ID embeddings, following the instruction.

Use extract_face_embed_all.py if you need. Perhaps additional libraries need to be installed.

At the end of the day, you may have the facial embeddings under ~/data/ffhq-dataset/face_embeddings_padding100_det640_all.

• 2-3. Captions and preprocessed attributes

ChatGPT caption and preproessed attributes

Download the data from the link and put them under ~/data/ffhq-dataset.

The caption file needs to be unzipped by tar -xvzf chatgpt_caption.tar.gz.

Running script

bash train_attadapter_face.sh

Convert checkpoint

python convert_checkpoint.py

You may want to change the iteration in the python file.

Save loss plot

python save_loss_plot.py

Inference

Setup

1. Download pretrained models ($pwd should be under Att-Adapter)

mkdir models # can be skipped if you already have one
cd models
wget -O ip-adapter-faceid_sd15.bin https://huggingface.co/h94/IP-Adapter-FaceID/resolve/main/ip-adapter-faceid_sd15.bin?download=true
wget -O att_adapter_ffhq.bin https://huggingface.co/WonwoongCho/Att-Adapter/resolve/main/att_adapter_ffhq.bin?download=true
wget -O shape_predictor_68_face_landmarks.dat https://huggingface.co/WonwoongCho/Att-Adapter/resolve/main/shape_predictor_68_face_landmarks.dat?download=true # dlib face landmark detector. For inference with ip-adpater.
cd ..

Random face generation

python test_att_adapter.py

• Options
  • --gender: woman or man. Default value is man.
  • --race: Select one of asian, indian, black, white, middleeast, latino. Default value is asian.
  • --prompt: This prompt is used in the fixed template: a {gender} {prompt}. For example, If the gender is woman and the prompt is 'smiling', the formatted prompt in the code is 'a woman smiling'. Default value is 'smiling'.
  • --scale: alpha in the paper. Scale for the Att-adapter. Default value is 0.8.
  • --att_strength: Strength for the attribute interpolation. Recommended range: [2.5, 6.0]. This will be multipled with a std value for visualizing the attribute change. Default value is 5.0.
  • --target_attr: Target attribute to change. Select one of [brightness, yaw, pitch, width_of_mouth, gap_between_eyes, width_of_eyes, height_of_eyes, width_of_nose, height_of_nose, width_of_mouth, height_of_mouth, width_of_face, height_of_face, height_between_nose_mouth, height_between_mouth_chin, width_of_eyebrow, height_between_eyebrow_eye]. Default is brightness.
  • --seed: Random seed. Default is 46.

Face image conditional generation with ipadapter

python test_att_adapter_ip_adapter.py

• Options
  • --image_path: Image path. Default value is ./assets/jennifer.jpg.
  • --gender: woman or man. Default value is woman.
  • --race: Select one of asian, indian, black, white, middleeast, latino. Default value is white.
  • --prompt: This prompt is used in the fixed template: a {gender} {prompt}. For example, If the gender is woman and the prompt is 'in the jungle like tarzan', the formatted prompt in the code is 'a woman in the jungle like tarzan'. Default value is 'in the jungle like tarzan'.
  • --scale: alpha in the paper. Scale for the Att-adapter. Default value is 0.35.
  • --att_strength: Strength for the attribute interpolation. Recommended range: [2.5, 6.0]. This will be multipled with a std value for visualizing the attribute change. Default value is 4.0.
  • --target_attr: Target attribute to change. Select one of [brightness, yaw, pitch, width_of_mouth, gap_between_eyes, width_of_eyes, height_of_eyes, width_of_nose, height_of_nose, width_of_mouth, height_of_mouth, width_of_face, height_of_face, height_between_nose_mouth, height_between_mouth_chin, width_of_eyebrow, height_between_eyebrow_eye]. Default is yaw.
  • --seed: Random seed. Default is 46.

Pretrained Models

Model	Base model	Description	Resolution
Att-Adapter	Att-Adapter ffhq	The model used in the paper (200MB).	(512, 512)

Citation

@inproceedings{attadapter2025,
  title={Att-Adapter: A Robust and Precise Domain-Specific Multi-Attributes T2I Diffusion Adapter via Conditional Variational Autoencoder},
  author={Wonwoong Cho and Yan-ying Chen and Matt Klenk and David Inouye and Yanxia Zhang},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2025},
  note={To appear}
}