Jiaxiang Cheng, Pan Xie*, Xin Xia, Jiashi Li, Jie Wu, Yuxi Ren, Huixia Li, Xuefeng Xiao, Min Zheng, Lean Fu (*Corresponding author)
AutoML, ByteDance Inc.
We propose ResAdapter, a plug-and-play resolution adapter for enabling diffusion models of arbitrary style domains to generate resolution-free images: no additional training, no additional inference and no style transfer.
The above images are generated by ResAdapter with dreamlike-diffusion-1.0
[2024/03/12]We support gradio demo inreplicate.com(by @chenxiwh)[2024/03/12]We release the resadapter-sdv15 that supports128~1024resolution.[2024/03/05]We release the paper about resadapter to arxiv.[2024/03/04]We release the inference codes and resadapter-sdv15-i that supports128~512, and resadapter-xl-i that supports256~1024.
🔥 Standalone Example with SDXL-Lightning
# pip install diffusers, transformers, accelerate, safetensors, huggingface_hub
import torch
from diffusers import StableDiffusionXLPipeline, UNet2DConditionModel, EulerDiscreteScheduler
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from torchvision.utils import save_image
generator = torch.manual_seed(12638721376)
width, height = 512, 512
base = "stabilityai/stable-diffusion-xl-base-1.0"
repo = "ByteDance/SDXL-Lightning"
ckpt = "sdxl_lightning_4step_unet.safetensors" # Use the correct ckpt for your step setting!
# Load SDXL-Lighting.
unet = UNet2DConditionModel.from_config(base, subfolder="unet").to("cuda", torch.float16)
unet.load_state_dict(load_file(hf_hub_download(repo, ckpt), device="cuda"))
pipe = StableDiffusionXLPipeline.from_pretrained(base, unet=unet, torch_dtype=torch.float16, variant="fp16").to("cuda")
pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing")
# Inference SDXL-Lighting
images = pipe(
"A girl smiling",
width=width,
height=height,
num_inference_steps=4,
guidance_scale=0,
num_images_per_prompt=5,
output_type="pt",
generator=generator,
).images
save_image(images, f"/baseline_{width}.png", normalize=True, padding=0)
# Load ResAdapter
pipe.load_lora_weights(
hf_hub_download(
repo_id="jiaxiangc/res-adapter",
subfolder="sdxl-i",
filename="resolution_lora.safetensors",
),
adapter_name="res_adapter",
)
pipe.set_adapters(["res_adapter"], adapter_weights=[1.0])
# Inference ResAdapter
images = pipe(
"A girl smiling",
width=width,
height=height,
num_inference_steps=4,
guidance_scale=0,
num_images_per_prompt=5,
output_type="pt",
generator=generator,
).images
save_image(images, f"resadapter_{width}.png", normalize=True, padding=0)🔥 [Update at 2024/03/12] We are happy to provide demos in replicate.com.
Relicate demo: bytedance/res-adapter, by (@chenxiwh)
We will support ComfyUI within a week.
# Step1: Enter to res-adapter directory
cd res-adapter
# Step2: Install dependency
pip install -r requirements.txt
# Step3: Download diffusion models, and make the directory structure as follows:
models
├── res_adapter
│ ├── res_adapter-v1.5
│ ├── res_adapter-xl
├── diffusion_models
│ ├── ...
├── controlnet
│ ├── ...
├── ip_adapter
│ ├── ...
└── lcm-lora
└── ...🎉 [Update at 2024/03/12] We release resadapter-sd1.5 that supports 128~1024.
| Models | Parameters | Resolution Range | Links |
|---|---|---|---|
| resadapter-v1.5-i | 0.8M | 128<= x <= 512 | Download |
| resadapter-xl-i | 0.4M | 256 <= x <= 1024 | Download |
| resadapter-v1.5 | 0.9M | 128 <= x <= 1024 | Download |
| resadapter-xl | 0.5M | 256 <= x <= 1536 | Coming soon |
We provide some personalized models for sampling style images with ResAdapter. More personalized models can be found in CivitAI.
| Models | Structure Type | Domain Type | Links |
|---|---|---|---|
| Base model | |||
| SDv1.5 | - | General | Download |
| SDXL1.0 | - | General | Download |
| Personalized model | |||
| RealisticVision | SDv1.5 | Realism | Download |
| Dreamlike | SDv1.5 | Fantasy | Download |
| DreamshaperXL | SDXL | 2.5D | Download |
| AnimeartXL | SDXL | Anime | Download |
We support demos about ControlNet, IP-Adapter, LCM-LoRA.
| Modules | Name | Type | Links |
|---|---|---|---|
| ControlNet | lllyasviel/sd-controlnet-canny | SD1.5 | Download |
| ControlNet | diffusers/controlnet-canny-sdxl-1.0 | SDXL | Download |
| IP-Adapter | h94/IP-Adapter | SD1.5/SDXL | Download |
| LCM-LoRA | latent-consistency/lcm-lora-sdv1-5 | SD1.5 | Download |
| LCM-LoRA | latent-consistency/lcm-lora-sdxl | SDXL | Download |
We provide simple scripts for sampling images of resadapter and baseline.
# Step1: Choose a task example of config file.
# Step2: Run the following script.
python main.py --config /path/to/file😄 For better image generation, we provide two advice:
- For text to image tasks, please use personalized models instead of base models.
- For other tasks, use base models.
Prompt: (masterpiece), (extremely intricate), (realistic), portrait of a girl, the most beautiful in the world, (medieval armor), metal reflections, upper body, outdoors, intense sunlight, far away castle, professional photograph of a stunning woman detailed, sharp focus, dramatic, award winning, cinematic lighting, octane render unreal engine, volumetrics dtx, (film grain, blurry background, blurry foreground, bokeh, depth of field, sunset, motion blur), chainmail.
Prompt: A bird.
Prompt: A man.
Prompt: portrait, action pose, slow motion, (old male human wizard) old male human wizard wearing yellow and black robes (majestic evoker cloth armor), (wrinkles, steampunk), (archmage robes, runic patterns), (insanely detailed, bloom), (analog), (high sharpness), (detailed pupils), (painting), (digital painting), detailed face and eyes, Masterpiece, best quality, (highly detailed photo), 8k, photorealistic, very long straight white and grey hair, grey streaks, ecstatic, (60-year old Austrian male), sharp, (older body), stocky, realistic, real shadow 3d, (highest quality), (concept art, 4k), (wizard labratory in backgound), by Michelangelo and Alessandro Casagrande and Greg Rutkowski and Sally Mann and jeremy mann and sandra chevrier and maciej kuciara, inspired by (arnold schwarzenegger) and (Dolph Lundgren) and (Albert Einstien)
- Provide weights of resadapter-sdv15-i that supports
128~512. - Provide weights of resadapter-sdv15-i that supports
128~1024. - Provide weights of resadapter-xl-i that supports
256~1024. - Provide weights of resadapter-xl-i that supports
256~1536. - Supporting resadapter in ComfyUI during mid to late March.
Our target about the number of star: 1k+. We will keep maintain our repository.
@article{cheng2024resadapter,
title={ResAdapter: Domain Consistent Resolution Adapter for Diffusion Models},
author={Cheng, Jiaxiang and Xie, Pan and Xia, Xin and Li, Jiashi and Wu, Jie and Ren, Yuxi and Li, Huixia and Xiao, Xuefeng and Zheng, Min and Fu, Lean},
booktitle={arXiv preprint arxiv:2403.02084},
year={2024}
}