SmartControl: Enhancing ControlNet for Handling Rough Visual Conditions

---

1. Abstract

For handling the disagreements between the text prompts and rough visual conditions, we propose a novel text-to-image generation method dubbed SmartControl, which is designed to align well with the text prompts while adaptively keeping useful information from the visual conditions. Specifically, we introduce a control scale predictor to identify conflict regions between the text prompt and visual condition and predict spatial adaptive scale based on the degree of conflict. The predicted control scale is employed to adaptively integrate the information from rough conditions and text prompts to achieve the flexible generation.

Release

• [2024/3/31] 🔥 We release the code and models for depth condition.

Installation

pip install -r requirements.txt
# please install diffusers==0.25.1 to align with our forward

Download Models

you can download our control scale predictor models from here. To run the demo, you should also download the following models:

• stabilityai/sd-vae-ft-mse
• SG161222/Realistic_Vision_V5.1_noVAE
• ControlNet models
• realisticvision-negative-embedding

How to Use

• If you are interested in SmartControl, you can refer to smartcontrol_demo

• For integration our SmartControl to IP-Adapter, please download the IP-Adapter models and refer to smartcontrol_ipadapter_demo

  # download IP-Adapter models
  cd SmartControl
  git lfs install
  git clone https://huggingface.co/h94/IP-Adapter
  mv IP-Adapter/models models