controlnet_hinter: ControlNet control image preprocess library

What is this?

ControlNet by @lllyasviel is a neural network structure to control diffusion models by adding extra conditions. This library was created to assist 🤗Diffusers when building ControlNet models with Diffusers. It is intended to be easy to use with Diffusers’ StableDiffusionControlNetPipeline (under development).

Usage Example

Image preprocess only

import controlnet_hinter
control_image = controlnet_hinter.hint_canny(some_pil_image)

Full example with Diffusers: Canny Edge Detection

import controlnet_hinter
from diffusers import StableDiffusionControlNetPipeline
from diffusers.utils import load_image

# load some image in PIL.Image format
original_image = load_image("https://huggingface.co/datasets/diffusers/test-arrays/resolve/main/stable_diffusion_imgvar/input_image_vermeer.png")

# this one line to add
control_image = controlnet_hinter.hint_canny(original_image)

# prepare pipeline
pipe = StableDiffusionControlNetPipeline.from_pretrained("takuma104/control_sd15_canny", torch_dtype=torch.float16).to("cuda")
pipe.enable_xformers_memory_efficient_attention()

# generate image
image = pipe(prompt="best quality, extremely detailed", 
             negative_prompt="lowres, bad anatomy, worst quality, low quality",
             image=control_image).images[0]
image.save('generated.png')

To install

Relatively stable version:

pip install controlnet_hinter

Bleeding-edge version:

pip install git+https://github.com/takuma104/controlnet_hinter

API

Currently, supports the following:

API	Description	Model(Checkpoint) to be applied
hint_canny()	canny edge detection	takuma104/control_sd15_canny
hint_depth()	Midas depth estimation	takuma104/control_sd15_depth
hint_hed()	HED edge detection (soft edge)	takuma104/control_sd15_hed
hint_hough()	M-LSD straight line detection	takuma104/control_sd15_mlsd
hint_normal()	Normal map estimation	takuma104/control_sd15_normal
hint_openpose()	Human pose estimation using OpenPose	takuma104/control_sd15_openpose
hint_scribble()	Conversion from user scribble. White backgorund	takuma104/control_sd15_scribble
hint_fake_scribble()	Synthesize scribbles from input images	takuma104/control_sd15_scribble
hint_segmentation()	Semantic segmentation estimation	takuma104/control_sd15_seg