Control Net Tile model

[ryanto]

Hello! I'm wondering if it's possible to use the Controlnet tile model, lllyasviel/control_v11f1e_sd15_tile, with Swift?

From the huggingface example it looks like the controlnet_conditioning_image is size (1024x1024), but when I try to run the swift code with an image larger than the expected 512x512 from SD 1.5 I get an error:

> swift run StableDiffusionSample "high quality photo of a dog on the beach" \
  --negative-prompt "blur, lowres, bad anatomy, bad hands, cropped, worst quality" \
  --resource-path ./mlpackages/512x512/Resources \
  --controlnet LllyasvielControlV11F1ESd15Tile \
  --controlnet-inputs ./inputs/dog-1024.png \
  --output-path ./images \
  --seed 100 \
  --step-count 20 \
  --image-count 1 \
  --scheduler dpmpp \
  --compute-units cpuAndGPU \
  --strength 0.5
  
Building for debugging...
Build complete! (0.13s)
Loading resources and creating pipeline
(Note: This can take a while the first time using these resources)
Sampling ...
Error: Failed to obtain prediction for sample 0

I'm unsure how to go about getting the larger conditioning image into the pipeline.

@jrittvo I saw you have a bunch of converted controlnet models on hugging face. Did you have any luck getting the tile model working?

[jrittvo]

Not sure where the 1024x1024 is coming from. With a 512x512 base model and the Tile CN model also converted for 512x512, which would be the default size if no other size is specified, you’d need to use a 512x512 starting image. I have used the Tile CN model I converted that way and at the other sizes I made. As long as all 3 pieces are the same size, it should work.

[jrittvo]

Maybe the HF example you were looking at is for an SDXL model. There are now some ControlNet models for use with SDXL and the SDXL default size is 1024x1024.

[ryanto]

Thanks for the reply @jrittvo! Really appreciate you taking the time to answer.

I believe one of the points of the Tile CN model is to be able to upscale, so you can take an image that is 64x64, resize it to 1024x1024, feed it into the pipeline, and get back a 1024x1024 detailed image. This is what the example on the lllyasviel/control_v11f1e_sd15_tile repo is doing.

He's using SD 1.5 in the example.

It'd be nice to know how a to convert the example into Swift code.

[jrittvo]

I don't think Core ML models have this kind of flexibility built into their architecture, to be capable of working with mixed sizes and/or adapt to a particular size on-the-fly. At least not yet. This kind of thing is highly requested, but I suspect it requires a ground up reworking of the Swift Core ML pipelines and tools.

The Tile CN in Swift gives you subtle variations of the input file, and that's all. No upscaling, no picking a small area from a large image to work with, etc.

Maybe someone else can add some info on the status of more flexible Core ML model developments.

[jrittvo]

Have you tried feeding it a smaller image than the model size? In the example, the input image looks to be a 64x64 and the script upsizes it before feeding it to the Tile ControlNet. Maybe the Swift implementation is capable of doing the upsizing internally? I doubt it, but it is quick to try.

The error message: Error: Failed to obtain prediction for sample 0 means it was expecting one size for the input image but it found a different size. So you'll know right away if the smaller size is also "illegal".

[ryanto]

Yup- it doesn't work with smaller images either. Image needs to be 512x512.