Outpainting with Diffusers

[asomoza]

Hi, I have made a 3 part series about outpainting with Diffusers, each one with a different method.

Outpainting I - Controlnet version

Outpainting using a controlnet trained for this. This method does change the original image but has the advantage that the ourpaining its more creative and seamless.

Outpainting II - Differential Diffusion

This method uses differential diffusion which allows us to use a soft inpainting mask that generates seamless outpainting, as the first method I can change the original image but not as much.

Outpainting III - Inpaint Model

The final method uses an inpainting model but it will preserve the original image so it's more suitable for professional work.

[tolgacangoz]

[asomoza]

fixed, thanks.

[crapthings]

control_image = Image.open(BytesIO(response.content))
new_controlnet_image = Image.new("RGBA", control_image.size, "WHITE")
new_controlnet_image.alpha_composite(control_image)

why this dones't work?

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[11], line 17
     14 b = midas(input_image)
     16 new_controlnet_image = Image.new('RGB', input_image.size, 'WHITE')
---> 17 new_controlnet_image.alpha_composite(ccc)
     18 # new_controlnet_image = Image.composite(input_image, new_controlnet_image, input_image)
     20 display(new_controlnet_image)

File [~/.local/lib/python3.10/site-packages/PIL/Image.py:1790](http://192.168.50.251:8888/lab/tree/workspace/ai-pipe/ai-pipe-replace-model-new/.local/lib/python3.10/site-packages/PIL/Image.py#line=1789), in Image.alpha_composite(self, im, dest, source)
   1787 else:
   1788     background = self.crop(box)
-> 1790 result = alpha_composite(background, overlay)
   1791 self.paste(result, box)

File [~/.local/lib/python3.10/site-packages/PIL/Image.py:3328](http://192.168.50.251:8888/lab/tree/workspace/ai-pipe/ai-pipe-replace-model-new/.local/lib/python3.10/site-packages/PIL/Image.py#line=3327), in alpha_composite(im1, im2)
   3326 im1.load()
   3327 im2.load()
-> 3328 return im1._new(core.alpha_composite(im1.im, im2.im))

ValueError: image has wrong mode

[asomoza]

When you're doing alpha composite you need both images to have an alpha channel. Probably the image you're loading as a control_image is a RGB image instead of a RGBA

[juzstu]

I find that the BrushNet also performs well in outpainting.

[asomoza]

Thanks, I have it under my radar, I'm waiting for the SDXL version but it seems that's going to take more time.

[juzstu]

@asomoza For your Outpainting III - Inpaint Model, I use lora to finetune the RealVisXL_V4.0, but the image quality was really bad, if not use the lora weights, the image quality turn to normal, can you tell me what case this and how can I use lora weights correctly to generate a normal quality image?

[asomoza]

That is a step by step guide, I provide a full script at the end but it's just so people can learn and follow it.

If you're just going to use it and change something, you should divide it into two parts, one for generating the temp background and the second part to actually generate the background with an inpainting model. Also you'll need to remove the fixed seeds from the code.

I'll need a little more information on what you're trying to do, are you loading the LoRA in the first step? the second? both? For what are you using the LoRA? the LoRA works ok if you just use it with a normal generation?

For example, if I just want to change the style of the generated background, I can just load the pixel art LoRA on the second step, and change the prompt for that one with the trigger for the LoRA.

source image	bg removed

temp background	final background	pixel art backgound

Hope the images explain the process a little better, but for me to be able to help you, I'll need more information on what you're trying to do.

[juzstu]

Thanks for your replay. I use the my own data to lora finetune the RealVisXL_V4.0 and use it in the first step, if I drop the zoe-depth model from the controlnets, the image will be normal. As you mentioned that the inpaint model likes to add details, but if I use zoe-depth or canny/depth mdel, the background only has pure color and no or just little details.

[asomoza]

In my last example I changed the source image, the prompts, the seeds and added a LoRA and it worked just fine, the controlnet works as intended even if you change those.

So the problem lies in one or a couple of what I mentioned, I'm guessing you're not sharing those because you can't, so I really can't help you anymore with the information you can give me.

My last suggestion is that you can try different settings for the depth controlnet, try lowering the controlnet_conditioning_scale or the control_guidance_end, maybe your LoRA needs more freedom to work.

[juzstu]

Thank you for your suggestion, I will give it a try.

[peki12345]

@asomoza I have carefully read these three plans and have been greatly inspired. Thank you very much.
However, I have to mention an important issue: when I only have text2img model, some lora or controlnet, and can't accept changes in no-mask areas, I can only use plan 2.
But after trying plan 2, I found that it could hardly complete the structure of the human body (object), let alone the face. Moreover, when the outpainting area is too large, the quality of generated content is low, and there may be problems connecting with no-mask areas.

For example, I used Plan 2 to double the repainting of this image, but the results I obtained were not very satisfactory.

But the result of Fooocus is always good

[madstuntman11]

Still have some transition artifacts, but it works for my task. Maybe the next step is to add ip-adapter with style block active.

[peki12345]

You're comparing big images with really small ones, if you want to achieve the same results as fooocus, you can scale down the image so SDXL can generate the background in one pass, that's what fooocus is doing. If you do that, you can use the third method with the inpaint model.

@peki12345 Tried it. Works good for my case (not photo realistic, but artistic). Just do not pad with simple white or black, fill with something reasonable. Model from the 3rd example will keep it white of black it this case. For example, use random edge tiles from original image.

This sounds reasonable, I will try it. But I don't know exactly how to do it yet. Can you provide a simple code?

[peki12345]

I was going to start working on this, but what always impresses me is how people don’t look closely at the fooocus results and think it’s really good when it isn’t.

Now that I look closer to your image, I can clearly see the seams and a lot of defects, changes in skin tones, patterns, it really makes the image not usable in any kind of serious work.

on the other side, the ones you generated with diffusers don't have the seams or the clear defects, it's a smooth transition but it doesn't do the person generation. Did you try with a prompt?

I will do the guide because I really think it's outdated even if it's a couple of months old, but there was a lot of improvements in this field. Still I think trying to achieve fooocus results is no longer a high standard, I think I can get better results with just inpainting in ComfyUI right now.

prompt

Yes, I must admit that fooocus has many defects. However, the seam appears abrupt because I directly pasted the original image without merge.

Regarding the issue of prompt, both fooocus and diffusers used a simple prompt that is independent of the image content. I believe a more detailed prompt will enhance the effect, but I want to use one prompt to infer all the images. I don't intend to create an interactive interface like fooocus, which is why I chose diffusers.

[asomoza]

but I want to use one prompt to infer all the images

I can tell you with that it will fail with all the complex images if you want to use a simple prompt for all the images. What you can do is to use a VLLM to generate the prompt without user intervention, but those 3 methods I posted will need a good prompt for complex images.

I'll try to achieve what you want in the new version but I can't be sure it will work, I have some ideas on how to do it.

Also, with this:

I directly pasted the original image without merge

It's been a while since I used fooocus, you mean inside fooocus or using a external software? I remember that fooocus didn't have that many options, you use the image and tell it to expand it or use a mask on a smaller image, all of my tests that I did with those options produced horrible results (similar to the images you posted) so I just assumed I was just bad at using it.

[peki12345]

It's been a while since I used fooocus, you mean inside fooocus or using a external software? I remember that fooocus didn't have that many options, you use the image and tell it to expand it or use a mask on a smaller image, all of my tests that I did with those options produced horrible results (similar to the images you posted) so I just assumed I was just bad at using it.

Using VLLM is indeed a good idea, but I need to check if my GPU memory is sufficient.
fooocus is only the main part, there are also some pre-processing and post-processing. In pre-processing, I used some tricks to make the obtained results more stable. If you use the original fooocus, there is a certain probability that you will receive horrible results, which is not your problem.

[xduzhangjiayu]

Hey, is there any training-free method for out-painting?

[asomoza]

Hi, I did a guide for image fill, I commented at the end that you can use the same technique for out painting.

fffiloni which is very talentend and does some of the best spaces, took this knowledge and the space I made for it and implemented outpainting, you can check it here.

Even if this is not what I consider the final and the best I can do, a lot of people commented that this is the best outpainting they've seen and it's really fast. That space has a lot of cool additions too.

Also what do you mean by training-free? Even if the ones here are old and outdated, they were training-free, just inference with pretrained models.

[xduzhangjiayu]

Hi, I did a guide for image fill, I commented at the end that you can use the same technique for out painting.

fffiloni which is very talentend and does some of the best spaces, took this knowledge and the space I made for it and implemented outpainting, you can check it here.

Even if this is not what I consider the final and the best I can do, a lot of people commented that this is the best outpainting they've seen and it's really fast. That space has a lot of cool additions too.

Also what do you mean by training-free? Even if the ones here are old and outdated, they were training-free, just inference with pretrained models.

Thanks for the reply!
Training-free is just use the base model (SDXL or SD3), not include ControlNet. Doesn't seem to have a this approach right now?

[asomoza]

The only way of doing what you're suggesting is with inpainting or differential diffusion, that approach always existed but it it's not good, as far as I know there is no good outpainting with just the base model.

Have you seen somewhere else a good outpainting with just the base model? if it exists, it would be easy to just port it to diffusers.

[xduzhangjiayu]

The only way of doing what you're suggesting is with inpainting or differential diffusion, that approach always existed but it it's not good, as far as I know there is no good outpainting with just the base model.

Have you seen somewhere else a good outpainting with just the base model? if it exists, it would be easy to just port it to diffusers.

Thanks!
I have tried sd3 inpainting using diffusers pipeline and sd3 base model, It works well for inpainting a small area, but if I Mask a large area, it doesn't perform as well.

[atharvasadanshive]

Code where the Error is detcted --->
zoe = ZoeDetector.from_pretrained("lllyasviel/Annotators")
image_zoe = zoe(white_bg_image, detect_resolution=512, image_resolution=1024)

Error --->>

TypeError: expected size to be one of int or Tuple[int] or Tuple[int, int] or Tuple[int, int, int], but got size with types [<class 'numpy.int64'>, <class 'numpy.int64'>]

Can someone please help me understand why am I am getting error in the zoe detector line of code.

[asomoza]

Hi, that's probably because of the image you're using but I'm more intrigued on why are you using this discussion to post a controlnet preprocessor issue you have, you probably won't get that much help here since this is an outdated discussion about outpainting.

[fozantalat]

Hi,

How can I use Outpainting using inpainting to change the swap the image of background with another image that I have, I have car image with background and I want to replace it with another one.

[fozantalat]

@asomoza Hi Alvaro, Did you got some time for sharing the code ?

[asomoza]

Hi, not yet, probably not in less than a week since I'm doing other things and if I'm going to do this, it would be better to do the guide at the same time, like I did for the image fill one, so it takes time.

Also for it to be fully automatized I need to add the remove background functionality to the image_gen_aux library too.

[fozantalat]

Ok , thanks for your support Alvaro, I can wait a Week,

[fozantalat]

One more thing , can we add shadows , so that it looks more natural ?

[asomoza]

the second image has more shadows, you want more?

The answer is yes, I often do it but it requires human interaction, you have to manually add some basic drawings for the shadows with a low denoise, this means you will need a basic understanding on how lighting works and where to add them.

Mostly you don't have limits on what you can do but most people don't know or don't have the "basic artist" level to do it, you can change the lighting (day or night), add lights, reflections, shadows, coloring, film noise, etc.

I don't bother teaching that because people seem to like a lot more the "prompt understanding", text generation or something like the "IC Light" model, all of them are very basic and easy to do if you worked with images before AI and AI makes it even easier. Sadly this process can't be done with just code, at least for now.

These are some example that took me a couple of minutes to do, and can be done a lot better if I dedicate more time to them:

P.S.: I still think that normal people will have a hard time telling if the second one from before its a composition done with AI or a real image though.

[fozantalat]

Hi Alvaro,

By shadows I mean the shadows that car cast on ground just like the below image

[asomoza]

yeah I know, they are subtle but you can see the shadow and how much it's the difference between them. The model is realistic it won't add an amount of shadow that doesn't correspond to the scene lighting, you can still override it and add a lot of amount but that would also look fake, no shadows or excessive amount of it will make the final image fake.

original	0.2	0.5	more shadow

The original scene has a different time of the day with more sun, hence the more pronounced shadow, the background you provided is more in the afternoon with clouds, less light then less shadow. You can of course override all this but essentially you're making the image more fake.

The second one looks fake because as you requested, I preserved the background intact, for the others ones let it go a little so the model can work to make it more coherent.

Maybe you can spin a test and ask people if the last two image are composited or no, with this I mean normal people, people that work with AI tend to think everything is fake nowadays, I've seen people thinking that images are made with AI just because the hands are a little blurry or weird which can happen in real photos too.

If you ask me, the problem with the lasts two images is not the shadow or the background but with the car reflections, since you also asked me to preserve the car intact, the reflections look out of place with the new background, but to fix that, you have to let go the car too.

The only way to make both work flawlessly is to make a lora with the car, in my experience, that's the only way to get a really good image with everything making sense.

But as I answered before to your question, yes, you can add any amount of shadow you want, but with the caveat that it has to be done manually and you will override what the model think it's correct.

[fozantalat]

Thank you so much Alvaro, it makes sense now, Please do add a section in your guide/blog how can we use lora, to make everyhting flawless.

[asomoza]

oh sadly, training a lora it's a whole different thing and a lot more complex, I want to do a full guide on it but I can't seem to find something that I can use as an example for it, with this I mean, something real like the car you provided but that has no copyright, usage restrictions and that is ethical to use.

To explain how to train a lora I need a quality dataset that I can make public, which unless is something fictional or unrealistic, can't be done right now because of the previous issues.

[fozantalat]

Ok, no problem, please do share any resources on it if you find one, looking forward for your next blog post explaining the background replacement you did.
Thank you.

[fozantalat]

HI Alvaro,
Is there a way we can make the replaced background to be visible from the rear car window .

[asomoza]

the easiest way is to mask that part also for inpainting, but the models that remove background won't do that, so right now, the only way to achieve that is to manually mask that part.

Also you can lower the conditioning scale for the inpainting controlnet but that will also make the car less similar to the original one.

[Fozan-Talat]

Alright, Thanks Alvaro .

[fozantalat]

Hi Alvaro,

I was able to code the steps that you provided and here are the result, but its not close to yours

and here is the code, can you please have a look it at and let me know what should I change so that I'm able to replicate it just like the image you provided.I do have the mask already which I got from Birefnet model.

Thank you.

Code:

import os, sys
from torchvision import transforms
from PIL import Image
import torch
import numpy as np
import matplotlib.pyplot as plt

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

— Inputs —

car_img = Image.open("car_images/38N129953_8.jpg").convert("RGB")
mask_img = Image.open("car_images/38N129953_8_mask.png")
bg_img = Image.open("car_images/bacdrop.jpg").convert("RGB")

— Step 2: Segment & Composite (unchanged) —

mask_l = mask_img.convert("L")
segmented_car = car_img.copy()
segmented_car.putalpha(mask_l)
bg_rgba = bg_img.convert("RGBA")
car_resized = segmented_car.resize(bg_rgba.size)
composite = Image.alpha_composite(bg_rgba, car_resized)

— Step 4: Control Image via image_gen_aux (unchanged) —

from image_gen_aux import TeedPreprocessor, LineArtStandardPreprocessor

teed_p = TeedPreprocessor.from_pretrained("OzzyGT/teed").to(device)
lineart_p = LineArtStandardPreprocessor()
teed_edges = teed_p(composite.convert("RGB"))[0]
lineart_edges = lineart_p(composite.convert("RGB"))[0]

control_img = Image.blend(
teed_edges.convert("RGBA"),
lineart_edges.convert("RGBA"),
alpha=0.5
)

TARGET = (1024, 1024)
composite = composite.resize(TARGET, Image.LANCZOS)
control_img = control_img.resize(TARGET, Image.LANCZOS)
mask_l = mask_l.resize(TARGET, Image.NEAREST)

— Helper: PIL → float16 tensor on CUDA —

def pil_to_tensor(img: Image.Image, is_mask: bool=False):
arr = np.array(img.convert("RGB" if not is_mask else "L"), dtype=np.float32) / 255.0
t = torch.from_numpy(arr)
if not is_mask:
t = t.permute(2, 0, 1) # HWC → CHW
t = t.unsqueeze(0) # add batch dim
return t.to(device, dtype=torch.float16)

Prepare tensors

composite_t = pil_to_tensor(composite, is_mask=False)
control_t = pil_to_tensor(control_img, is_mask=False)
mask_t = pil_to_tensor(mask_l, is_mask=True)

— Step 5: ControlNet-Union Inpainting —

from diffusers.models.controlnets.controlnet_union import ControlNetUnionModel
from diffusers import StableDiffusionXLControlNetUnionInpaintPipeline, AutoencoderKL, UniPCMultistepScheduler

controlnet_union = ControlNetUnionModel.from_pretrained(
"xinsir/controlnet-union-sdxl-1.0",
torch_dtype=torch.float16,
use_safetensors=True
)
vae = AutoencoderKL.from_pretrained(
"madebyollin/sdxl-vae-fp16-fix",
torch_dtype=torch.float16
).to(device)

pipe_inpaint = StableDiffusionXLControlNetUnionInpaintPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
controlnet=controlnet_union,
vae=vae,
torch_dtype=torch.float16
).to(device)

Faster scheduler + memory optimizations

pipe_inpaint.scheduler = UniPCMultistepScheduler.from_config(pipe_inpaint.scheduler.config)
pipe_inpaint.enable_attention_slicing()
pipe_inpaint.enable_xformers_memory_efficient_attention()

Inpaint

result_inpaint = pipe_inpaint(
prompt="",
image=composite_t,
control_image=control_t,
control_mode=[0],
controlnet_conditioning_scale=1.0,
mask_image=mask_t,
num_inference_steps=50,
strength=1.0,
).images[0]

— Step 6: Juggernaut Img2Img Refinement —

from diffusers import StableDiffusionXLImg2ImgPipeline
from huggingface_hub import hf_hub_download

model_file = hf_hub_download(
"RunDiffusion/Juggernaut-XL-v9",
"Juggernaut-XL_v9_RunDiffusionPhoto_v2.safetensors"
)

pipe_juggernaut = StableDiffusionXLImg2ImgPipeline.from_single_file(
model_file,
torch_dtype=torch.float16
).to(device)
pipe_juggernaut.vae = vae

Convert result_inpaint (PIL) → tensor

inpaint_pil = result_inpaint.convert("RGB")
inpaint_t = pil_to_tensor(inpaint_pil, is_mask=False)

res1 = pipe_juggernaut(
prompt="",
image=inpaint_t,
strength=0.2,
num_inference_steps=30
).images[0]

res2 = pipe_juggernaut(
prompt="",
image=inpaint_t,
strength=0.5,
num_inference_steps=30
).images[0]

— Step 7 & 8: Lighting Overlay & Display —

overlay = Image.new("RGBA", composite.size, (255,255,230,50))
final1 = Image.alpha_composite(res1.convert("RGBA"), overlay)
final2 = Image.alpha_composite(res2.convert("RGBA"), overlay)

titles = ["Segmented Car", "Composite", "Control", "Inpainted", "Jug (0.2)"]
imgs = [segmented_car, composite, control_img, result_inpaint, final1]

plt.figure(figsize=(12, 6))
for i, img in enumerate(imgs):
plt.subplot(1, 5, i + 1) # 1 row, 5 columns
plt.imshow(img)
plt.title(titles[i])
plt.axis("off")

plt.tight_layout()
plt.savefig("comparison_figure.png", dpi=300, bbox_inches='tight')
plt.show()

[asomoza]

@fozantalat you didn't have any issues while using DMD2 with a CFG of 1.0 with the pipeline? I don't use the Controlnet Union Pipelines because I implemented them myself before they were part of diffusers as you can see in my guides, but it seems they have some problems now that I'm testing them.

Probably the best way to tackle this is to make a custom pipeline again and then I'll try to fix the ones in diffusers but I'm getting a lot worse results than with my own, even after fixing the CFG issue.

[fozantalat]

@asomoza I didn't used the DMD2 , but I did used controlnet union pipelines , but couldn't replicate the results that you shared , I dont know how to build a img2img pipeline with double controlnet union, as you mentioned after reviewing my code, also couldn't find any example where someone did img2img with inpainting, I hope you get this issue resolved and complete the code and guide .

[Fozan-Talat]

@asomoza How much time will it it take to complete the code and guide?

[asomoza]

@Fozan-Talat sadly my priority is not this right now, I'll try to do it as fast as I can but I can't promise any dates yet, one thing is to just do the script but a completely different one is to fix a bug that I still don't know where it is.

Also, it works with modular diffusers so I'm still thinking if I it's better to do it with a modular pipeline, but for that, we need to merge the PR first, but it will also have the added bonus that I can also share an UI workflow with it that people can just use.

I'll test this with kontext too, maybe that's a solution for you now since they released the local model.

[Fozan-Talat]

@asomoza Alright Alvaro, If it works with modular diffuser then plz do it with it, as I'm waiting for it , so that I can build a POC around it.

[fozantalat]

Also after refering to your image fill guide , its kind of reverse of image fill e.g if you look at this image which is in the image fill guide

we are trying to do the opposite, so I believe your upcoming guide will complement it as well.