Feature Request: IPAdapter and Latent Upscale (Neural Network)

[vcm07]

Ipadapter is a better and more robust Revision with more control. It can work as a real-time 1 image LoRa. People are doing revision + ipadapter and other combinations using Comfy. It would be really nice to have this in MRE.

ComfyUI Custom Node: https://github.com/laksjdjf/IPAdapter-ComfyUI

Link to the model: https://huggingface.co/h94/IP-Adapter/tree/main/sdxl_models

About IPAdapter: https://github.com/tencent-ailab/IP-Adapter

===

Upscale

SAG on Fooocus helps a lot, but MRE would need an effective upscale strategy to be even better. Some ideas:

The attempt by city96: https://github.com/city96/SD-Latent-Upscaler

By WAS/Ttl: https://github.com/Ttl/ComfyUi_NNLatentUpscale

These are some suggestions to create an upscaler for MRE; it would make this project even better.

Thank you, MoonRide, for this excellent project. You made Foocus much better, and thank you for the Colab edition. It's good for us folks that don't have a proper computer to run SDXL, and I'm using it every day.

[Ptbird1]

Adding some form of image (re)mixing, iterations and upscaling would be a HUGE boon, one of my favorite features of MJ is the image remixing and iterations, give it an image you like and ask it to make similar ones or mix someone with a blobfish for hilarious results. Upscaling is just the icing on the cake. Would be very interested to see if this can be implemented.

[MoonRide303]

@Ptbird1 Simplest way to create variations of existing image(s) are img2img and Revision modes. If you want output images to look more like original you can just lower denoise in Image-2-Image settings. In latest version you can also use Canny Edge Control-LoRA (works as ControlNet - see CN tab), which is also great way to produce variants of original image :).

Example:

->

[Ptbird1]

@MoonRide303 Thanks for the prompt (hehe) reply! I have been AFK for a week so didn't know you implemented canny Cn!! That's so cool and I will definitely be using that once I'm back at my PC! I've used img2img a bunch and love it but what I was referring to was mixing two pre-existing images into a new generated image that contains the features of both originals. See photo

I'm unsure if this would be feasible in SDXL (at least yet) as after looking at op's link for IPadaptrer that has that feature it uses sd 1.5. Thanks again for a fantastic fork of a fantastic program! Keep up the good work and I'll be keeping a close eye on any updates, this stuff is just too fun!

[MoonRide303]

@aimarxjg It should automatically pull update from github when you start it via run-mre.bat. Alternatively - if you have git installed in your system - you can just use git pull command in Fooocus-MRE folder. New version should automatically download any additional model files necessary for new features (like Canny) to work.

[vcm07]

I'm unsure if this would be feasible in SDXL (at least yet) as after looking at op's link for IPadaptrer that has that feature it uses sd 1.5. Thanks again for a fantastic fork of a fantastic program! Keep up the good work and I'll be keeping a close eye on any updates, this stuff is just too fun!

It is feasible; actually, it is already happening; people are already using it for SDXL on comfyui (only works there). All the links I've posted are for SDXL (check ip_adapter_sdxl_demo.ipynb on the tencent link). Ipadapter for SDXL is one of the hottest things happening in the scene right now. And there is a need for an upscaler on Foocus/MRE, or at least something similar to the "highresfix" (from Auto1111), to correct eyes, etc., and then upscale.

but what I was referring to was mixing two pre-existing images into a new generated image that contains the features of both originals. See photo

Yep, that is exactly it. Your baby yoda-cat example illustrates it nicely. Here is another example:

[MoonRide303]

@vcm07 I like those results, but problem is I don't see IPAdapter in base ComfyUI code - and I'd rather not pull any custom nodes not maintained by comfy. I can get back to it when IPAdapter will be available in comfy backend - I've added feature request for that on ComfyUI side (ComfyUI issue #1392).

[MoonRide303]

@aimarxjg If MRE will find those files in proper folders then he won't try to download them, again. So if you already have those files, or can download it in a different way faster - feel free to speed up the process, and just put them in proper places :).

[vcm07]

@vcm07 I like those results, but problem is I don't see IPAdapter in base ComfyUI code - and I'd rather not pull any custom nodes not maintained by comfy. I can get back to it when IPAdapter will be available in comfy backend - I've added feature request for that on ComfyUI side (ComfyUI issue #1392).

No problem. I agree. Watching more closely, I see that most of these workflows and custom nodes with Ipadapter are very volatile. It happened a few days ago when Comfy updated recently; it broke everything, even the workflows, and these people (third parties) do add lots of new stuff (new custom nodes), change a lot of stuff, and this could create nightmarish upgrade problems, so you are right, it would be better to wait for an official implementation. Thanks for adding the feature request on the ComfyUI side.

[MoonRide303]

Upscaling is now possible in v2.0.12 via Image-2-Image - see example in Wiki.

[vcm07]

So, we are almost there, Fooocus with IPAdapter: lllyasviel#557

MRE with this = perfection;

[MoonRide303]

@vcm07 Those new features are still being stabilised in vanilla, but I will look into merging it within next couple of days.

[vcm07]

Fantastic! Thanks for all the work you're doing