-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multi batch and process bypass, for a massive improvement. #249
Comments
yes. im waiting this for so long |
I've moved my original comment to a new issue so it gets visibility: #268 |
COOL! I'll check it later. |
Thanks for your suggestions! But little confused about 3.ControlNet Bypass, could you share some examples to help me understand how to properly “blend with”? |
As we know, img2img organically influences controlnet, so I assume you mean how the raw image would be merged. And I have probably underestimated this problem. |
I also mentioned this in #74 with the Video Loopback script. |
Is point 3 about some sort of the feature recreated with the technique in this video: https://www.youtube.com/watch?v=_xHC3bT5GBU ? |
No. In this video a generated image is used, which is similar to the original because it is surely made with the same model. Point 3 is about preserve any original image as they are, to be blended with another. |
I think a second pass with the lighting technique would be very welcome and would still require functionality for batching img2img and controlnet separetely. This, imo, is blending light with the original image in another pass and would count for point 3. |
Surely there are alternatives to try, but the main problem in point 3 is "raw" inference. Currently, both images are generated from noise, and this allows for an organic blend. This means that SD will be able to create from full light to full dark from scratch, respecting both images. |
Looks like point 3 is somehow impl in Composer (#359), but it also requires some modification to img2imgalt.py |
Thanks for the update. Very interesting paper. I have been researching various inversion and latent interpolation methods related to this task. Fortunately, as you have pointed out, many tasks already have their solution counterpart through controlnet with a lower cost of human input, instead processing hungry methods. About img2imgalt and controlnet, my tests without prompt + sigma adjustment are encouraging. Without standing out, its simplicity and low cost make it worth a try. At 0.8 denoising it reaches a similarity that could be enough to achieve strong effects in a reconstruction with other compositions, as we already have seen examples. Finally, I now know that a batch in img2img with nothing in controlnet automatically mixes the image across all layers. However, being able to mix two sources, either one static or both dynamic, is still a major impediment and need to consider. |
Great. Will it influence by different controlnet input? |
If the question was whether CN influences that image, the answer is yes. A concurrent CN with img2alt has an effect on denoising 1, but not in ways worth mentioning. Only the reverse path seems to do the magic. |
Point 1 and 2 almost there. Quick question. Its difficult to make a control just another img2img? With cfg and denoising. Less powerful than img2alt, but would add utilities. |
Thank you very much for your invaluable work. I will leave three changes that I consider top priority, and would be a complete revolution in content generation:
1 - Own Controlnet batch, without Img2Img bypass. (For controlnet blend composition)
2 - Multi-Batch. Img2Img +Controlnet simultaneous batch, for dynamic blend.
3 - Controlnet bypass. Let the chosen image remain "raw" and blend with the one from Img2Img. This can be so powerful for editing of images and videos with full temporal coherence.
Hope you can find a way. I don't know if there is any type of specific complication in any of them. As long as it's not writing code, I offer to research for alternatives and ideas. 👍🏿
The text was updated successfully, but these errors were encountered: