Suggestion: Add "gobig" mode (a la lowfuel's progrockdiffusion) for huge images

[lendrick]

https://github.com/lowfuel/progrockdiffusion

progrockdiffusion is a Disco Diffusion CLI tool that has a mode called "gobig". When you run it in gobig mode, it generates an image, splits it into overlapping vertical strips, scales those up 2x (or some smaller factor), runs those strips through as init images with the same prompt to add detail, and reassembles the strips into a larger image (with some feathering on the edges to hide the seams). I did this process manually with stable diffusion, and the results were amazing.

[blessedcoolant]

This would be pretty darn neat to have. Good suggestion.

[Oceanswave]

Hmm.. I used a similar technique on Dall*E - however rather than "zooming-in", "zoomed-out"

Since they're locked to 1024x1024, but they have in-painting support, one could create a larger image by supplying back an edge strip of the original image, get the resulting image in the original style and then stitch the images together in photopea or similar - and ta da, you now have a 1024x2048 image

Maybe with in-painting support we could do something similar - if it's a PNG auto create a mask from transparent pixels, run it through, auto-stitch. hmm..

Either mechanism would be really interesting..

Tangentially, this could let folks with low amounts of VRAM to generate bigger images (maybe that's what the optimized one does...)

[softyoda]

Maybe it could be possible to merge this newly code : https://github.com/jquesnelle/txt2imghd
It would be cool to have those feature on this repo as it have good shell engine (with batch process) and also a web interface.

[JesseJS]

I'll add on to this feature request and share that it looks like someone has created a version of the main txt2img script with progrockdiffusion implementation: Txt2imgHD

Maybe the dream script could be made to situationally call this txt2imghd script in addition to the base txt2img?

[softyoda]

It seems there is also code for progstable (that have gobig feature) https://github.com/lowfuel/progrock-stable
It's also from lowfuel.
I hope those feature could be integrated in any ways to one repository, as it is more convenient to have all tools centralized without jumping from one conda env to another.
It would also allow to engage more people (artists not specifically tech/dev) and think about other features like for example a vector motion image sequence that goes along the gobig code that would allow moving in sort of 3D, that would give more artistic control.

[darthdeus]

however rather than "zooming-in", "zoomed-out"

This feels like a killer features to add tbh. One of the problems I feel with SD is that often the images are cropped in a very inconvenient way, where "zooming out a little bit" would solve the problem.

It doesn't seem this is possible with the inpaint.py though, or at least that's what I initially hoped to use it for, but the results are more like "photoshop context aware fill" where it tries to preserve local consistency, even when the image as a whole doesn't make sense (for example inpainting a mouth just replaces the mouth with skin, rather than re-painting the mouth).

[lstein]

Just folded in experimental support for the GFPGAN package, which does face fixing pretty well and upscaling really well. It's a little unstable at the moment, but will be reliable soon.

[blessedcoolant]

Just folded in experimental support for the GFPGAN package, which does face fixing pretty well and upscaling really well. It's a little unstable at the moment, but will be reliable soon.

Just a note here. The gobig mode is functionally different from just upscaling. While upscaling just enlarges the generated image, the gobig mode takes a more generative approach.

In the gobig mode, the generated image is cut into multiple overlaying crops and all these individual images are fed back to the generator to generate greater detail for them. These new images are then stitched and blended together. This effectively produces a large scale version similar to the original generation but with far greater detail -- original detail. Not upscaled.

Given how good we handle the stitching and sub generation process, these results can turn out to be absolutely astounding.

Upscaling can be used in sync with this but I believe implementing this mode will have its own benefits.

I'd also recommend calling it a highdetail mode rather than gobig mode to avoid any confusion.

I implemented this a few days prior locally (now deleted) and it works but I feel a lot can be done to make it a worthy mode for a larger variety of prompts. I'm working on how to go about achieving that. When I work it out, I'll make a PR with this feature -- unless someone else is willing to do it before that.

[janekm]

Just adding to this that GFPGAN is not a replacement for "Gobig" mode. It's a first step in the Gobig process but can't replace it. The SD model has fundamental issues with faces at the middle distance due to the autoencoder used. "face fixing" code doesn't work well for those issues as it a) often doesn't work at those detail levels either and b) often moves too far away from the style of the prompt.

GoBig mode as implemented in the other repos is a big help for fixing faces (also can do wonders for certain high-detail prompts). It does introduce issues in other parts of the image, but there might be a clever way to use masks to blend with the GFPGAN image.

[lkewis]

Further to these two comments, it would be great to have this implemented into the main dream.py script, currently I'm adding txt2imghd and Real-ESRGAN into my local repo in order to use it separately. They just added --img for input images to the txt2imghd script which makes it easier to use as a post process now, but plugging it into the main loop would make a more seamless experience.

I'm used to this being called 'upstitching' from the Huemin Jax Diffusion notebooks, and it is far superior to just regular upscaling since it adds all the details back in or the strength can be cranked to get even more unique aesthetic from the originally generated image.