Saving fetched generation progress images; apply VAE at selected Diffuse steps

[alex-ie]

I have tried text-2-image several months ago in koboldcpp (stable-diffusion.cpp downstream) and at the time I thought why not generate images from latent space at every step and be able to stop when quality becomes acceptable (since whole VAE seems to be computationally easier than one Diffusion step). Now kcpp's SD UI have "Fetch generation progress" where images are displayed at every step and it seems not to slow down progress significantly.

I propose:

1. Relatively easy - just save all those fetched images (add option for that).
2. Generate full scale image at selected steps (of with some step interval) - how difficult will it be? Maybe the work is already in progress for that, I am new here, I beg your pardon.

I want to help develop that if help is needed.

TIA

[wbruna]

1. Relatively easy - just save all those fetched images (add option for that).

This is already supported, since #1895 .

2. Generate full scale image at selected steps (of with some step interval) - how difficult will it be? Maybe the work is already in progress for that, I am new here, I beg your pardon.

By "Full scale", I believe you mean "VAE-decoded"?

IMHO, a full "two kinds of previews in parallel" would probably be too complex at the API and command-line levels to be worth it. Some way to specify "full preview each n steps, proj preview otherwise" could be more doable: say, either new "proj+vae" and "proj+tae" modes or a new boolean flag, and repurposing the preview interval to mean "full preview at N, proj otherwise".

[alex-ie]

"By "Full scale", I believe you mean "VAE-decoded"?"

My knowledge of SD is mostly from answers of some LLMs and I was informed latent space is not a decreased resolution picture, but some abstract space, created by VAE? initially. If that is true, do we not need VAE already to fetch progress? I meant just full resolution. But if my above knowledge is not correct, please help me learn correct process - where to read summary (wikipedia maybe)? TIA

[wbruna]

I was informed latent space is not a decreased resolution picture, but some abstract space, created by VAE? initially. If that is true, do we not need VAE already to fetch progress?

That is correct, but the VAE's job can be approximated by smaller neural networks (like TAE, often with pretty good results), or roughly approximated by simple linear projections (like the proj previews). For the later, we usually don't bother mapping a full-scale image, and just (if I remember it correctly) keep the latent width and height dimensions, mapping the latent values on each position to rgb pixel values. So the result is not a 'downscaled version' of the full image, but often is close enough to be useful as a quick preview.

[alex-ie]

Some way to specify "full preview each n steps, proj preview otherwise" could be more doable

Yes, I agree.

VAE's job can be approximated by smaller neural networks (like TAE ... oughly approximated by simple linear projections (like the proj previews)

Thanks for the explanations! Something for me to learn.

[alex-ie]

I have run sd.cpp -cli and it supports arguments to set preview decoder.
https://github.com/leejet/stable-diffusion.cpp/blob/master/docs/taesd.md does not list TAE for flux (which z-image uses for VAE), even though there is tae flux.1 uploaded by madebyollin. It means sd.cpp not yet supports that tae, correct?

P.S. from my observations, proj previews times are negligible, but vae increase generation duration ~1.15x if VAE fits in VRAM and 1.4x if tiling is used.

[wbruna]

tae flux.1 uploaded by madebyollin. It means sd.cpp not yet supports that tae, correct?

The docs need to be updated: taef1 is supported.

vae increase generation duration ~1.15x if VAE fits in VRAM and 1.4x if tiling is used

See if --vae-conv-direct helps.