(Idea/help/future pull request) I'm trying to add a "Stop at convergence" feature, but I can't get the elements I added to the UI to actually affect the generation parameters

[yspacedev]

I modified the sd_samplers.py file so that every time store_latent() is called, it compares the current and last latent tensors. The changes that Stable Diffusion makes to the image gets less and less over time, so the similarity between successive images should get higher and higher until they become the same or almost the same. I've mostly implemented that functionality.

I don't know how to add gradio elements to the ui.py file (not as a custom script) and make them change values in the program. The code is really complex, and I have no idea how to do this. Everything I have tried has not worked. Also, for whatever reason, it seems like the ui.py code does not allow elements that have float values (which is kind of necessary for this. (or at least floats never work for me) (So how do I add a slider or number box that can have a float value)

I have 3 values:
go_to_convergence - a checkbox to turn on convergence mode
reduce_convergence_tolerance_step - the step at which to increase (the name is a bit misleading) the convergence tolerance (the difference between images (no difference is 0) that the program considers to be fully converged.
reduced_convergence_tolerance - the aforementioned increased (reduced) convergence tolerance (is a percentage)

I'm also new-ish to GitHub (I've never done a pull request before), so help will be greatly appreciated. I'll attach the code later.

TL;DR: UI is big and confusing. How do I add stuff to it without changing everything and all things that depend on it?

[grexzen]

Use the scripts tutorial in the wiki. There you can easily change UI elements, just check the other scripts for reference.

This is a great idea, I also programmed an aesthetic scorer, and I just generated a couple hundred thousand images to test the correlation between step 1 gen and step 20, and it seems to have a limit at score 4.

So in my code if the gen scores 4 or less at step 1, it moves to next prompt+seed to not waste time w trash.

With your code if aesthetic score test 1 passes, and score test 2 passes (score above 7.7 for best pics), and if convergence already approximated 0, then stop.

Do calculate what is the 2 sigma from the mean on your convergence function, so you know a good limit and test it across a couple 1000 samples.

[yspacedev]

I can't implement a custom script because the convergence detector lives in sd_samplers.py. Also, I'm thinking that this is something that you would use with other custom scripts.

This feature is not for textual inversion (although that should be a good idea). Instead, it's for the case where someone us like, "I want to generate the best image that SD can generate with this prompt, but I don't know how many steps it should take. So what I'll do is set the steps number really high and have it stop iterating and return the image when SD stops removing noise and adding significantly to the image."

I could make a custom script for testing purposes, but I would like to integrate it into the base ui in the final pull request.

Also I have no idea what "2 sigma from the mean on your convergence function" means because I'm not that familiar with SD, and I don't know that math.

The current way the convergence comparator works is by taking two latent tensors (current and last), putting them through a thing that quickly turns them into an approximate image (based on this hugging face thread: https://discuss.huggingface.co/t/decoding-latents-to-rgb-without-upscaling/23204/4), comparing the numpy arrays of the approximate images using np.mean(latent1 != latent2), and getting the difference value as a percent. The program then raises the state.interrupted (will be changed in the future) flag when that difference value is below a certain number.

[yspacedev]

I could use a custom script for testing, though

[yspacedev]

I did make a custom script, and the feature does work as intended. However, adding a convergence function is not a very good addition to Stable Diffusion because if you set the number of steps to a big number (like 1000), less noise is added in between images, and therefore, you get worse images at step 20 of a 1000 step process than step 20 of a 20 step process. Also, in a 1000 step process, the images become very similar between steps because less noise is added and less noise is removed each step, which would cause the convergence indicator to think that the images have converged even when the resulting convergence stopped image will be blurry and terrible. Maybe if you make SD de-noise the image the same amount each step regardless of the number of steps, and have the latent comparator detect convergence, it would be better. But, I don't know if developing that would be worth it.

[yspacedev]

I'm just going to mark this as completed since I solved the problem by creating a simple custom script, and it's going to take a lot more work to make an actually good convergence feature.