Initial Commit for POC for Aesthetic Gradients

[MartinCairnsSQL]

Describe what this pull request is trying to achieve.

Add the ability to use aesthetic gradients embedding training from https://github.com/vicgalle/stable-diffusion-aesthetic-gradients

Additional notes and description of your changes

This is just an initial POC to show the functionality working in the WebUI.
Helper functions are needed for get lists of embeddings
The injection point of the aesthetic gradient training needs to be looked at to make sure it's at the current level in the prompt parser.
A tab or script needs to be added to train a new aesthetic gradients embedding on a folder of images.

Environment this was tested in

OS: Windows 10
Browser: Firefox
Graphics card: NVIDIA RTX 3090 24GB,

Screenshots or videos of your changes

Prompt: A portal towards other dimension Steps: 50, Sampler: PLMS, CFG scale: 7, Seed: 332, Size: 512x512, Model hash: 7460a6fa
Aesthetic Embeddings: sac_8plus.pt

Aesthethic Embedding Steps: 0

Aesthethic Embedding Steps: 15

Aesthethic Embedding Steps: 20

Original Images from https://github.com/vicgalle/stable-diffusion-aesthetic-gradients

[moorehousew]

Niiice. Went to bed after reading the paper, woke up and saw the pull request.

[MartinCairnsSQL]

I've added the quicksettings functionality to the PR, quick settings needs to be changed to "sd_model_checkpoint, aesthetic_embedding, aesthetic_embedding_steps" to get access to the settings.

[moorehousew]

You also might want to remove the .pt files in the aesthetic_embeddings directory from the commit and add aesthetic_embeddings/*.pt to the .gitignore- I don't imagine we'd want to bundle them with the web UI, but that's up to @AUTOMATIC1111.

[MrCheeze]

Not yet working with --no-half:

Traceback (most recent call last):
  File "C:\Users\Matthew\Documents\git\stable-diffusion-webui\modules\ui.py", line 186, in f
    res = list(func(*args, **kwargs))
  File "C:\Users\Matthew\Documents\git\stable-diffusion-webui\webui.py", line 65, in f
    res = func(*args, **kwargs)
  File "C:\Users\Matthew\Documents\git\stable-diffusion-webui\modules\txt2img.py", line 43, in txt2img
    processed = process_images(p)
  File "C:\Users\Matthew\Documents\git\stable-diffusion-webui\modules\processing.py", line 398, in process_images
    c = prompt_parser.get_multicond_learned_conditioning(shared.sd_model, prompts, p.steps)
  File "C:\Users\Matthew\Documents\git\stable-diffusion-webui\modules\prompt_parser.py", line 209, in get_multicond_learned_conditioning
    aesthetic_conditioning = adjust_for_aesthetic_gradient(learned_conditioning, prompt)
  File "C:\Users\Matthew\Documents\git\stable-diffusion-webui\modules\aesthetic_gradients.py", line 16, in adjust_for_aesthetic_gradient
    return _adjust_for_aesthetic_gradient(device, prompt, aesthetic_embedding, aesthetic_embedding_path, aesthetic_embedding_steps)
  File "C:\Users\Matthew\Documents\git\stable-diffusion-webui\modules\aesthetic_gradients.py", line 51, in _adjust_for_aesthetic_gradient
    loss = getLoss()
  File "C:\Users\Matthew\Documents\git\stable-diffusion-webui\modules\aesthetic_gradients.py", line 48, in getLoss
    sim = text_embs @ image_embs.T
RuntimeError: expected scalar type Float but found Half```

[MarkovInequality]

I noticed a lack of code for generating these embeddings in this PR. Would it be possible to include it in the webui? I found it here: https://github.com/vicgalle/stable-diffusion-aesthetic-gradients/blob/d0bee3589aa937360024fc01057ae6ed078e6c20/scripts/gen_aesthetic_embeddings.py
This way we can create the aesthetic embeddings from the UI as well.

[MartinCairnsSQL]

I'm planing to include the ui for the building, once I solve the issue of finding the correct torch device settings and the correct sd model hijack point to change how the tokens become the embedding model. As that will be the problem for merging into the main project.

For the creation in the mid term I'll change that script to take inputs, as for my first and only aesthetic embedding I just editted the script to make them.

[bmaltais]

I think there might be a memory leak (or lack of VRAM deallocation) after the training phase because my computer run out of VRAM after about 6 or 7 image generation... but this is cool work.

The images it produces are very interesting. I created an embeding from a series of images for a style and it sort of produce results more like the style... but not really the same. Still trying to figure out how to make use of this...

[MartinCairnsSQL]

After reviewing #2585 I going to close this draft PR as the other PR looks complete.