-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
VAE (vae-ft-mse-840000-ema-pruned.ckpt) Documentation #1279
Comments
what where the prompts you used? |
But I assume it generalizes to other prompts. |
Would like to compare what they look on my end, since I did not have those problematic eyes. Will test after MacOS Update is done ^^ |
Are there any instructions how to use it with InvokeAI? |
@michaelezra In
add vae to the model(s). Download: https://huggingface.co/stabilityai/sd-vae-ft-mse-original/tree/main |
@mauwii Interesting. I didn't use One thing. When you run
I assume after doing this, your command changed and you passed some post-processing options? e.g. |
Oh, yes, sorry, should have mentioned this. What happened: When I created the first picture i was totally iritated since my GFPGAN was broken as well so even postprocessing wasnt working anymore (which I guess came from about 100 reinstallations while "playing" with the environment-mac.yml 🙈). Long story short - the second picture was postprocessed with GFPGAN 🤦🏻♂️ Hope I did not lead you on some non-existing route! |
I edited the post and added the prompt |
Do you think it may be useful to include these results / comparisons in https://github.com/invoke-ai/InvokeAI/blob/development/docs/features/POSTPROCESS.md? For example, using GFPGAN does help, but it may be useful to use with the new VAE. See in #1279 (comment) how GFPGAN-only eyes are a bit more blurry than VAE-only eyes and VAE-GFPGAN eyes. Note: There's the option of increasing -G (face restoration strength), to try to improve the eyes, but it will also smoothen details in the face, and you can't decouple it from eyes-restoration-only as far as I know. So combining GFPGAN with VAE may be a nice idea (e.g. say you want wrinkles in the face for an old person but also good eyes). |
I guess your excelent comparison-screenshots would ofc fit very well onto the postprocessing docs, since Pics (especially in this case) sometimes say more than 100 lines of docs could do 🙈 |
@Any-Winter-4079 Thanks! in models.yaml I tried v1.5:
I created a symlink to vae-ft-mse-840000-ema-pruned.ckpt `(invokeai) michaelezra@Michaels-MBP stable-diffusion % PATH_TO_CKPT="/Users/michaelezra/ai/CKPT" (invokeai) michaelezra@Michaels-MBP stable-diffusion % python scripts/invoke.py --web --model stable-diffusion-1.5
Does this look right - no message about vae. |
Totaly agree. I can maybe give you a good advice about how to edit the docs comfortably: # while in repository-root
python -m venv .venv
. .venv/bin/activate
pip install -r requirements-mkdocs.txt
mkdocs serve this spins up a local version of the github-page so that you have a live preview (which also reloads as soon as you save some changes). I used it so often that in the meantime I added the mkdocs requirements to my conda base env 😅
Yeah, I also noted that "blurrynes". I mostly used a value between 6-7.5 with GFPGAN, which varies depending on the picture itself, but here it is recommended even between 5-8, in the end it's also kind of a personal flavor I guess. Combining the postprocessors Sounds legit, but unfortunatelly I didn't dive that deep into it yet 🙈 (I even did never use the whole stuff like inpainting, outpainting, ..., but the day will come ;P ) @michaelezra: since I just want to try this as well, this is how it looks for me: models.yml: stable-diffusion-1.5:
config: configs/stable-diffusion/v1-inference.yaml
weights: models/ldm/stable-diffusion-v1/v1-5-pruned-emaonly.ckpt
vae: models/ldm/stable-diffusion-v1/vae-ft-mse-840000-ema-pruned.ckpt
description: Stable Diffusion inference model version 1.5
width: 512
height: 512 running invoke.py: python scripts/invoke.py --model stable-diffusion-1.5
* Initializing, be patient...
NOTE: Redirects are currently not supported in Windows or MacOs.
>> GFPGAN Initialized
>> CodeFormer Initialized
>> ESRGAN Initialized
>> Using device_type mps
>> Loading stable-diffusion-1.5 from models/ldm/stable-diffusion-v1/v1-5-pruned-emaonly.ckpt
| LatentDiffusion: Running in eps-prediction mode
| DiffusionWrapper has 859.52 M params.
| Making attention of type 'vanilla' with 512 in_channels
| Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
| Making attention of type 'vanilla' with 512 in_channels
| Using more accurate float32 precision
| Loading VAE weights from: models/ldm/stable-diffusion-v1/vae-ft-mse-840000-ema-pruned.ckpt
>> Model loaded in 12.44s
>> Setting Sampler to k_lms
* Initialization done! Awaiting your command (-h for help, 'q' to quit)
invoke> VAE weights are also "only" symlinked to models/ldm/stable-diffusion-v1/vae-ft-mse-840000-ema-pruned.ckpt |
I just downloaded the file and put it into models/ldm/stable-diffusion-v1, so I don't know what the issue may be. If a symlink doesn't work in your setup, try putting the ckpt directly in the folder. |
strange.. I am still not getting this, after placing the file directly and changing the order in models.yaml, putting vae: after weights: Loading VAE weights from: models/ldm/stable-diffusion-v1/vae-ft-mse-840000-ema-pruned.ckpt |
So 1.5 is the default model you load?
so unless you switch models, it shouldn't load the vae ckpt. Basically, for every model you want to use this vae with, you have to add the vae line, e.g.
And on startup, it will load whichever has default: true, e.g.
|
I used to explicitly specify, which model to load like this: now I added default: true to v1.5
now it is loading v1.4 be default
still no luck with vae. I am using master branch, is that ok? |
you have 4 spaces in your yaml, I only 2, maybe this is the problem, dunno stable-diffusion-1.5:
description: The newest Stable Diffusion version 1.5 weight file (4.27 GB)
weights: ./models/ldm/stable-diffusion-v1/v1-5-pruned-emaonly.ckpt
config: ./configs/stable-diffusion/v1-inference.yaml
width: 512
height: 512
vae: ./models/ldm/stable-diffusion-v1/vae-ft-mse-840000-ema-pruned.ckpt
default: true |
I copied your yaml file, but no difference:
|
When I remove v1.5 from models.yaml I am getting this error - seems like v1.4 is being defaulted from some place else python scripts/invoke.py --web
|
could you maybe post a complete copy paste like this one: python scripts/invoke.py --model stable-diffusion-1.5
* Initializing, be patient...
NOTE: Redirects are currently not supported in Windows or MacOs.
>> GFPGAN Initialized
>> CodeFormer Initialized
>> ESRGAN Initialized
>> Using device_type mps
>> Loading stable-diffusion-1.5 from models/ldm/stable-diffusion-v1/v1-5-pruned-emaonly.ckpt
| LatentDiffusion: Running in eps-prediction mode
| DiffusionWrapper has 859.52 M params.
| Making attention of type 'vanilla' with 512 in_channels
| Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
| Making attention of type 'vanilla' with 512 in_channels
| Using more accurate float32 precision
| Loading VAE weights from: models/ldm/stable-diffusion-v1/vae-ft-mse-840000-ema-pruned.ckpt
>> Model loaded in 12.44s
>> Setting Sampler to k_lms
* Initialization done! Awaiting your command (-h for help, 'q' to quit)
invoke> so from the command you enter to the invoke prompt. In the last one there is did you try to delete the conda env, clean your git branch and start fresh? |
here it is:
I didn't try to "delete the conda env, clean your git branch and start fresh". |
ok.. cleaned git, conda, reinstalled all.. and result is the same:) |
are you using the main branch, where this is functionality is not yet implemented 🤔 At least in this branch there are still those 4 whitespaces instead of 2 in the models.yaml, and when I use it I also just get: python scripts/invoke.py --model stable-diffusion-1.5
* Initializing, be patient...
NOTE: Redirects are currently not supported in Windows or MacOs.
>> GFPGAN Initialized
>> CodeFormer Initialized
>> ESRGAN Initialized
>> Using device_type mps
>> Loading stable-diffusion-1.5 from ./models/ldm/stable-diffusion-v1/v1-5-pruned-emaonly.ckpt
| LatentDiffusion: Running in eps-prediction mode
| DiffusionWrapper has 859.52 M params.
| Making attention of type 'vanilla' with 512 in_channels
| Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
| Making attention of type 'vanilla' with 512 in_channels
| Using more accurate float32 precision
>> Model loaded in 9.96s
>> Setting Sampler to k_lms
* Initialization done! Awaiting your command (-h for help, 'q' to quit)
invoke> YAML was of course updated |
oh...yes I am in the main branch:)) |
well, if you need to ask, you should maybe better wait for a release ;P |
I switched to development branch.. |
@Any-Winter-4079 I re-deployed a 2 week old version of the docs where it wasnt that broken as now, just take a look at https://mauwii.github.io/stable-diffusion/installation/INSTALL_MAC/, when the script blocks startk there is another very nice functionality with those switchable tabs. Or this site https://mauwii.github.io/stable-diffusion/features/PROMPTS/ is also something which could never look so good in github itself. But well, humanity is doomed to not be able to have nice things I guess 🤷🏻♂️ |
Houston, we have a victory!:) python scripts/invoke.py --web --model stable-diffusion-1.5
Thanks for your help! |
I think this is from #1278 🙈but yep, your docs look awesome! |
Edit: Lol - should first have clicked, then written xD Every link in the main README.md points to it (like Installation, inpainting, outpainting, bob ross painting, ....) as well as a link in the top of the README.md and in the InfoBox of the of the Repo in the upper right corner, so not sure how many links would be necesarry 🤷🏻♂️ BTW: just try out the search bar in the MkDocs and you directly know why it is totaly imba vs plain old markdowns on GH ;P |
Oh my bad! Then it's fine 🙈 Odd as it is, I barely scroll down (I mainly use Issues, Pull Requests, etc. tabs at the top, and the code files, but I almost never scroll down to the readme) |
without having them cound, I guess the MkDocs Page is linked at least 20 times in the readme 🤗 But if you have any other good Ideas how to point to the site I (and 4sure @lstein as well) would apreciate it a lot 👻 |
@Any-Winter-4079, I'm hijacking this discussion thread just to check up. I haven't heard anything from you for about two weeks and wonder if you're still active on the project? |
Busy week (3 exams and 4 projects) here that ends in 2 days. |
I believe the technical issues were resolved here. Hope your exams went well @Any-Winter-4079 ! |
Finally got to try
![image](https://user-images.githubusercontent.com/50542132/198444197-945c2293-00e5-4d43-a9c7-d5f4c045324e.png)
vae-ft-mse-840000-ema-pruned.ckpt
and my first impression is wow: seems to do better with eyes! Have you noticed some other benefits (e.g. for landscapes, etc.)? I think it would be good to document the strengths and weaknesses of this new vae and also add it to the docs.The text was updated successfully, but these errors were encountered: