[bug]: Extremely low image quality on some models

[JantDarvus]

Is there an existing issue for this problem?

• I have searched the existing issues

Operating system

Windows

GPU vendor

Nvidia (CUDA)

GPU model

GTX 1070

GPU VRAM

8GB

Version number

4.1.0

Browser

Chrome 124.0.6367.61

Python dependencies

{
"accelerate": "0.29.2",
"compel": "2.0.2",
"cuda": "12.1",
"diffusers": "0.27.2",
"numpy": "1.26.4",
"opencv": "4.9.0.80",
"onnx": "1.15.0",
"pillow": "10.3.0",
"python": "3.11.7",
"torch": "2.2.2+cu121",
"torchvision": "0.17.2+cu121",
"transformers": "4.39.3",
"xformers": "0.0.25.post1"
}

What happened

When using some models, image quality is incredibly poor. The models I have encountered this with worked on a previous version of InvokeAI fine (unsure of version, definitely 3.x). However, I reinstalled and updated to the latest version today and these models stopped functioning properly.
Example images provided both had a prompt of solely 'fox, portrait' at 512x512 using DPM++ 2M Karras at 40 steps, CFG 7.5, default VAE.
The first is using easyfluff_5000DefaultvaeFFR3CS2 (easily searched for as 'easyfluff'), the second is using yiffymix_v43. Both were pointed at their appropriate config file and checked using both epsilon and v_prediction, both were also tested as both a checkpoint and a diffusers model.

What you expected to happen

I had expected these models to work as they previously did. For example, this is one using the yiffymix_v43 two days ago.

How to reproduce the problem

Using version 4.1.0, download either yiffymix_v43 or easyfluff_5000DefaultvaeFFR3CS2 and attempt to generate any image at all.

Additional context

I should note that using any of the starter models listed in this version of invoke, such as Deliberate_v5, works absolutely fine. I have also redownloaded all the models and associated files that are having this issue to ensure they haven't mysteriously become corrupted or somesuch at the same time as the update.

Discord username

jantdarvus

[psychedelicious]

Yikes!

For example, this is one using the yiffymix_v43 two days ago.

To clarify: Were you still on 3.x two days ago?

I believe this is the model: https://civitai.com/models/129996/easyfluff

Based on the description there, this model uses v-prediction. I noticed a couple issues in our handling:

1. We aren't correctly determining checkpoint scheduler prediction on import, so the model is set as epsilon.
2. When you go and change it to v-pred, it doesn't change the config file path, which is required to convert the model to diffusers and use it.
3. Even if you manually fix the config file path, if the model was converted once before, it will not be converted again. The diffusers conversion cache must be emptied to trigger the conversion again.

Once I manually fix the config file path (stable-diffusion/v1-inference-v.yaml is the correct value) and delete the model conversion cache, I get better images. Using CLIP Skip 2 and CFG Rescale 0.7 per the model page:

So no longer totally broken, but still not right. They all have this grainy look. I tried enabling upcast_attention, which I think is related to v-pred and re-converting - this didn't seem to change anything.

At this point, I'm not sure how to proceed further.

@lstein Any ideas for fixing the root issue?

I think there are clear solutions for each of the 3 issues I noted:

1. Figure out how to determine the prediction type from the state dict for SD1. Right now, it doesn't try, jsut defaults to epsilon. I don't know how to do this.
2. Do not hardcode the ckpt config file paths unless the user explicitly sets it. When we convert, if there is no config file path, we determine the right config file using the same logic the probe uses.
3. Invalidate the ckpt conversion cache if a change to a model record means it would convert differently. We could simply invalidate the conversion cache on any model config change. I think that is reasonable, though it is less efficient.

[JantDarvus]

I was still on 3.x two days ago, yeah. I had used an old installer I still had on my desktop at the time.
The model you're using is correct, but just to check, is that using the config file provided with the model? Not sure if that might be causing the grainy look there or if there's indeed still an issue.

[psychedelicious]

No, I didn't notice there was a config file provided. Unfortunately, using it doesn't change the outputs - still grainy. I made sure to re-convert the model.

In v3, we used the diffusers ckpt conversion logic with some fixes. Those fixes were upstreamed to diffusers, and in v4 we moved back to the standard diffusers conversion. I suspect we are missing an arg in the conversion calls, or maybe diffusers has a bug.

[JorgeR81]

@JantDarvus, sorry for this unrelated question:

Can you run InvokeAI 4 properly with a GTX 1070 ?

I have the same GPU and I use the Comfy UI with no issues.

I was looking at InvokeAI, but they don't recommend this card.

https://invoke-ai.github.io/InvokeAI/installation/INSTALL_REQUIREMENTS/

Can you run SDXL models with it ?