Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] SD3 unconditional seems to be incorrect, model can do the infamous 'woman laying in grass' image generating the unconditional a different way #3800

Closed
CodeExplode opened this issue Jun 20, 2024 · 2 comments

Comments

@CodeExplode
Copy link

CodeExplode commented Jun 20, 2024

The Clip Text Encode (Negative Prompt) seems to be generating the incorrect negative conditioning for the model. Using the SD3 triple text encoder node with empty padding: None to generate the unconditional seems to greatly improve things.

It would appear that it is generating an unconditional tensor of 1x1x4096 shape, rather than 1x154x4096 shape, based on some early testing.

See the workflow in these images:

SD3_02449_

SD3_02445_

@CodeExplode CodeExplode changed the title [Bug] SD3 unconditional is incorrect, model can do the infamous 'woman laying in grass' image just fine. [Bug] SD3 unconditional seems to be incorrect, model can do the infamous 'woman laying in grass' image generating the unconditional a different way Jun 20, 2024
@CodeExplode
Copy link
Author

After some testing, it looks like SD 2B's best unconditional is a single encoded T5 token (the T5 EOS token according to Kenji on Discord) which is 1x4096 in dimensions. You can use zeroes in those dimensions and it may look a little less cooked, but it seems to introduce phantom limbs. For pooled it's just 1x2048 zeroes. This seems to greatly improve results over the current implementation.

@mcmonkey4eva
Copy link
Contributor

Your images are coming out cleanly because you're not doing the infamous woman lying in grass prompt, you're doing an actual decent prompt, and the model is keeping the character upright (not sideways).

The same prompt works perfectly fine on a pure default workflow with none of the overcomplication
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants