Prompt still used with weight 0 (with .and() syntax)

[jonahclarsen]

When passing this as the prompt_embeds param to the StableDiffusionImg2ImgPipeline:

compel("('a man eating an apple', 'sitting on the roof of a car', 'high quality, trending on artstation, 8K UHD').and(1, 0.5, 0)")

gives a different result than

compel("('a man eating an apple', 'sitting on the roof of a car', 'a beautiful sunset').and(1, 0.5, 0)")

even though the third prompt should have a weight of 0 and therefore no effect on the output.

Am I misunderstanding something? Or is this a bug?

[damian0815]

does the same thing happen when you use "? compel doesn't recognise single quotes ', which means it's passing your entire prompt as-is to CLIP in each case

[jonahclarsen]

does the same thing happen when you use "? compel doesn't recognise single quotes ', which means it's passing your entire prompt as-is to CLIP in each case

With double quotes I get the "tensor not same size" error that's discussed in the other issue. It wasn't clear to me how to get around that, could you provide a full working code example, including the .and() syntax?

[damian0815]

does the same thing happen when you use "? compel doesn't recognise single quotes ', which means it's passing your entire prompt as-is to CLIP in each case

With double quotes I get the "tensor not same size" error that's discussed in the other issue. It wasn't clear to me how to get around that, could you provide a full working code example, including the .and() syntax?

sure, you need to do

prompt = '("a man eating an apple", "sitting on the roof of a car", "a beautiful sunset").and(1, 0.5, 0)'
negative_prompt = "distorted, deformed, ugly"
prompt_embeds = compel(prompt)
negative_prompt_embeds = compel(negative_prompt)
[prompt_embeds, negative_prompt_embeds] = compel.pad_conditioning_tensors_to_same_length([prompt_embeds, negative_prompt_embeds])

[jonahclarsen]

Finally got it working, but I have the same issue - with the .and() syntax, weights of 0 don't ignore the corresponding prompt. I'll put the whole program so it's reproducible, see "Comment out this line" (I'm using mps, not sure if cuda/cpu give different results):

pipe = StableDiffusionImg2ImgPipeline.from_pretrained( "stabilityai/stable-diffusion-2-1")
pipe = pipe.to(device)
pipe.enable_attention_slicing()

init_image = Image.open("IMG.jpg").convert("RGB").resize((512, 512))

generator = torch.Generator(device=device).manual_seed(93285438492190234)

from compel import Compel
compel = Compel(tokenizer=pipe.tokenizer, text_encoder=pipe.text_encoder)
prompt = '("A field of golden retrievers", "sitting on the roof of a car", "a beautiful sunset").and(1, 0, 0)'
prompt = '("A field of golden retrievers").and(1)'  # Comment out this line and I would expect to get the same result, but I don't
negative_prompt = ""
prompt_embeds = compel(prompt)
negative_prompt_embeds = compel(negative_prompt)
[prompt_embeds, negative_prompt_embeds] = compel.pad_conditioning_tensors_to_same_length([prompt_embeds, negative_prompt_embeds])

images2 = pipe(prompt_embeds=prompt_embeds,
              negative_prompt_embeds=negative_prompt_embeds,
              image=init_image,
              generator=generator,
              strength=0.7,
              guidance_scale=7.5,
              num_inference_steps=15).images

images2[0].show()

Am I right in assuming that giving one prompt with a weight of 1 should give the same results as that one prompt with a weight of 1 + two prompts each with a weight of 0, as in this example?

[damian0815]

Am I right in assuming that giving one prompt with a weight of 1 should give the same results as that one prompt with a weight of 1 + two prompts each with a weight of 0, as in this example?

that is indeed a reasonable assumption, but it mathematically doesn't work out, for the same reason that if you do run SD with a prompt of "" (empty string) you'll get an image that isn't nothing (weighting works by lerping from "").

try this:

prompt = '("A field of golden retrievers", "", "").and(1, 0, 0)'

and also this:

prompt = '("A field of golden retrievers", "", "").and(1, 1, 1)'

i believe these should both give you the same output as

prompt = '("A field of golden retrievers", "sitting on the roof of a car", "a beautiful sunset").and(1, 0, 0)'

one solution for this is that compel could filter out the 0-weighted terms so that ("A field of golden retrievers", "sitting on the roof of a car", "a beautiful sunset").and(1, 0, 0) would be pushed through as ("A field of golden retrievers").and(1). however this means there's going to be a big jump in output from ("A field of golden retrievers", "sitting on the roof of a car", "a beautiful sunset").and(1, 0.01, 0.01) to ("A field of golden retrievers", "sitting on the roof of a car", "a beautiful sunset").and(1, 0, 0).

the decision i made here is for numerical stability when you're iterating over different weights, rather than for logical consistency at syntax-level. it's a similar kind of question as for how to handle regular upweight (a cat playing with a (ball)0 vs a cat playing with a (ball)0.01).

[damian0815]

@jonahclarsen i'm going to close this, please re-open if you need