Images generated with square brackets ("[]") in prompt can't be reproduced anymore

[Meizr]

I suspect something might have been changed in the way comfyui handles square brackets in prompts.

Consider the following example image:

https://civitai.com/images/1777586

I've reproduced this very image in my machine in April 20, 2024, but can't reproduce it anymore as of May 18, 2024.

UPDATE 3

I've found out what causes difference in behaviour:

It's the pair (pair because of dependency) transformers and tokenizers

With transformers 4.30.2 and tokenizers 0.13.3, [expression] is the same as ([expression]:1)

With transformers 4.39.3 and tokenizers 0.15.2, [expression] is different from ([expression]:1)

UPDATE 2

After more than 20 hours, I've found out.

If I use the following prompt in the recent comfyui build:

car, filmed by ([David Fincher]:1), ancient, winter in space, a fairy tale

I can reproduce the image generated from older build.

Also, in older build ("ComfyUI_windows_portable_nvidia_cu118_or_cpu_31_07_2023") I could get the exact same image (comparared pixel by pixel) by using "the alternate" syntax.

Take a look at both results (older build, showing both sintaxes):

Now take a look at both results from a recent build (updated less than 7 days ago)

So:

this older build's "exclusive" syntax:

car, filmed by [David Fincher], ancient, winter in space, a fairy tale

is the same as newer (and also older!) build's

car, filmed by ([David Fincher]:1), ancient, winter in space, a fairy tale

I don't know if it's a regression, or an expected behaviour....

I remember seeing some commit about the change in [] in file paths, but nothing in prompts....

Old history:

I've always started Comfyui using the following arguments:
--windows-standalone-build --disable-auto-launch --disable-xformers --temp-directory "%tmp%\Comfyui"

Even so, I've tried with xformers, but the result is pretty similar, no significant change.

Also, I've tried some arguments (one at a time), with no success:

--use-split-cross-attention
--disable-cuda-malloc
--force-fp32
--bf16-unet
--disable-ipex-optimize
--force-upcast-attention
--dont-upcast-attention
--deterministic

The one that impacted the most (yet slightly) the generated image was --force-fp32, but nothing significant. The speed was much slower with this one.

Below, both images side by side:
-at the left, the image that could be generated in/before April 20, 2024;
-at the right, the image that is currently generated;

UPDATE 1:

I've downloaded "ComfyUI_windows_portable_nvidia_cu118_or_cpu_31_07_2023" (older build), run it from another drive side-by-side with current build, and could reproduce the images.

I was starting to think it could be a NVIDIA Driver change, but that's not the case:

The older comfyui build can reproduce the images, The newer can't.

I've then created another image with a much simpler workflow: small prompt, no refiner, no negative prompt.

See results below:

Below is the image generated in current build: (torch 2.2.2+cu121)

Below is the image generated in 31_07_2023 build: (torch 2.0.1+cu118)

As you can see, the image is significantly different, eventhough prompt is very simple and small, and there's no negative prompt.

Below is the image generated if I remove square brackets from prompt: (image is very similar in both builds, differences aren't significant):

I've also tried replacing cudnn DLLs ("cudnn_adv_infer64_8", etc) in the current build with those from the older build, but the results were the same with downgraded DLLs. As it also happens when using xformers, I assume torch version doesn't matter (might be wrong here), so I don't know what's the cause. Maybe it there was a change in the way comfyui treats square brackets in prompts...

Below, the workflows:

square brackets workflow.json

civitai workflow.json

Models used in workflows:
https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0/blob/main/sd_xl_base_1.0_0.9vae.safetensors
https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-1.0/blob/main/sd_xl_refiner_1.0_0.9vae.safetensors

Test machine:

Running Windows 11 Version 23H2 (OS Build 22631.3235)
CPU: i9-9900K (stock)
GPU: RTX 4080 (stock), 551.61 GRD
RAM: 64GB (DDR4)

[andreszs]

Please spare my ignorance, but exactly why would you insert "#" anywhere in the prompt for that matter?

[Meizr]

Please spare my ignorance, but exactly why would you insert "#" anywhere in the prompt for that matter?

I'd also like to know it. I think it's just a hashtag, like those used in social media posts. (that image wasn't generated by me)
I've updated the post with more details.

[NeedsMoar]

I can't recall what square brackets were supposed to do... were they implicit downweighting like how () upweights slightly?

[NeedsMoar]

According to the community manual [] doesn't do anything in prompts. Downweighting is via () with a number less than 1 specified. [] did downweighting in shark, so the equivalent would be (foo:0.9)

Maybe try that and see if it works? It's possible [] used to downweight and was dropped.

[Meizr]

I can't recall what square brackets were supposed to do... were they implicit downweighting like how () upweights slightly?

I couldn't find an official explanation for what square brackets do in Comfyui.

I was assuming it was decreasing attention for the enclosed expression, but when testing with (David Fincher:-1.10) and other values, I couldn't reproduce that same picture. I didn't included this test in issues' description because I don't really know what square brackets are doing here.
But removing them from the prompt causes an almost identical image.

[NeedsMoar]

I just did a once-over on the conditioning parser. [] do nothing.

[NeedsMoar]

The only thing I can think of is that they were being stripped out before... they don't seem to be now, so they could just be literal [] in the final prompt.

[Meizr]

According to the community manual [] doesn't do anything in prompts. Downweighting is via () with a number less than 1 specified. [] did downweighting in shark, so the equivalent would be (foo:0.9)

Maybe try that and see if it works? It's possible [] used to downweight and was dropped.

Thanks for your reply. I had thought about that as well: that [] was downweighting in older buillds, but I didn't mention it because I could find nothing about it.

I had tried using () with different values, but all results are nowhere near the reference image (generated with the older build). All images remain consistent with what the current build generates (front and side of the car can be seen).

Take a look what I get vs what I should get:

I tried also escaping square bracket by using [ EXPRESSION ] , with no success

The next step is using comfyui files from newer build with whole python from the older. I'll do that later.

[Meizr]

Well, I've found out:

If I use the following prompt in the recent comfyui build:

car, filmed by ([David Fincher]:1), ancient, winter in space, a fairy tale

I can reproduce the image generated in older build.

Also, in older build I could get the exact same image by using "the alternate" syntax. Image is the same, comparared pixel by pixel.

Take a look at both results (older build, showing both sintaxes):

So:

this older build's "exclusive" syntax:
car, filmed by [David Fincher], ancient, winter in space, a fairy tale

is the same as newer (and also older!) build's

car, filmed by ([David Fincher]:1), ancient, winter in space, a fairy tale

I was testing for decimal values instead of 1, very confident it wasn't 1 after reaching more similar images, but then I realized the older build I'm using isn't deterministic (pytorch, and --deterministic argument isn't supported), so I switched to xformers and tested in older build, finally reaching the value of 1.

Now I know whether this change is an expected behaviour....

I remember seeing some commit about the change in [] in file paths, but nothing in prompts....

[NeedsMoar]

How absolutely bizzarre...

All I can seem to find is that it has some meaning in ComfyUI-prompt-control custom nodes where it can do various things... most require specific syntax but one just means (I think) "not emphasized" which I suppose is equivalent to (foo bar:1)
!emphasized: "(" prompt? ")"
| "(" prompt ":" prompt ")"
| "[" prompt "]"
I'm guessing all it would do is treat the contained items as a single token for attention purposes where normally they'd be split and locality would be used. Did you try (David Fincher:1) to see if it does the same thing? @comfyanonymous could probably answer this but it's mostly academic I guess if you've found the workaround.

Any chance those were installed in the version you were using? You'll have a "Conditioning (Advanced)" node if it's around I think.

With other syntaxes it can also be used to schedule items contained in the square brackets for temporal weighting in batches

[Meizr]

All I can seem to find is that it has some meaning in ComfyUI-prompt-control custom nodes where it can do various things... most require specific syntax but one just means (I think) "not emphasized" which I suppose is equivalent to (foo bar:1) !emphasized: "(" prompt? ")" | "(" prompt ":" prompt ")" | "[" prompt "]" I'm guessing all it would do is treat the contained items as a single token for attention purposes where normally they'd be split and locality would be used.

Interesting.....

I had tried also using [Fincher], and ([Fincher]:1) yesterday, but in all tests, results were consistent with using a "compound" expression ("David Fincher").

I also did another very interesting test:

Using a clean and recent comfyui, I've copied comfyui files from 31_07_2023 build (so that embedded python would be from recent build), and square bracket behaviour was the one from NEWER builds.

So it seems it's something inside python (maybe torch and its dependencies) that's causing this difference.

At this point, maybe there's not much that can be done in comfyui's side.

Did you try (David Fincher:1) to see if it does the same thing? @comfyanonymous could probably answer this but it's mostly academic I guess if you've found the workaround.

I had tried, but it's different from both [prompt] and ([prompt]:1)

Also, (David Fincher:1) gives different results in both builds, but that's probably some change in attention thing...

See comparison below, newer and older builds:

Recent build:

31_07_2023 build:

Any chance those were installed in the version you were using? You'll have a "Conditioning (Advanced)" node if it's around I think.

No, just checked, this node isn't here. My customnodes folder is currently without any third party node folder, and there's no "Conditioning (Advanced)" node available.

[Meizr]

I've found out what causes difference in behaviour:

It's the pair (pair because of dependency) transformers and tokenizers

With transformers 4.30.2 and tokenizers 0.13.3, [expression] is the same as ([expression]:1)

With transformers 4.39.3 and tokenizers 0.15.2, [expression] is different from ([expression]:1)

Maybe it's the tokenizers alone, but I can't confirm it due to dependency issues.