DualClipLoader ignoring clip encoders?

[ekimeister]

Hey!

I was wondering between the differences of using the T5 XXL Q5_K_M encoder paired with clip_L or clip-vit-large-patch14 respectively, and I wanted to try both of them out to see which one would be better for generating images with Flux Dev in a VRAM constrained (12 GB) scenario.

When I tried to generate complex images with a fixed seed, changing only between clip_L and clip-vit-large-patch14 after generations, I got identical results, no matter the prompt. I tried multiple different prompts and different seeds but the end result was always identical, down to SHA-256. I even tried restarting ComfyUI in between.

Is this by design or is something wrong with my setup? For background information, I am running the latest version of ComfyUI and ComfyUI-GGUF with no additional nodes. I have one GPU.

I am very much a hobbyist when it comes to generating AI art so the technical aspects are not very well known to me. I might've misunderstood something completely. If you could enlighten me on this, I would be grateful!

Anyways, the GGUF node and your GGUF quantizations work flawlessly, so thanks for your work for making Flux Dev accessible to us without RTX 5090s. :)

[city96]

That should be expected, since clip-vit-large-patch14 is in fact clip_L (the L stands for large). The main difference is that the file named clip_l has the stuff that's not used by the image generation model removed, while the full one has some stuff that's not used still there (or at least that's how it usually tends to be).

If you run with --verbose DEBUG as the launch argument you'll actually see these being skipped in the cli:

To actually see a difference, you'd have to use a model that's been finetuned in some way. The only one for clip-l that I remember are LongCLIP plus the various finetuned versions by zer0int on Huggingface. I think some of them were supported by ComfyUI natively so they'd just work as-is, though some of the other ones weren't, so you might get random loading errors. I haven't played around with that in a good while.

[ekimeister]

Thanks for the prompt reply. I did not realise they were the same clip models, since the size difference was so large. But it makes sense, that the Flux text encoders only have the stuff in them that Flux needs for text encoding. This would also explain why the results were identical with both models.

I really appreciate the in-depth response, I learned something new today. Cheers.

[city96]

For the size difference, there's also a chance that the original was in FP32 precision, while the converted one is in FP16. The reason you wouldn't see a difference from this is the fact that ComfyUI just converts FP32 to FP16 (or BF16 in some cases) on the fly when loading models, so they likely end up being exactly the same.

Here's the stats for the ones I have around, where you can see exactly this:

Stats for clip-vit-large-patch14.bin:
 >text model:  198 [ 471MB] @ [torch.int64, torch.float32]
 >other keys:  394 [1159MB] @ [torch.int64, torch.float32]

Stats for clip_l.safetensors:
 >text model:  196 [ 234MB] @ [torch.float16]
 >other keys:    0 [   0MB] @ []