GGUF Text Encoder support

[city96]

This should allow using the T5 encoder in other precisions than full FP16 and FP8. On a quick test with the Q4_0 unet the results between a BF16 T5 and a Q5_K_M one were almost the same - obviously this needs more testing but preliminary results look promising.

To use it simply replace the text encoder loader with the GGUF variant. You do not need GGUF versions of CLIP, it can load normal models with .safetensors/.bin extensions too. Models can be downloaded from here

Important

You will have to update ComfyUI as well.

[RandomGitUser321]

So far, the results are really good. I'm going try throwing a bunch of prompts with text captions in them to really test it though. Here's my ram usage with q8 versions of both t5 and flux:

It didn't spill into the page file, which is really nice and the prompt encoding was super quick.

[RandomGitUser321]

After doing quite a bit of testing with toy boxes on store shelves(yeah I know, it's a stupid sounding prompt theme, but it works really well), I'm kind of seeing a trend of the q4km and fp8 following very similar paths in scene composition. The q8 will usually still be similar to them, but the little nuances aren't quite the same and it tends to diverge a little in minor details. I don't have the fp16 version of t5 on here right now, but I might redownload it to compare. Have a feeling that q8 and fp16 might follow similar trends with each other, but that would make complete sense. (people who work a lot with LLMs probably know what I mean).

Obviously, my overall sample size is trivial, but I'll keep experimenting and updating any findings. One test I really want to run is with q4 versions of both the t5 and flux. That would open up a really big opportunity for people with weaker hardware.

[city96]

@RandomGitUser321 Thanks for the tests, and do report back on your further findings.

One test I really want to run is with q4 versions of both the t5 and flux

As you can see in the example image, using it with Q4_0 (or preferably Q4_K_M) is at least viable, so this should definitely help run it on weaker hardware.

[RandomGitUser321]

@city96 I'll probably spam this thread a bit over the next day or so with anything I find. And yeah, I see that in your example now, definitely looks pretty good!

So it looks like the GGUFs are all following more in line with fp16, fp8 is the oddball, from what I've seen in a lot of tests. Not a huge oddball or anything, just slightly different on average that I can notice at a quick glance. Like I figured, Q8 and FP16 are extremely close. Here's a goofy example:

and these were the settings I used:

Obviously, it's not going to hold true 100% of the time. A subtle shift in some weight path half way through the t5 can completely change the final output, almost like a seed. So you'll sometimes see convergence or divergence between all the different versions.

Overall, it looks like it's going to be a big win. Thanks for implementing this!