on 3090 win10 i get full vram fill even tho i use q5 that is 12gb

[1blackbar]

Flux Q8 12 gb GGUF loads fast but this one Q5 that is also 12GB takes forever to load and EACH TIME i generate image.And i get this that runs for like 1 minute and then starts generating -

Falling back to numpy dequant for qtype: 21
Falling back to numpy dequant for qtype: 21
Falling back to numpy dequant for qtype: 21

From what i noticed - Q2 and Q3 GGUF versions of Llama cause fallback but Q6 is just too slow
My speed with Q4 llama and Q2 t5 (the only ones that i got loaded fast ) is this for 1024x1024 18steps
got prompt
Requested to load HiDreamTEModel_
loaded completely 8356.927012252807 7869.5537109375 True
100%|██████████████████████████████████████████████████████████████████████████████████| 18/18 [00:59<00:00, 3.32s/it]
Prompt executed in 75.36 seconds

[city96]

The IQ quants are only supported via a very slow fallback. You could try go for Q4_K_S or Q3_K_M instead (basically anything that just has "Q" instead of "IQ" in the name).

The initial run being slow could be because the model only gets loaded on first use, so if you swap models it'll reload from disk again (depending on how much system ram you have, it might stay cached).

[1blackbar]

which ones run partially on gpu ? Also is it possible to modify GGUF clip loader and let some run on GPU and some on CPU ? That would be best scenario.
https://huggingface.co/QuantFactory/Meta-Llama-3-8B-GGUF/tree/main

[city96]

Not sure what the default comfy logic is when loading the text encoders, though even with lowvram they'd all end up swapping to/from the GPU unless you force all to run on CPU with another node.

There is this node pack which has more advanced memory management, though I don't think the quad loader has been added yet. You could try to put in a feature request for selecting offload targets per text encoder for the dual/triple/quadruple CLIP loader nodes, doing it with the base memory management would be hard since the current logic for the load/offload device treats all 4 text encoders as just one model.

[1blackbar]

yeah , so that is about 15gb for text encoders alone.... i understand that hidream has some gemini like capabilities with built in llama but this is not made for consumers... and 3090 is now low vram.
I wonder if we can swap that llama with qwen 1.5b