Support for Flux

[extremeheat]

New diffusion model - https://blackforestlabs.ai/announcing-black-forest-labs/

Reference implementation: comfyanonymous/ComfyUI@1589b58

[65a]

There's a reference diffusers config in a PR as well: https://huggingface.co/black-forest-labs/FLUX.1-dev/discussions/3/files

[stduhpf]

That's probably the best open model so far, but it's pretty big. I'd love to be able to use it quantized and with partial GPU offload.

[Green-Sky]

The "schnell" distillation seems like a good candidate though.

That's probably the best open model so far, but it's pretty big. I'd love to be able to use it quantized and with partial GPU offload.

[diimdeep]

12B is massive, lower than f16 quants could become more popular, f8 or even q5
(upd: corrected)

[extremeheat]

No, 12B parameters at fp32 is 12 * 4 bytes = 48GB memory. Not including clip/t5/vae/etc.
fp16 ~= 12 * 2 = 24GB memory
8 bit quant ~= 12 * 1= 12GB
4 bit quant ~= 12 * 0.5 = 6GB
5 bit ~= 12 * (5/8) = 7.5GB

[red-scorp]

That's probably the best open model so far, but it's pretty big. I'd love to be able to use it quantized and with partial GPU offload.

Can SD.cpp partially offload a model to GPU? I was unable to do this. Can you give a hint hot to do this?

• One more upvote for FLUX support.

[red-scorp]

No, 12B parameters at fp32 is 12 * 4 bytes = 48GB memory. Not including clip/t5/vae/etc. fp16 ~= 12 * 2 = 24GB memory 8 bit quant ~= 12 * 1= 12GB 4 bit quant ~= 12 * 0.5 = 6GB 5 bit ~= 12 * (5/8) = 7.5GB

Can image generation models be quantized down to 4-5 bits? I saw it done to LLMs with mixed results, but never saw it working with SD and Co.

[SkutteOleg]

Can image generation models be quantized down to 4-5 bits? I saw it done to LLMs with mixed results, but never saw it working with SD and Co.

stable-diffusion.cpp can quantize models using --type command line argument

[DGdev91]

The "schnell" distillation seems like a good candidate though.

That's probably the best open model so far, but it's pretty big. I'd love to be able to use it quantized and with partial GPU offload.

Different stuff. It's more like the LCM models on SD, it takes less steps to make a picture, with minor quality loss over the standard version.

12B is massive, lower than f16 quants could become more popular, f8 or even q5 (upd: corrected)

There's already an unofficial FP8 version wich works on ComfyUI https://huggingface.co/Kijai/flux-fp8

But i don't know if comfy is able to do interference on more aggressive quantizations

Anyway.... +1 for flux support. Or for any new big image model.

As those model become bigger and bigger, running a quantized version could be the only way to run them on consumer-grade GPUs, and stable-diffusion.cpp could become a good solution.

[Amin456789]

indeed flux is amazing, after working with it i wanna use it with cpp too if possible

@FSSRepo if i remmeber correctly u were working on q2 and k variant, please work on it if u can, we can quantize it to something like 3-4 gb maybe
@leejet please support for lora for quantized models if possible, we really need for flux so we dont need to download and quantize new models all the time

[teddybear082]

this repo has q4 flux schnell: https://huggingface.co/city96/FLUX.1-schnell-gguf/tree/main. Agree would be great to get support, my computer is not even close to being able to run non quantized

[PierreSnell-Appox]

There are quantized files already (GGUF) claimed to have been made with stable-diffusion.cpp.
The smallest is only 4Gb but does not seems to work (gguf_init_from_file: tensor 'double_blocks.0.img_attn.norm.key_norm.scale' of type 10 (q2_K) number of elements (128) is not a multiple of block size (256))
The Q4_0 model (second smallest) is not working as well get sd version from file failed: '../../Downloads/flux1-schnell-Q4_0.gguf'

https://huggingface.co/aifoundry-org/FLUX.1-schnell-Quantized/tree/main
https://huggingface.co/city96/FLUX.1-dev-gguf/tree/main

However, the ability to have a small REST/API like Llama.cpp would be amazing to host thins kind of models !

[DGdev91]

Indeed, in the last weeks lots of developers made several attempts for using a quantized version of flux in both ComfyUI and StableDiffusion webui Forge.
Currently the most popular methods seems to be fp8 and nf4, but i've seen mamy experiments with gguf too.

However, the ability to have a small REST/API like Llama.cpp would be amazing to host thins kind of models !

Both ComfyUI and StableDiffusion Webui (well, of course Forge too) already have an API, and except from some plugin wich try to integrate them with program like Photoshop or Krita, i didn't saw many projects using them.

It can still be an interesting feature, but i don't think it should be the priority.
Also, it can be developed as a separate project.

[stduhpf]

The GGUF experiments aren't using proper ggml though. They are just using GGUF as a way to compress the weights, and they are dequantizing on the fly during inference, which is very inefficient.