An idea regarding the diffusion models

[Jit-Roy]

Hi, i was working on similar idea.
But it was about diffusion models.
https://github.com/Jit-Roy/WeeLLM
It runs large diffusion models with as little as 4 GB of VRAM, without any quantization. It dynamically determines how many layers can fit within the available VRAM and streams the text encoder and transformer layers to the GPU layer by layer, enabling inference on hardware with limited VRAM. It supports both safetensors and GGUF models.

[glitch2027]

Wait this is cool ! But comfyui does the same thing. Isn't it?

[Jit-Roy]

Yeah, ComfyUI can do VRAM offloading, but for me I dont know but somehow most of the models were not running in comfyui without quantization according to my vram constraints. For example I could not run minimx H3 on comfyui, but with my repo I can. And I dont use any quantization.

[halumhalym64-max]

wait you ran Minimax on 4gb? Is this for real?

[Jit-Roy]

wait you ran Minimax on 4gb? Is this for real?

yes!