QWEN Inference completely run using RAM instead of VRAM

[JasperG]

Haoming02,
First off, thank you for your ongoing work on this project. Much appreciated.

I appear to find myself in a situation where any QWEN inference is run using system RAM instead of VRAM.
Regardless of model or text-encoder selected, VRAM hovers around 3 to 5 gb of usage, and everything appears loaded into system RAM.

Given RTX4090 24GB and 128GB of system RAM;
Running neo, git pull 2cd1657 ( updated 08:35 Friday, 9 January 2026 zulu time );

Clean install of neo, default config;
No x-formers or attention optimizations. Have tried with and without --fast-fp16.
GPU weights in webui set to 22515 MB

Tried most recent nvidia studio driver and nvidia game ready driver, currently on Studio Driver 591.74

webui-user.bat contains --cuda-malloc --api --fast-fp16 --onnxruntime-gpu

python: 3.11.14  •  torch: 2.9.1+cu130  •  gradio: 4.40.0

Exhaustively tested on these models (filenames, hashes on request);

• qwen_image_fp8_e4m3fn.safetensors
• qwen_image_edit_2509_fp8_e4m3fn.safetensors
• qwen_image_2512_bf16.safetensors
• Qwen-Rapid-AIO-NSFW-v14.safetensors

And text-encoders (filenames, hashes on request);

• qwen_2.5_vl_7b_fp8_scaled.safetensors
• qwen_2.5_vl_7b.safetensors

Output:

venv "E:\Stable Diffusion\Software\sd-webui-forge-neo\venv\Scripts\Python.exe"
Python 3.11.14 (main, Oct 31 2025, 22:57:10) [MSC v.1944 64 bit (AMD64)]
Version: neo
CUDA 13.0
Launching Web UI with arguments: --uv --cuda-malloc --api --ckpt-dir 'E:\Stable Diffusion\Models' --lora-dir 'E:\Stable Diffusion\LoRA' --vae-dir 'E:\Stable Diffusion\VAE' --fast-fp16 --onnxruntime-gpu
Using cudaMallocAsync backend.
Total VRAM 24564 MB, total RAM 130839 MB
pytorch version: 2.9.1+cu130
allow_fp16_accumulation: True
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync
VAE dtype preferences: [torch.bfloat16, torch.float32] -> torch.bfloat16
CUDA Using Stream: False
Using PyTorch Cross Attention
Using PyTorch Attention for VAE
ControlNet preprocessor location: E:\Stable Diffusion\Software\sd-webui-forge-neo\models\ControlNetPreprocessor
[-] ADetailer initialized. version: 25.3.0, num models: 10
09:41:29 - ReActor - STATUS - Running v0.7.1-b3 on Device: CUDA
[ControlNet] - INFO - ControlNet UI callback registered.
Model selected: {'checkpoint_info': {'filename': 'E:\Stable Diffusion\Models\Qwen\qwen_image_2512_bf16.safetensors', 'hash': 'bcfd55a8'}, 'additional_modules': ['E:\Stable Diffusion\Software\sd-webui-forge-neo\models\text_encoder\qwen_2.5_vl_7b_fp8_scaled.safetensors', 'E:\Stable Diffusion\VAE\qwen_image_vae.safetensors'], 'unet_storage_dtype': None}
Using online LoRAs in FP16: True
Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch().
Startup time: 19.8s (prepare environment: 3.5s, launcher: 0.2s, forge init: 4.3s, bypass basicConfig: 1.4s, shared init: 0.1s, misc. imports: 1.1s, load scripts: 2.7s, create ui: 2.9s, gradio launch: 1.3s, add APIs: 2.1s).
Loading Model: {'checkpoint_info': {'filename': 'E:\Stable Diffusion\Models\Qwen\qwen_image_2512_bf16.safetensors', 'hash': 'bcfd55a8'}, 'additional_modules': ['E:\Stable Diffusion\Software\sd-webui-forge-neo\models\text_encoder\qwen_2.5_vl_7b_fp8_scaled.safetensors', 'E:\Stable Diffusion\VAE\qwen_image_vae.safetensors'], 'unet_storage_dtype': None}
[Unload] Trying to free all memory for cuda:0 with 0 models keep loaded ... Done.
[Unload] Trying to free all memory for cpu with 0 models keep loaded ... Done.
StateDict Keys: {'transformer': 1933, 'vae': 194, 'text_encoder': 1446, 'ignore': 0}
Using Detected Qwen2.5 Data Type: torch.float8_e4m3fn
K-Model Created: {'storage_dtype': torch.bfloat16, 'computation_dtype': torch.bfloat16}
Model loaded in 1.1s (unload existing model: 0.4s, forge model load: 0.8s).
Skipping unconditional conditioning when CFG = 1. Negative Prompts are ignored.
[Unload] Trying to free 12356.23 MB for cuda:0 with 0 models keep loaded ... Done.
[Memory Management] Target: JointTextEncoder, Free GPU: 22970.47 MB, Model Require: 8947.53 MB, Previously Loaded: 0.00 MB, Inference Require: 1619.20 MB, Remaining: 12403.74 MB, Moving model(s) has taken 5.57 seconds
[Unload] Trying to free 56270.60 MB for cuda:0 with 0 models keep loaded ... Unload model JointTextEncoder Done.
[Memory Management] Target: KModel, Free GPU: 22887.81 MB, Model Require: 38967.90 MB, Previously Loaded: 0.00 MB, Inference Require: 1619.20 MB, Remaining: -17699.29 MB, CPU Swap Loaded (blocked method): 38957.32 MB, GPU Loaded: 10.58 MB
Moving model(s) has taken 2.22 seconds
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [02:00<00:00, 6.03s/it]
[Unload] Trying to free 5421.68 MB for cuda:0 with 0 models keep loaded ... Done.██████| 20/20 [01:39<00:00, 3.48s/it]
[Memory Management] Target: WanVAE, Free GPU: 22854.12 MB, Model Require: 242.03 MB, Previously Loaded: 0.00 MB, Inference Require: 1619.20 MB, Remaining: 20992.89 MB, Moving model(s) has taken 0.03 seconds
Total progress: 100%|██████████████████████████████████████████████████████████████████| 20/20 [01:39<00:00, 4.98s/it]

Please forgive me, in case this is user error.

Any assistance warmly welcomed.

[Haoming02]

I am working on rewriting the entire memory management right now

So just wait for that (or test it if you wish to), since the generation does finish, right?

[JasperG]

Yes sir, it does finish, it just takes a very long time in comparison to some previous pull I have been using in the past (unknown which one).

Thank you for your attention. I will await progress.

[Haoming02]

In the meantime,

GPU weights in webui set to 22515 MB

Try set it to 12288 MB

Exhaustively tested on these models

I recommend using the Nunchaku version instead

[JasperG]

I will promptly try again with weights set to 12288.

With all due respect; Excuse my naivity when asking: Would using a nunchaku version (to my understanding a quantized model), not defeat the purpose of having built a high-end machine? I would love to exchange time spent diffusing for higher quality outputs. (Having said this, I shall try nunchaku none-the-less).
Thanks again.

[Haoming02]

Would using a nunchaku version not defeat the purpose of having built a high-end machine?

I mean, even a RTX 5090 (32 GB) cannot load the official Qwen-Image to begin with (40GB)

So might as well use the version with quality comparable to fp8, but significantly faster

[JasperG]

Closing - Expect issue will be resolved in the future.

Thanks.