Native NVFP4 (Blackwell) Loading Failure on RTX 5090 - Wan 2.2/Flux2Dev/LTX2 Workflow

[freddiefiasco]

Custom Node Testing

• I have tried disabling custom nodes and the issue persists (see how to disable custom nodes if you need help)

Expected Behavior

Expected Behavior When loading the Wan_2.2_nvfp4_mixed checkpoint on an RTX 5090 with comfy-kitchen (CUDA backend) active, the Load Diffusion Model node should natively map the 4-bit weights to the TensorCoreNVFP4Layout without intermediate upcasting.

The expected VRAM footprint for the 14B model should be approximately 7.5 GB (reflecting 4-bit quantization). The console should confirm a direct load into the NVFP4 container, enabling the RTX 5090’s native FP4 Tensor Core acceleration, rather than falling back to torch.float16 (28GB) or torch.float8_e4m3fn (14GB) via a safety cast.

Actual Behavior

Actual Behavior When attempting to load the Wan_2.2_nvfp4_mixed or others e.g. ltx2, flux2dev nvfp4 checkpoint, the Load Diffusion Model node fails to preserve the native 4-bit quantization, resulting in one of two failure modes:

Default Configuration: When weight_dtype is set to default, the loader defaults to torch.float16. This triggers a massive ~28 GB memory allocation (standard 14B size), causing an immediate torch.OutOfMemoryError as it exceeds the available VRAM on a single RTX 5090 instance when loaded alongside the T5 encoder.

FP8 Configuration: When weight_dtype is set to fp8_e4m3fn, the system forcibly upcasts the 4-bit weights to 8-bit. The console log reports loaded completely; 14745.55 MB loaded, confirming an FP8 load (14GB) rather than the expected NVFP4 load (~7GB).

In all successful load attempts, the logs show manual cast: torch.float16 or similar casting warnings, indicating that the comfy-kitchen NVFP4 backend was bypassed. The model never enters the optimized TensorCoreNVFP4Layout, forcing the system to rely on slower, memory-intensive standard operations.

Steps to Reproduce

Steps to Reproduce

Environment Setup: Configure a Windows environment with an NVIDIA RTX 5090 (Blackwell), CUDA 13.0, and PyTorch Nightly (2.9.1+cu130). Ensure comfy-kitchen is installed and the CUDA backend is initialized (verified via startup logs).

Clean Installation: Use a standard ComfyUI installation with minimal custom nodes (removing ComfyUI-MultiGPU or other patchers to isolate the issue).

Workflow Construction:

Add a standard Load Diffusion Model node.

Select the Wan_2.2_nvfp4_mixed.safetensors checkpoint.

Test Case A (Default Behavior):

Set the loader's weight_dtype to default.

Queue a prompt.

Result: The console logs a manual cast: torch.float16 warning. Memory allocation spikes to ~28GB, triggering a torch.OutOfMemoryError on a single GPU instance (when loaded with T5).

Test Case B (Forced FP8):

Set the loader's weight_dtype to fp8_e4m3fn.

Queue a prompt.

Result: The console logs loaded completely with a size of ~14,700 MB. This indicates the 4-bit weights were read but upcasted to 8-bit to satisfy the dtype request, rather than utilizing the 7GB NVFP4 layout.

Debug Logs

[ComfyUI-Manager] All startup tasks have been completed.
got prompt
...
Requested to load WanTEModel
loaded completely; 6419.48 MB loaded, full load: True
Requested to load WanVAE
loaded completely; 392.62 MB loaded, full load: True
Found quantization metadata version 1
Detected mixed precision quantization
Using mixed precision operations
model weight dtype torch.float8_e4m3fn, manual cast: torch.float16  <-- CRITICAL FAILURE
model_type FLOW
Using sage attention mode: sageattn3
Requested to load WAN21
0 models unloaded.
loaded completely; 14745.55 MB loaded, full load: True  <-- PROOF OF UPCAST (Should be ~7500 MB)

File "\custom_nodes\ComfyUI_bnb_nf4_fp4_Loaders\__init__.py", line 12, in functional_linear_4bits
    out = bnb.matmul_4bit(x, weight.t(), bias=bias, quant_state=weight.quant_state)
File "site-packages\bitsandbytes\autograd\_functions.py", line 376, in matmul_4bit
    assert quant_state is not None
AssertionError

model weight dtype torch.float16, manual cast: torch.float16
model_type FLOW
Requested to load WAN21
Allocated memory 54580 MiB
torch.OutOfMemoryError: Allocation on device

Other

Detailed Environment Specification

System Hardware

GPU: 2x NVIDIA GeForce RTX 5090 (Blackwell SM_120)

CPU: AMD Ryzen Threadripper 9960X 24-Cores

RAM: ~128 GB (Total System RAM)

OS: Windows 11 (Platform Release 11)

Core Software Stack

NVIDIA Driver: 591.74

CUDA Toolkit: 13.0 (Targeted via PyTorch cu130 build)

Python: 3.12.10 (tags/v3.12.10:0cc8128) [MSC v.1943 64 bit (AMD64)]

PyTorch: 2.9.1+cu130 (Nightly Build)

ComfyUI Versioning

Backend Version: 0.9.1

Frontend Version: 1.38.1

Comfy-Kitchen Backend: cuda (Initialized: {'available': True, ... 'capabilities': ['quantize_nvfp4', 'scaled_mm_nvfp4']})

Critical Libraries & Dependencies

SageAttention: 2.2.0 (Custom wheel: sageattention-2.2.0+cu130.torch2.11-cp311... equivalent build for Py3.12)

BitsAndBytes: 0.49.1 (Verified via Manager log refusing downgrade)

Crystools: 1.27.4

Launch Arguments

Flags: --listen --normalvram (Tested with --gpu-only and --highvram with identical failures)

Environment Overrides:

CUDA_VISIBLE_DEVICES=0 (To isolate single GPU behavior)

COMFY_KITCHEN_BACKEND=cuda (To force non-eager execution)

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

Model File Under Test

Checkpoint: Wan_2.2_nvfp4_mixed.safetensors (GitMylo / Wan-AI)

Target Format: NVFP4 / E2M1 (Expected Load Size: ~7.5 GB)

[comfyanonymous]

I'm not aware of an nvfp4 wan checkpoint that works with our quant system. Try something like https://huggingface.co/black-forest-labs/FLUX.2-dev-NVFP4/tree/main

[freddiefiasco]

Unfortunately, this upcast error applies to all nvfp4 models with the latest comfyui update. including ltx2 and flux2dev even z-image turbo nvfp4

It's so apparent, that I can easily run ltx2 1080p 81 frames locally using the fp8 model (~27gb)
yet, when I switch to the fp4 model (~20gb) I get OOM errors

This seems to be a result of comfy upcasting the nvfp4 model to bf16

[freddiefiasco]

@comfyanonymous can you replicate the issue on your end? I've tried on several rtx 5090 machines running recent / latest comfy backends and they all produce the same result

[comfyanonymous]

No I can't reproduce this on my setup. Did you try a clean install without any custom nodes?

[freddiefiasco]

@comfyanonymous I will try now, what python, cuda, comfy etc etc version are you using?

[comfyanonymous]

Just download the normal portable package it comes with pytorch 2.9.1 cu130.

[freddiefiasco]

Just downloaded the latest normal portable package and ran it on a fresh windows 11 machine with an rtx 5090 on the latest drivers and all the same issues appeared. Appeared on both LTX2 nvfp4 and Flux2dev nvfp4

as mentioned, the ltx2-fp8 quant uses less vram than the ltx2-nvfp4 quant. Due to the consistent upcasting error

see images for details

[freddiefiasco]

@comfyanonymous any word back on this. I'm getting OOM on flux2dev nvfp4 on an rtx 5090, fp8 works fine. Something very buggy going on.

[jorismak-infi]

I'm trying making WAN nvfp4mixed quants, and Qwen nvfp4mixed quants... they load fine (quality issues though). An existing wan2.2-i2v from huggingface also loads fine.

The supplied z-image / flux1 nvfp4 versions work, as well as flux2klein 9b. (Didn't try 4b). LTX2 also works, but I load the kijai 'split' versions, not the big checkpoint.

Anyway, windows 11 pytoch 2.9.1cu130, rtx 5060ti 16gb. It's not the portable version though, I created a python venv, installed pytorch and comfyui manually in it.

But maybe it's driver related or something else that's conflicting? I can at least confirm that ComfyUI doesn't 'upcast' nvfp4 always.

[freddiefiasco]

@jorismak-infi glad to hear it's working on your end.

Do you notice a significant reduction in vram usage? I'd love to see your logs if possible.

[guymcguymcguy]

Popping in to say I seem to be having the same/a very similar issue, no matter what node and configuration I use the NVFP4 models get forced to either Float16 or FP8 depending on the exact settings, proper pytorch and sageattention versions are installed, been going through similar troubleshooting steps with no luck. Even a custom node meant to force FP4 results in it running on Float16. Hope a solution gets found soon!

[Auswolf-debug]

@jorismak-infi glad to hear it's working on your end.

Do you notice a significant reduction in vram usage? I'd love to see your logs if possible.

I can confirm that according to my logs that, it does manual cast: torch.float16. But it still loads the quant version while loading.
And i am 100% sure, simply because i can do a full load of all the models.

My enviroment
Total VRAM 16311 MB, total RAM 32532 MB
pytorch version: 2.10.0+cu130
xformers version: 0.0.34
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 5060 Ti : cudaMallocAsync
Using async weight offloading with 2 streams
Enabled pinned memory 14639.0
working around nvidia conv3d memory bug.
Using sage attention
Python version: 3.14.3 (tags/v3.14.3:323c59a, Feb 3 2026, 16:04:56) [MSC v.1944 64 bit (AMD64)]
ComfyUI version: 0.12.3
ComfyUI frontend version: 1.38.13

And of course a picture of my logs featuring the worlds fastest inference.

And again this is 1280*704 @ 8 steps.

I have made ti2v and FC nvfp4 5B_multistep models, hopefully i finish the other 2 and get them up on huggingface today.

[insecure-erasure]

I'm not a Python developer so please take this with a grain of salt, but while looking into this I noticed that unet_manual_cast in comfy/model_management.py has explicit early-return guards for float16 and bfloat16, but not for FP8. I might be missing something, but, when weight_dtype is torch.float8_e4m3fn the function would fall through to the loop and return float16.

Would something like this make sense?

def unet_manual_cast(weight_dtype, inference_device, supported_dtypes=[torch.float16, torch.bfloat16, torch.float32]):
[...]
    if weight_dtype in FLOAT8_TYPES and supports_fp8_compute(inference_device):
        return None

Meanwhile, unet_dtype seems to handle FP8 explicitly:

def unet_dtype(device=None, model_params=0, supported_dtypes=[torch.float16, torch.bfloat16, torch.float32], weight_dtype=None):
[...]
    fp8_dtype = None
    if weight_dtype in FLOAT8_TYPES:
        fp8_dtype = weight_dtype

    if fp8_dtype is not None:
        if supports_fp8_compute(device): #if fp8 compute is supported the casting is most likely not expensive
            return fp8_dtype

        free_model_memory = maximum_vram_for_weights(device)
        if model_params * 2 > free_model_memory:
            return fp8_dtype
[...]

On my end (RTX 5060, SM120, CUDA 13.1, PyTorch 2.10) logs show model weight dtype torch.float8_e4m3fn, manual cast: torch.float16, too.

I'm not sure about how to interpret the log message about the manual cast: I can't tell whether it means the model is actually being stored in float16 in VRAM, or whether it just means that compute operations will run in float16 while the weights remain in other format. I also don't understand the role comfy-kitchen plays in this decision.

Not sure if this is helpful, but I'll try to test this crackpot theory locally anyway and see what happens.

[insecure-erasure]

I've been looking into this a bit more and there's something that might be relevant. I think at least part of the problem is in the models themselves, not in ComfyUI:

• Prefix mismatch in quantization metadata: if the prefixes in _quantization_metadata don't match the keys in the state dict, MixedPrecisionOps fails to load and the weights end up as plain float16, in a way that some users might miss.

• Tooling fragmentation: there are quite a few conversion tools out there and the quality varies a lot, and people seem to be generating models without really understanding what each option does.

• Missing activation calibration: without it, dynamic scale computation adds memory overhead on every diffusion step.

From what I've seen, silveroxides' convert_to_quant covers both (--actcal for calibration, --comfy_quant for the prefix issue), though people don't seem to use them. I haven't had time to try this myself though.

Anyway, hope this is useful to someone.