GPU Benchmark Flux DEV fp8 5090 4090 3090

[metamountain]

Lets do a benchmark for flux dev fp8.

Just use the workflow template. Post the time of second or third render , no loading ....

[metamountain]

Nvidia 3090: 26s

[simonlui]

Intel A770 on Fedora Linux. Using Pytorch Nightly.

100%|███████████████████████████████████████████████████████████████████████| 20/20 [00:52<00:00,  2.61s/it]
Prompt executed in 53.26 seconds

If I go back to Pytorch Extension for Pytorch v2.3.110+xpu

100%|███████████████████████████████████████████████████████████████████████| 20/20 [00:46<00:00,  2.33s/it]
Prompt executed in 47.13 seconds

Edit: ComfyUI options used was --fp8_e4m3fn-unet --bf16-vae --bf16-text-enc --lowvram --async-offload --mmap-torch-files --reserve-vram 0.2 with --async-offload omitted for testing Pytorch v2.3.110+xpu.

[eric-rolph]

5090 8s

[eric-rolph]

5090 5.46s with some ComfyUI optimization, but left the workflow template unchanged

[CaptailTyler]

Just wondering what do you mean by optimizations did you do in ComfyUI? I didn't know that you can do any optimizations outside of the workflow? Thanks

[Rogala]

Total VRAM 16311 MB, total RAM 32599 MB
pytorch version: 2.8.0+cu128
xformers version: 0.0.32+635a8ab2.d20250804
Enabled fp16 accumulation.
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 5060 Ti : cudaMallocAsync
Using sage attention (2.2.0)
Python version: 3.12.10 (tags/v3.12.10:0cc8128, Apr 8 2025, 12:21:36) [MSC v.1943 64 bit (AMD64)]
ComfyUI version: 0.3.48
ComfyUI frontend version: 1.23.4

*.bat --use-sage-attention --fast

loaded completely 13400.370919891357 11350.067443847656 True
100%|██████████████████████████████████████████████| 20/20 [00:23<00:00, 1.20s/it]
Requested to load AutoencodingEngine
loaded completely 176.43157958984375 159.87335777282715 True
Prompt executed in 72.58 seconds
loaded completely 13240.49756211853 11350.067443847656 True
100%|██████████████████████████████████████████████| 20/20 [00:23<00:00, 1.20s/it]
Prompt executed in 25.71 seconds
got prompt
loaded completely 13240.49756211853 11350.067443847656 True
100%|██████████████████████████████████████████████| 20/20 [00:23<00:00, 1.20s/it]
Prompt executed in 25.89 seconds

[CaptailTyler]

Inno3D RTX 5090 X3 OC

Distributor ID: Ubuntu
Description: Ubuntu 25.04
Release: 25.04
Codename: plucky

Getting 8.78s at 2.38it/s for 3 runs.

[metamountain]

Now on 4090. 11.3s...

100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:10<00:00, 1.85it/s]
Prompt executed in 11.28 seconds

Total VRAM 24564 MB, total RAM 64630 MB
pytorch version: 2.9.1+cu128
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync
Using async weight offloading with 2 streams
Enabled pinned memory 29083.0
working around nvidia conv3d memory bug.
Using sage attention
Python version: 3.13.6 (tags/v3.13.6:4e66535, Aug 6 2025, 14:36:00) [MSC v.1944 64 bit (AMD64)]
ComfyUI version: 0.6.0

[esp-dev]

RTX 5080 (without OC, W11)
Total VRAM 16303 MB, total RAM 130621 MB
pytorch version: 2.10.0+cu130
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 5080 : cudaMallocAsync

This workflow is not optimal (Prompt executed in 16.71 seconds):

got prompt
model weight dtype torch.float8_e4m3fn, manual cast: torch.bfloat16
model_type FLUX
Using pytorch attention in VAE
Using pytorch attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
Requested to load FluxClipModel_
loaded completely; 13314.67 MB usable, 4777.54 MB loaded, full load: True
Requested to load Flux
loaded completely; 13118.07 MB usable, 11350.07 MB loaded, full load: True
100%|██████████| 20/20 [00:15<00:00, 1.33it/s]
Requested to load AutoencodingEngine
Unloaded partially: 1888.77 MB freed, 9461.34 MB remains loaded, 135.04 MB buffer reserved, lowvram patches: 0
loaded completely; 316.54 MB usable, 159.87 MB loaded, full load: True
Prompt executed in 29.81 seconds
loaded completely; 12958.19 MB usable, 11350.07 MB loaded, full load: True
100%|██████████| 20/20 [00:15<00:00, 1.33it/s]
Unloaded partially: 1888.77 MB freed, 9461.34 MB remains loaded, 135.04 MB buffer reserved, lowvram patches: 0
Prompt executed in 16.70 seconds

Change it to (Prompt executed in 10.94 seconds):

got prompt
Using pytorch attention in VAE
Using pytorch attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
clip missing: ['text_projection.weight']
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
Requested to load FluxClipModel_
loaded completely; 13314.67 MB usable, 9319.23 MB loaded, full load: True
Found quantization metadata version 1
Detected mixed precision quantization
Using mixed precision operations
model weight dtype torch.bfloat16, manual cast: torch.bfloat16
model_type FLUX
Requested to load Flux
loaded completely; 13118.07 MB usable, 11756.14 MB loaded, full load: True
100%|██████████| 20/20 [00:09<00:00, 2.21it/s]
Requested to load AutoencodingEngine
Unloaded partially: 2319.55 MB freed, 9436.63 MB remains loaded, 144.05 MB buffer reserved, lowvram patches: 0
loaded completely; 341.25 MB usable, 159.87 MB loaded, full load: True
Prompt executed in 21.88 seconds
Unloaded partially: 60.85 MB freed, 99.02 MB remains loaded, 13.50 MB buffer reserved, lowvram patches: 0
loaded completely; 13019.05 MB usable, 11756.14 MB loaded, full load: True
100%|██████████| 20/20 [00:09<00:00, 2.21it/s]
Unloaded partially: 2292.54 MB freed, 9463.64 MB remains loaded, 144.05 MB buffer reserved, lowvram patches: 0
loaded completely; 314.24 MB usable, 159.87 MB loaded, full load: True
Prompt executed in 10.84 seconds

Or even change it to (Prompt executed in 6.66 seconds):

got prompt
Found quantization metadata version 1
Detected mixed precision quantization
Using mixed precision operations
model weight dtype torch.bfloat16, manual cast: torch.bfloat16
model_type FLUX
Requested to load Flux
loaded completely; 12958.19 MB usable, 8763.64 MB loaded, full load: True
100%|██████████| 20/20 [00:06<00:00, 3.23it/s]
Prompt executed in 10.92 seconds
100%|██████████| 20/20 [00:06<00:00, 3.23it/s]
Prompt executed in 6.67 seconds