Severe performance degradation over consecutive runs after upgrading to V0.18 (ksampler slows from 1.79s/t to 35s/t)

[glolionline]

Custom Node Testing

• I have tried disabling custom nodes and the issue persists (see how to disable custom nodes if you need help)

Expected Behavior

Performance should remain stable across consecutive runs without progressive slowdown.

Actual Behavior

After upgrading ComfyUI to V0.18, a previously stable workflow exhibits severe performance degradation. The ksampler step time (s/t) increases linearly with each consecutive generation, causing overall runtime to become progressively slower.
Total execution time increased from 30.71 seconds(The first run includes first loading time) to 168.86 seconds over consecutive runs(same input image).
Note that lowvram patches count accumulates over runs: 0 → 41 → 58 → 75 → 76 → 93 → 110 → 127
The log below is from a basic Flux Klein 9B distilled editing workflow. Previously, on V0.17, I used the --disable-dynamic-vram argument because a separate Qwen edit workflow with GGUF CLIP would throw errors without it. This Klein 9B workflow ran normally under that setup before. Workflow run time was around 12-15 seconds.

Steps to Reproduce

workflow is pretty simple, a flux klein 9b distilled edit workflow
add disable dynamic vram to comfy.settings.json:
"Comfy.Server.LaunchArgs": {
"preview-method": "latent2rgb",
"disable-dynamic-vram": ""
},

Debug Logs

got prompt
Using pytorch attention in VAE
Using pytorch attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
Requested to load AutoencoderKL
loaded completely; 6971.51 MB usable, 160.31 MB loaded, full load: True
Found quantization metadata version 1
Using MixedPrecisionOps for text encoder
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
Requested to load Flux2TEModel_
loaded completely; 9441.49 MB usable, 8263.34 MB loaded, full load: True
Found quantization metadata version 1
Detected mixed precision quantization
Using mixed precision operations
model weight dtype torch.bfloat16, manual cast: torch.bfloat16
model_type FLUX
Requested to load Flux2
loaded partially; 7240.42 MB usable, 6864.00 MB loaded, 2132.02 MB offloaded, 624.00 MB buffer reserved, lowvram patches: 41
100%|██████████| 4/4 [00:07<00:00,  1.79s/it]
Requested to load AutoencoderKL
Unloaded partially: 976.00 MB freed, 5888.00 MB remains loaded, 624.00 MB buffer reserved, lowvram patches: 58
loaded completely; 200.41 MB usable, 160.31 MB loaded, full load: True
Prompt executed in 30.71 seconds
got prompt
loaded partially; 7269.41 MB usable, 6912.00 MB loaded, 2084.02 MB offloaded, 624.00 MB buffer reserved, lowvram patches: 0
100%|██████████| 4/4 [00:23<00:00,  5.91s/it]
Requested to load AutoencoderKL
Unloaded partially: 960.00 MB freed, 5952.00 MB remains loaded, 624.00 MB buffer reserved, lowvram patches: 75
loaded completely; 180.02 MB usable, 160.31 MB loaded, full load: True
Prompt executed in 34.69 seconds
got prompt
Unloaded partially: 64.00 MB freed, 5888.00 MB remains loaded, 624.00 MB buffer reserved, lowvram patches: 76
loaded partially; 7237.42 MB usable, 6864.00 MB loaded, 2132.02 MB offloaded, 624.00 MB buffer reserved, lowvram patches: 0
100%|██████████| 4/4 [00:19<00:00,  4.76s/it]
Requested to load AutoencoderKL
Unloaded partially: 976.00 MB freed, 5888.00 MB remains loaded, 624.00 MB buffer reserved, lowvram patches: 93
loaded completely; 200.41 MB usable, 160.31 MB loaded, full load: True
Prompt executed in 28.98 seconds
got prompt
loaded partially; 7237.42 MB usable, 6864.00 MB loaded, 2132.02 MB offloaded, 624.00 MB buffer reserved, lowvram patches: 0
100%|██████████| 4/4 [01:07<00:00, 16.95s/it]
Requested to load AutoencoderKL
Unloaded partially: 976.00 MB freed, 5888.00 MB remains loaded, 624.00 MB buffer reserved, lowvram patches: 110
loaded completely; 200.41 MB usable, 160.31 MB loaded, full load: True
Prompt executed in 81.61 seconds
got prompt
loaded partially; 7237.42 MB usable, 6864.00 MB loaded, 2132.02 MB offloaded, 624.00 MB buffer reserved, lowvram patches: 0
100%|██████████| 4/4 [02:20<00:00, 35.04s/it]
Requested to load AutoencoderKL
Unloaded partially: 976.00 MB freed, 5888.00 MB remains loaded, 624.00 MB buffer reserved, lowvram patches: 127
loaded completely; 200.41 MB usable, 160.31 MB loaded, full load: True
Prompt executed in 168.86 seconds

Other

No response

[rattus128]

Im still taking any dynamic VRAM errors. Are you able to run that qwen edit and dump it as a fresh report on v0.18? GGUF CLIP is supported insofar as I expect it to work in the fallback mode.

Curious if dyn vram helps you in this new case although, that lowvram patch counter instability on the partial unloader is weird and suggests something is up.

We do need at least your system specs to complete the picture. Can you post the comfy startup log up to "to see the GUI ...".

[rattus128]

Also can you bypass your Lora and retest and let us know? The outputs will be trash but the performance will tell us something.

[glolionline]

Im still taking any dynamic VRAM errors. Are you able to run that qwen edit and dump it as a fresh report on v0.18? GGUF CLIP is supported insofar as I expect it to work in the fallback mode.

The attached with_dynamic_vram.txt shows the Qwen edit workflow with a GGUF CLIP loader (failed) and a normal CLIP loader (succeeded).
disable_dynamic_vram.txt shows the Flux Klein 9B simple edit workflow (workflow image also attached) without LoRA. This time, the lowvram patch count did not increase as in the previous test, but the processing speed still slowed down progressively.
Both logs (with dynamic VRAM disabled/enabled) include the startup log from the beginning.
System specifications are also attached.

disable_dynamic_vram.txt
with_dynamic_vram.txt

[rattus128]

This looks like two independent issues. Thanks for the detailed reports. Ill start with just the qwen stuff.

I got some qwen GGUF, both TE and model and cant reproduce although admittedly I do not have the same load messages as you so there might be some sensistivity to the very exact GGUF model. I dont see any relation to dynamic VRAM though, the problem with those qwen gguf just dissappears with disable dynamic VRAM on latest?

The error message indicates a reasonably scary conflict in your custom node packs. You have both the ComfyUI-GGUF and calcius GGUF node packs installed and the backtrace shows a mix and match between the assets of the two. If you can un-install the calcius and retry the error message should at least change to something a bit more predictable and if we are lucky that one is a fix.

I will have a look at your flux stuff in a bit.

[glolionline]

This looks like two independent issues. Thanks for the detailed reports. Ill start with just the qwen stuff.

I got some qwen GGUF, both TE and model and cant reproduce although admittedly I do not have the same load messages as you so there might be some sensistivity to the very exact GGUF model. I dont see any relation to dynamic VRAM though, the problem with those qwen gguf just dissappears with disable dynamic VRAM on latest?

The error message indicates a reasonably scary conflict in your custom node packs. You have both the ComfyUI-GGUF and calcius GGUF node packs installed and the backtrace shows a mix and match between the assets of the two. If you can un-install the calcius and retry the error message should at least change to something a bit more predictable and if we are lucky that one is a fix.

I will have a look at your flux stuff in a bit.

Thanks for your time. Here is the situation now:

qwen edit:
I disabled the calcius gguf node pack. The qwen edit workflow can now work with dynamic vram enabled.
I thought it was caused by dynamic vram because it worked with "disable-dynamic-vram", but not with dynamic vram enabled,the dynamic vram is the only variable.
So now the issue should be opened on calcius's gguf repo.

flux klein 9b:
Since on my comfyui, the qwen workflow can work with dynamic vram now, and the klein 9b workflow works fine in that mode too,
technically there is no more issue for my setup.
But the klein workflow still behaves strangely with "disable-dynamic-vram" mode. Still, I think it's worth mentioning.

Here is the klein 9b workflow log with "disable-dynamic-vram":

got prompt
Using pytorch attention in VAE
Using pytorch attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
Requested to load AutoencoderKL
loaded completely; 7018.10 MB usable, 160.31 MB loaded, full load: True
Found quantization metadata version 1
Using MixedPrecisionOps for text encoder
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
Requested to load Flux2TEModel_
loaded completely; 9443.49 MB usable, 8263.34 MB loaded, full load: True
Found quantization metadata version 1
Detected mixed precision quantization
Using mixed precision operations
model weight dtype torch.bfloat16, manual cast: torch.bfloat16
model_type FLUX
Requested to load Flux2
loaded partially; 7282.55 MB usable, 6912.00 MB loaded, 2084.02 MB offloaded, 624.00 MB buffer reserved, lowvram patches: 40
100%|██████████| 4/4 [00:06<00:00, 1.60s/it]
Requested to load AutoencoderKL
Unloaded partially: 960.00 MB freed, 5952.00 MB remains loaded, 624.00 MB buffer reserved, lowvram patches: 57
loaded completely; 195.84 MB usable, 160.31 MB loaded, full load: True
Prompt executed in 27.06 seconds
got prompt
loaded partially; 7281.55 MB usable, 6912.00 MB loaded, 2084.02 MB offloaded, 624.00 MB buffer reserved, lowvram patches: 0
50%|█████ | 2/4 [00:10<00:09, 4.87s/it]FETCH ComfyRegistry Data [DONE]
[ComfyUI-Manager] default cache updated: https://api.comfy.org/nodes
FETCH DATA from: C:\ComfyUI\user__manager\cache\1514988643_custom-node-list.json [DONE]
[ComfyUI-Manager] All startup tasks have been completed.
100%|██████████| 4/4 [00:24<00:00, 6.21s/it]
Requested to load AutoencoderKL
Unloaded partially: 960.00 MB freed, 5952.00 MB remains loaded, 624.00 MB buffer reserved, lowvram patches: 74
loaded completely; 195.84 MB usable, 160.31 MB loaded, full load: True
Prompt executed in 33.53 seconds
got prompt
loaded partially; 7281.55 MB usable, 6912.00 MB loaded, 2084.02 MB offloaded, 624.00 MB buffer reserved, lowvram patches: 0
100%|██████████| 4/4 [00:32<00:00, 8.13s/it]
Requested to load AutoencoderKL
Unloaded partially: 960.00 MB freed, 5952.00 MB remains loaded, 624.00 MB buffer reserved, lowvram patches: 91
loaded completely; 195.84 MB usable, 160.31 MB loaded, full load: True
Prompt executed in 41.82 seconds
got prompt
loaded partially; 7281.55 MB usable, 6912.00 MB loaded, 2084.02 MB offloaded, 624.00 MB buffer reserved, lowvram patches: 0
100%|██████████| 4/4 [00:25<00:00, 6.29s/it]
Requested to load AutoencoderKL
Unloaded partially: 960.00 MB freed, 5952.00 MB remains loaded, 624.00 MB buffer reserved, lowvram patches: 108
loaded completely; 195.84 MB usable, 160.31 MB loaded, full load: True
Prompt executed in 31.94 seconds
got prompt
loaded partially; 7281.55 MB usable, 6912.00 MB loaded, 2084.02 MB offloaded, 624.00 MB buffer reserved, lowvram patches: 0
100%|██████████| 4/4 [00:24<00:00, 6.16s/it]
Requested to load AutoencoderKL
Unloaded partially: 960.00 MB freed, 5952.00 MB remains loaded, 624.00 MB buffer reserved, lowvram patches: 125
loaded completely; 195.84 MB usable, 160.31 MB loaded, full load: True
Prompt executed in 31.40 seconds
got prompt
loaded partially; 7281.55 MB usable, 6912.00 MB loaded, 2084.02 MB offloaded, 624.00 MB buffer reserved, lowvram patches: 0
100%|██████████| 4/4 [01:22<00:00, 20.56s/it]
Requested to load AutoencoderKL
Unloaded partially: 960.00 MB freed, 5952.00 MB remains loaded, 624.00 MB buffer reserved, lowvram patches: 142
loaded completely; 195.84 MB usable, 160.31 MB loaded, full load: True
Prompt executed in 95.84 seconds

Compared to enabling dynamic vram, the same klein workflow runs repeatedly on the same picture:

got prompt
Using pytorch attention in VAE
Using pytorch attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
Requested to load AutoencoderKL
Model AutoencoderKL prepared for dynamic VRAM loading. 160MB Staged. 0 patches attached.
Model AutoencoderKL prepared for dynamic VRAM loading. 160MB Staged. 0 patches attached.
Model AutoencoderKL prepared for dynamic VRAM loading. 160MB Staged. 0 patches attached.
Found quantization metadata version 1
Using MixedPrecisionOps for text encoder
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
Requested to load Flux2TEModel_
Model Flux2TEModel_ prepared for dynamic VRAM loading. 8262MB Staged. 0 patches attached. Force pre-loaded 145 weights: 602 KB.
Found quantization metadata version 1
Detected mixed precision quantization
Using mixed precision operations
model weight dtype torch.bfloat16, manual cast: torch.bfloat16
model_type FLUX
Requested to load Flux2
0 models unloaded.
Model Flux2 prepared for dynamic VRAM loading. 8996MB Staged. 112 patches attached.
100%|██████████| 4/4 [00:06<00:00, 1.57s/it]
0 models unloaded.
Model AutoencoderKL prepared for dynamic VRAM loading. 160MB Staged. 0 patches attached.
Prompt executed in 25.27 seconds
got prompt
Model Flux2 prepared for dynamic VRAM loading. 8996MB Staged. 112 patches attached.
100%|██████████| 4/4 [00:05<00:00, 1.41s/it]
0 models unloaded.
Model AutoencoderKL prepared for dynamic VRAM loading. 160MB Staged. 0 patches attached.
Prompt executed in 9.31 seconds
got prompt
Model Flux2 prepared for dynamic VRAM loading. 8996MB Staged. 112 patches attached.
100%|██████████| 4/4 [00:05<00:00, 1.37s/it]
0 models unloaded.
Model AutoencoderKL prepared for dynamic VRAM loading. 160MB Staged. 0 patches attached.
Prompt executed in 8.74 seconds
got prompt
Model Flux2 prepared for dynamic VRAM loading. 8996MB Staged. 112 patches attached.
100%|██████████| 4/4 [00:05<00:00, 1.44s/it]
0 models unloaded.
Model AutoencoderKL prepared for dynamic VRAM loading. 160MB Staged. 0 patches attached.
Prompt executed in 9.07 seconds