DynamicVRAM breaks loading models

[AnnieTheEagle]

Custom Node Testing

• I have tried disabling custom nodes and the issue persists (see how to disable custom nodes if you need help)

Expected Behavior

Loading models works correctly and prompts are handled correctly

Actual Behavior

An error occurs when trying to load the checkpoint and encode the prompt:

!!! Exception during processing !!! Expected all tensors to be on the same device, but got index is on cuda:0, different from other tensors on cpu (when checking argument in method wrapper_CUDA__index_select)

Steps to Reproduce

Use the following start_comfyui.bat:
.\python_embeded\python.exe -s ComfyUI\main.py --windows-standalone-build --port 19000 --output-directory E:\AI\stable-diffusion\Outputs

OS: Windows 11 Pro (25H2, Build: 26200.7840)
CPU: AMD Ryzen 9 9950X3D
GPU: NVIDIA RTX 5090

The below debug log is from my workflow with custom nodes, however removing the custom nodes and using a simple Debug workflow (attached) also produces the same result

Debug.json

Debug Logs

got prompt
model weight dtype torch.float16, manual cast: None
model_type V_PREDICTION
Using pytorch attention in VAE
Using pytorch attention in VAE
VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
CLIP: [<REDACTED PROMPT>]
Requested to load SDXLClipModel
Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached.
Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached.
Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached.
Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached.
Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached.
!!! Exception during processing !!! Expected all tensors to be on the same device, but got index is on cuda:0, different from other tensors on cpu (when checking argument in method wrapper_CUDA__index_select)
Traceback (most recent call last):
  File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\execution.py", line 524, in execute
    output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                                                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\execution.py", line 333, in get_output_data
    return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\execution.py", line 307, in _async_map_node_over_list
    await process_inputs(input_dict, i)
  File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\execution.py", line 295, in process_inputs
    result = f(**inputs)
             ^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\nodes.py", line 80, in encode
    return (clip.encode_from_tokens_scheduled(tokens), )
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\sd.py", line 313, in encode_from_tokens_scheduled
    pooled_dict = self.encode_from_tokens(tokens, return_pooled=return_pooled, return_dict=True)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\sd.py", line 377, in encode_from_tokens
    o = self.cond_stage_model.encode_token_weights(tokens)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\sdxl_clip.py", line 59, in encode_token_weights
    g_out, g_pooled = self.clip_g.encode_token_weights(token_weight_pairs_g)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\sd1_clip.py", line 45, in encode_token_weights
    o = self.encode(to_encode)
        ^^^^^^^^^^^^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\sd1_clip.py", line 306, in encode
    return self(tokens)
           ^^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\sd1_clip.py", line 266, in forward
    embeds, attention_mask, num_tokens, embeds_info = self.process_tokens(tokens, device)
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\sd1_clip.py", line 213, in process_tokens
    tokens_embed = self.transformer.get_input_embeddings()(tokens_embed, out_dtype=torch.float32)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1776, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1787, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\ops.py", line 552, in forward
    return self.forward_comfy_cast_weights(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\ComfyUI\comfy\ops.py", line 544, in forward_comfy_cast_weights
    x = torch.nn.functional.embedding(input, weight, self.padding_idx, self.max_norm, self.norm_type, self.scale_grad_by_freq, self.sparse).to(dtype=output_dtype)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\AI\stable-diffusion\UI\comfy-ui\python_embeded\Lib\site-packages\torch\nn\functional.py", line 2567, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but got index is on cuda:0, different from other tensors on cpu (when checking argument in method wrapper_CUDA__index_select)

Prompt executed in 3.42 seconds

Other

Full debug logs attached (broken.log). Testing with --disable-dynamic-vram does work, log is also attached (working.log)

working.log
broken.log

[rattus128]

@AnnieTheEagle

I cant reproduce a crash with your json:

CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
Requested to load SDXLClipModel
Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached.
Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached.
Requested to load SDXL
Model SDXL prepared for dynamic VRAM loading. 4897MB Staged. 0 patches attached.
100%|██████████████████████████████████████████████████████████████████████████| 20/20 [00:02<00:00,  8.16it/s]
Requested to load AutoencoderKL
Model AutoencoderKL prepared for dynamic VRAM loading. 159MB Staged. 0 patches attached.
Prompt executed in 5.16 seconds

Your broken.log shows 5x "prepared for dynamic VRAM ..." Lines and you mentioned a custom node, while this stops at 2. Can I get the crash log for the debug.json exactly as it crashes for you to minimize variables?

[AnnieTheEagle]

@AnnieTheEagle

I cant reproduce a crash with your json:

CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
Requested to load SDXLClipModel
Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached.
Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached.
Requested to load SDXL
Model SDXL prepared for dynamic VRAM loading. 4897MB Staged. 0 patches attached.
100%|██████████████████████████████████████████████████████████████████████████| 20/20 [00:02<00:00,  8.16it/s]
Requested to load AutoencoderKL
Model AutoencoderKL prepared for dynamic VRAM loading. 159MB Staged. 0 patches attached.
Prompt executed in 5.16 seconds

Your broken.log shows 5x "prepared for dynamic VRAM ..." Lines and you mentioned a custom node, while this stops at 2. Can I get the crash log for the debug.json exactly as it crashes for you to minimize variables?

Okay strangely enough, I restarted everything, and now the Debug.json workflow does work (although you can see from the screenshot on the original issue that is from the exact same workflow.

So I suppose this must be an issue with WildcardEncode from ComfyUI-Inspire-Pack. I can take it up with them, unless you have any suggestions?

[rattus128]

@AnnieTheEagle
I cant reproduce a crash with your json:

CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
Requested to load SDXLClipModel
Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached.
Model SDXLClipModel prepared for dynamic VRAM loading. 1560MB Staged. 0 patches attached.
Requested to load SDXL
Model SDXL prepared for dynamic VRAM loading. 4897MB Staged. 0 patches attached.
100%|██████████████████████████████████████████████████████████████████████████| 20/20 [00:02<00:00,  8.16it/s]
Requested to load AutoencoderKL
Model AutoencoderKL prepared for dynamic VRAM loading. 159MB Staged. 0 patches attached.
Prompt executed in 5.16 seconds

Your broken.log shows 5x "prepared for dynamic VRAM ..." Lines and you mentioned a custom node, while this stops at 2. Can I get the crash log for the debug.json exactly as it crashes for you to minimize variables?

Okay strangely enough, I restarted everything, and now the Debug.json workflow does work (although you can see from the screenshot on the original issue that is from the exact same workflow.

So I suppose this must be an issue with WildcardEncode from ComfyUI-Inspire-Pack. I can take it up with them, unless you have any suggestions?

Ill take a very quick look myself if you can give me the workflow that repros.

[rattus128]

Also is it possible that in your screenshot run you didnt restart comfyui? I notice in your logs you restart which is good, but if this bug is corrupting the cached model from WildcardEncode it might just by sticky accross a clip node change.

[AnnieTheEagle]

Also is it possible that in your screenshot run you didnt restart comfyui? I notice in your logs you restart which is good, but if this bug is corrupting the cached model from WildcardEncode it might just by sticky accross a clip node change.

Here's the workflow that repros, freshly restarted. Log also attached

txt2img.json

log.txt

[rattus128]

I have debugged this a little.

The root cause is in WildcardEncode in that it will for some reason run the layer on the CPU. This is fine iff that usage stays consistent, however, if the model is reused by a different encoder that expects the clip to run on the GPU, the vbar cache hit hits a CPU tensor and causes this.

The workflow from @AnnieTheEagle contains a PC Encode Prompt node after the WildcardEncode.

The core can protect against this by protecting that cache against a CPU weight.

[rattus128]

The minimum reproducer:

@AnnieTheEagle are you able to make the downstream bug report against WildcardEncode?