Skip to content

v2.6.8 - Qwen Image ControlNet weight-supply contract fix (ComfyUI v0.44.0)

Latest

Choose a tag to compare

@ussoewwin ussoewwin released this 06 Nov 10:28
· 1 commit to main since this release
EN 中文

1. What changed, and why

1.1 Background: ComfyUI v0.44.0 supplies weights through a cast contract

Since ComfyUI v0.44.0 (dynamic VRAM / aimdo enabled by default), layer weights are no longer guaranteed to sit on the compute device when a module's forward runs. Weights stay in their file form (CPU / virtual-memory backed) until the module takes part in the core weight-supply contract:

# comfy/ops.py (v0.44.0, disable_weight_init.Linear.forward)
def forward(self, *args, **kwargs):
    run_every_op()
    if self.comfy_cast_weights or len(self.weight_function) > 0 or len(self.bias_function) > 0:
        return self.forward_comfy_cast_weights(*args, **kwargs)   # contract path
    else:
        return super().forward(*args, **kwargs)                    # reads self.weight directly

The contract path (forward_comfy_cast_weights -> CastBiasWeightContext -> cast_bias_weight) is what moves the weight to the input device: vbar-managed layers go through comfy/ops.py:396-414, plain tensors through comfy/ops.py:417-462. A module that never opts in reads self.weight directly and gets whatever device the weight happens to be on.

Modules created and loaded through a ModelPatcher are armed by the core. The problem sits outside that path.

1.2 The symptom

Qwen Image ControlNet does not call the diffusion model through the wrapper's forward. It reaches into the base model directly (comfy/ldm/qwen_image/controlnet.py:151-162):

hidden_states = base_model.img_in(hidden_states)
encoder_hidden_states = base_model.txt_norm(context)
encoder_hidden_states = base_model.txt_in(encoder_hidden_states)
temb = base_model.time_text_embed(timesteps, hidden_states)

base_model here is this package's ComfyQwenImageWrapper, whose __getattr__ forwards unknown attributes to the inner nunchaku model. The borrowed img_in is a plain disable_weight_init.Linear with comfy_cast_weights = False - it never joined the contract. The failure looks like this:

File "comfy/samplers.py", line 330, in _calc_cond_batch
    c['control'] = control.get_control(input_x, timestep_, c, len(cond_or_uncond), transformer_options)
File "comfy/controlnet.py", line 320, in get_control
    control = self.control_model(x=x_noisy.to(dtype), hint=self.cond_hint, ...)
File "comfy/ldm/qwen_image/controlnet.py", line 151, in forward
    hidden_states = base_model.img_in(hidden_states)
File "comfy/ops.py", line 641, in forward
    return super().forward(*args, **kwargs)
RuntimeError: Expected all tensors to be on the same device, but got mat1 is on cuda:0,
different from other tensors on cpu (when checking argument in method wrapper_CUDA_addmm)

1.3 Why the second run fails while the first succeeds

Measured on the affected workflow (10 consecutive runs, ComfyUI v0.44.0, torch 2.15.0+cu134):

Run ControlNet loader re-executed Result Wall time
1 yes OK 106.06 s
2 no (node cache) FAIL 31.56 s
3 yes OK 68.24 s
4-10 no (node cache) FAIL 5.5 - 9.5 s
  • When the loader re-executes, the ControlNet state dict is re-read (Found quantization metadata version 1) and the weights reach the GPU through the load path before sampling.
  • When ComfyUI's node-output cache returns the cached CONTROL_NET / MODEL objects, the loader does not run again. The workflow's end-of-run purge nodes have already unloaded the models, so the cached objects carry weights that were dropped back to CPU / file form - and the borrowed modules still read them directly.

This is a lifecycle gap: cache reuse + purge + a direct attribute borrow that sits outside the core's contract. It only became fatal when v0.44.0 changed the default weight placement.

2. The fix: opt the borrowed modules into the core contract

wrappers/qwenimage.py - ComfyQwenImageWrapper.__init__ now calls one static helper:

# wrappers/qwenimage.py (v2.6.8)
_CONTROLNET_BORROWED_MODULES = (
    "img_in",
    "txt_in",
    "txt_norm",
    "time_text_embed",
    "pe_embedder",
)

@staticmethod
def _enable_weight_supply_contract(model):
    """Opt the modules ControlNet borrows into ComfyUI's cast contract."""
    for attribute in _CONTROLNET_BORROWED_MODULES:
        root = getattr(model, attribute, None)
        if root is None:
            continue
        seen = set()
        stack = [root]
        while stack:
            module = stack.pop()
            if id(module) in seen:
                continue
            seen.add(id(module))
            if hasattr(module, "comfy_cast_weights") and not module.comfy_cast_weights:
                module.comfy_cast_weights = True
            stack.extend(module.children())

# ComfyQwenImageWrapper.__init__, last line:
self._enable_weight_supply_contract(model)

Properties:

  • Scoped: only the five attributes ControlNet actually accesses on base_model (img_in, txt_in, txt_norm, time_text_embed, pe_embedder) and their children are touched; the rest of the model stays exactly as constructed (verified below).
  • Idempotent: modules already opted in are skipped.
  • Follows the core instead of fighting it: dynamic VRAM stays enabled - no launcher flag, no ComfyUI-core edit, no nunchaku edit, no workflow edit. Whenever the core moves a weight, cast_bias_weight supplies it to the input device on every forward, including the ControlNet path.

3. Repository state after this release

Path Change Note
wrappers/qwenimage.py +38 / -0 ComfyQwenImageWrapper opts the borrowed modules into the contract
ComfyUI core / nunchaku package / workflows unchanged no file outside this package is modified
RELEASE_NOTES/changelog.md / zhmd/changelogzh.md changelog v2.6.8 entries (16 insertions, 2 deletions combined)
pyproject.toml / __init__.py version 2.6.7 -> 2.6.8

4. What was verified

  • Contract-off reproduction: with the weight on CPU and comfy_cast_weights = False, F.linear raises the exact RuntimeError shown above (python_embeded, torch 2.15.0+cu134, cuda:0).
  • Contract-on: the same layer returns a CUDA result; against an identity weight the output equals the input (maxdiff 0.0).
  • Scope: img_in / txt_in flip False -> True; a sibling Linear outside the five borrowed attributes stays False.
  • Deployed copy: byte-identical to the repo file (SHA256 20061EBC58DBFCC1BDCAF33EC8628F6AA86F7339B5229BDEBBF9328F55903262), py_compile clean.
  • Real workflow: repeated executions of the affected i2i workflow (ControlNet + Nunchaku Qwen Image + LoRA stack + EasyCache) complete normally on ComfyUI v0.44.0 - no RuntimeError, no fallback-latent output, an image produced on every run.

5. Action required

  • Update this package to v2.6.8 (ComfyUI Manager / git pull) and restart ComfyUI.
  • No workflow edits, no ComfyUI-core changes, no nunchaku changes, no launcher flags are needed.

6. Notes

  • The five attributes in _CONTROLNET_BORROWED_MODULES are the module attributes Qwen Image ControlNet accesses on base_model today (comfy/ldm/qwen_image/controlnet.py:143-162; process_img is a method call and needs no contract). If upstream adds more borrowed attributes, add them to the same tuple.
  • The workflow's purge nodes and ComfyUI's node-output cache are left untouched; with the contract in place they no longer matter for this path.

7. Links