Skip to content

v2.4.4 - Soft Purge CUDA Residue + HSWQ Full Surface / ZI Peel\

Choose a tag to compare

View all tags
@ussoewwin ussoewwin released this 03 Aug 09:25
· 23 commits to main since this release
v2.4.4
df333e8
EN 中文

Overview

v2.4.4 hardens soft VRAM reclaim (HSWQ toggle OFF) so MultiGPU Dynamic / NVFP4 stacks (e.g. Krea2) do not leave ~9–12 GB CUDA resident after model_unload()==True, while keeping CPU CLIP / text-encoder reload sources intact after Ollama-adjacent purges. It also expands HSWQ Method 2c / nuclear surfaces after v2.4.3 (full INT8/NVFP4/Detailer residual clear, Z Image ConvRot parity / bake peel, Hadamard / inplace pools). Package Registry version remains 2.4.3 until a separate bump is ordered; this tag documents the tip at 6190024 (includes zhmd/v2.4.4.md).

Canonical long guide (now tracked on main):

1. Soft path — MultiGPU / NVFP4 CUDA residue (HSWQ OFF)

1.1 Symptom A

  1. Load / run a heavy MultiGPU Dynamic or HSWQ NVFP4 model (Krea2-class).
  2. Run DisTorch Purge VRAM V2 with HSWQ unchecked (soft path only).
  3. model_unload() may return success, but NVIDIA-SMI / Comfy still shows ~9–12 GB tied to leftover CUDA param/buffer storage.
  4. Next load OOMs or stalls because soft purge never requested a real free.

1.2 Root cause

ComfyUI’s free_memory(0, device) is effectively a no-op for reclaim. Soft unload left CUDA .data on nn.Parameter / buffers after model_unload(). Memory Manager / Safe Memory Manager previously called free_memory(0, ...) the same way.

1.3 Countermeasure

On soft purge (purge_models / aggressive unload):

  1. Mark unused → partially_unload / model_unload.
  2. Unwrap to nn.Module → force-empty leftover CUDA storage only (_force_empty_cuda_storage).
  3. Pop from current_loaded_models → cleanup_models().
  4. unload_all_models() + free_memory(1e30) on every cuda:i.
  5. Optional soft_empty_cache().

Same 1e30 request is used in Memory Manager / Safe Memory Manager when reset_virtual_memory is on.

1.4 Soft-path excerpt (nodes/purge_vram.py / root purge_vram.py)

def _force_empty_cuda_storage(t) -> int:
    # NVFP4 / MultiGPU Dynamic: free leftover CUDA only.
    # Never wipe CPU tensors — after model_unload() they are
    # ComfyUI's reload source. Wiping to empty(0) made CLIP
    # Embedding.weight non-2D (Ollama purge → CLIPTextEncode
    # RuntimeError: 'weight' must be 2-D; reload logged 0.00 MB).
    ...
    if not is_cuda:
        return 0
    dtype = getattr(data, "dtype", torch.float32)
    empty = torch.empty(0, dtype=dtype, device="cpu")
    if hasattr(t, "data"):
        t.data = empty
    ...

# Hard free: unload_all + free_memory(1e30). free_memory(0) does nothing.
mm.unload_all_models()
for di in range(torch.cuda.device_count()):
    mm.free_memory(1e30, torch.device(f"cuda:{di}"))

1.5 Memory Manager excerpt (memory_manager.py)

# free_memory(0, ...) is a no-op and left Krea2 NVFP4 ~9GB resident.
for di in range(torch.cuda.device_count()):
    comfy.model_management.free_memory(1e30, torch.device(f"cuda:{di}"))

1.6 Expected soft-path log lines

Unloaded N model(s)
Force-killed ~X.XX GB CUDA storage from loaded models (MultiGPU/NVFP4 soft-unload residue)
unload_all_models() issued
free_memory(1e30) issued for all CUDA devices

2. Soft / HSWQ nuclear — CUDA-only empty(0) (CLIP / TE protection)

2.1 Symptom B

  1. Soft or HSWQ purge after workflows that also touch Ollama / large TE.
  2. Next CLIPTextEncode / Z Image TE encode fails:
RuntimeError: 'weight' must be 2-D
  1. Reload logs show 0.00 MB for CLIP / TE — CPU embedding storage was wiped to empty(0).

2.2 Root cause

An earlier “kill all leftover storage” pass called torch.empty(0) on CPU tensors as well as CUDA. After model_unload(), CPU weights are ComfyUI’s reload source. Wiping them left non-2D Embedding.weight.

2.3 Countermeasure

Both soft _force_empty_cuda_storage and HSWQ _kill_tensor_storage return early unless data is CUDA. Pinned CUDA host memory may still be unregistered; CPU-only tensors are left alone.

def _kill_tensor_storage(t) -> int:
    # Same rule as _force_empty_cuda_storage: unpin/CPU-safe only;
    # empty(0) only for CUDA. CPU wipe broke ZI TE after Ollama purge.
    ...
    if not is_cuda:
        return 0

3. HSWQ ON — Method 2c / nuclear expansions (after v2.4.3)

v2.4.3 fixed NVFP4 runtime pools (clear_nvfp4_runtime_pools / PyCapsule on second gen). v2.4.4 keeps that and adds surfaces that still poisoned the next model when HSWQ purge was armed.

3.1 Symptom C — Z Image → SDXL / product-stack poison

After Z Image NVFP4 ConvRot / LoRA bake, HSWQ purge cleared kitchen / some pools, but parity / bake hooks and INT8-protect load arms could remain on sys.modules / comfy.ops. Next SDXL load inherited ZI TC product stack or _hswq_int8_protect_* arms (Params.convrot=False / wrong VER bake).

3.2 Countermeasure — peel APIs inside Method 2c

Scan sys.modules for NVFP4 / Z Image modules and call when present:

API Role
_clear_zimage_parity_contamination_for_sdxl Drop ZI parity contamination before SDXL
restore_nvfp4_tc_product_stack Restore product TC stack
uninstall_zimage_nvfp4_lora_bake Remove ZI LoRA bake hooks

Also peels ZI INT8-protect load wrapping on ops._load_quantized_module (closure cells / PRODUCT wrap) so SDXL INT8 ConvRot is not re-armed incorrectly.

HSWQ INT8/NVFP4: HSWQ stack peel restore_nvfp4_tc_product_stack via ... -> ...
HSWQ INT8/NVFP4: HSWQ stack peel uninstall_zimage_nvfp4_lora_bake via ... -> ...

3.3 Symptom D — INT8 / Detailer / Hadamard / inplace residue

Second gen after HSWQ purge still hit residual:

  • Full INT8 / NVFP4 module bags beyond kitchen workspaces alone
  • Detailer / SEGS / PinCache leftovers (Method 0 family)
  • Hadamard / inplace CUDA pools held across prompts

3.4 Countermeasure

Method 0–2c remain toggle-gated only (no auto-arm when HSWQ is OFF). With HSWQ ON, Method 2c / nuclear path expands clears for:

  • Kitchen _cublas_workspaces / _empty_cuda_tensors (pre-existing)
  • NVFP4 runtime pools / CUDA graphs (v2.4.3)
  • Broader INT8/NVFP4/Detailer residual dicts and Hadamard / inplace pools
  • ZI peel APIs above

Log prefix remains: HSWQ INT8/NVFP4:.

4. Files touched (this release tip)

Path Change
purge_vram.py / nodes/purge_vram.py Soft CUDA force-empty + free_memory(1e30); CUDA-only kill; Method 2c peel / pool expansions
memory_manager.py / nodes/memory_manager.py free_memory(1e30) for virtual-memory reset
changelog/changelog.md v2.4.4 entry
md/* English purge / SA guides; tracked on main (.gitignore no longer ignores md/)
.gitignore Ignore narrowed to root /release_notes_*.md so md/RELEASE_NOTES_* can be tracked

Import preference unchanged: prefer DisTorchPurgeVRAMV2 from nodes/purge_vram.py (fallback root module kept in sync).

5. What this release does not change

  • HSWQ Methods 0–2c do not run when the HSWQ toggle is OFF (soft path only).
  • Ollama toggle behavior from v2.4.2 is unchanged in intent (separate toggle).
  • Comfy Registry / pyproject.toml version is not bumped in this documentation tag; Manager still reports the last published package version until an explicit version order.

6. Verification checklist

  1. Soft purge after MultiGPU/NVFP4: log shows force-killed GB + free_memory(1e30); VRAM drops ~9–12 GB class residue.
  2. Soft/HSWQ purge then CLIP / ZI TE: no 'weight' must be 2-D; reload sizes non-zero.
  3. HSWQ ON after ZI NVFP4 then SDXL: peel log lines present; no inherited ZI bake / protect arm on SDXL load.
  4. HSWQ ON NVFP4 second gen: still no pooled TC path failed / PyCapsule (v2.4.3 + expanded 2c).
  5. HSWQ OFF: no HSWQ INT8/NVFP4: Method 2c spam.
Assets 2
Loading