Skip to content

v3.4.4: ConvRot INT8 ControlNet Loader & Full Drop-in Compatibility

Choose a tag to compare

View all tags
@ussoewwin ussoewwin released this 26 Aug 07:55
· 50 commits to main since this release
v3.4.4
1156f00
EN 中文

1. Overview

v3.4.4 introduces the HSWQ Load ConvRot INT8 ControlNet Model (HSWQLoadConvRotINT8ControlNet) node.

This node enables loading and executing ConvRot / TensorWise INT8-quantized ControlNet checkpoints (e.g., Qwen Image Fun ControlNet) directly in ComfyUI. Weights are maintained in 8-bit precision (QuantizedTensor / TensorWiseINT8Layout) in VRAM, and forward execution utilizes comfy_kitchen's high-speed int8_linear GEMM kernel with online activation rotation (convrot).

Additionally, the node includes automatic format detection and fallback mechanisms, providing 100% backwards compatibility with conventional FP16 / BF16 / FP8 ControlNet models as a full drop-in replacement for ComfyUI's stock "Load ControlNet Model" node.

2. Root Cause Analysis: Stock ComfyUI ControlNet Loader Limitations

When loading INT8-quantized checkpoints via stock ComfyUI (controlnet_load_state_dict), two critical structural failures occur:

Stage Stock ComfyUI Behavior Root Problem & Consequence
Module Graph Construction Sets architecture dtype via unet_dtype = weight_dtype(sd) For INT8 checkpoints, weight_dtype(sd) returns torch.int8. Instantiating PyTorch module graphs with torch.int8 fails immediately: RuntimeError: Only Tensors of floating point and complex dtype can require gradients.
Quant-Aware Ops Dispatch ControlNet loading path lacks model_config.quant_config and never passes custom_operations Even if module initialization is forced to a float type, the *.comfy_quant and *.weight_scale tensors in the checkpoint are ignored, preventing quantized tensor attachment.

3. Architecture & Implementation

HSWQLoadConvRotINT8ControlNet addresses both issues through the following architecture:

[ safetensors Checkpoint ]
           │
           ▼
[_has_int8_comfy_quant] ──(No INT8 metadata)──► [ Stock load_controlnet_state_dict ] (FP16/FP8 Pass-through)
           │ (INT8 detected)
           ▼
[ model_options Construction ]
  ├── dtype = torch.bfloat16  (Prevents PyTorch INT8 gradient initialization crash)
  └── custom_operations = _int8_mixed_precision_ops() (Injects MixedPrecisionOps)
           │
           ▼
[ comfy.controlnet.load_controlnet_state_dict ]
           │
           ▼ (Consumes *.comfy_quant & *.weight_scale)
[ VRAM: TensorWiseINT8Layout (8-bit) ]
           │
           ▼ (Inference)
[ comfy_kitchen.int8_linear + ConvRot Online Activation Rotation ]
           │
           ▼
[ Output: Standard CONTROL_NET ] ──► [ Apply ControlNet Node ]

Core Components

  1. Automatic Metadata Detection (_has_int8_comfy_quant):

    • Inspects *.comfy_quant keys in the state dict and decodes the JSON payload.
    • Triggers the INT8 quantization loader path if format == "int8_tensorwise"; otherwise falls back seamlessly to standard loading.

  2. Explicit MixedPrecisionOps Construction (_int8_mixed_precision_ops):

    • Instantiates a mixed_precision_ops class configured with QUANT_ALGOS["int8_tensorwise"].
    • During _load_from_state_dict, every Linear layer attaches a QuantizedTensor (TensorWiseINT8Layout).

  3. Float Graph Instantiation (dtype = torch.bfloat16):

    • Forces the initial module graph to construct in torch.bfloat16, cleanly bypassing PyTorch INT8 parameter initialization errors before attaching quantized weights.

  4. Standard ComfyUI Compatibility:

    • Returns a standard CONTROL_NET object, seamlessly interoperating with standard Apply ControlNet nodes and existing workflows.

4. Feature Matrix

Feature Stock Load ControlNet Model HSWQ Load ConvRot INT8 ControlNet
ConvRot INT8 ControlNet Support ❌ (Crashes on init) ✅ (Native INT8 execution)
VRAM Footprint N/A (Cannot load) 8-bit (TensorWiseINT8Layout)
Execution Kernel — comfy_kitchen.int8_linear + ConvRot online act rotation
Conventional FP16 / BF16 ControlNet ✅ ✅ (Automatic fallback)
FP8 ControlNet ✅ ✅ (Automatic fallback)
Output Type CONTROL_NET CONTROL_NET (100% compatible)

5. Added & Modified Files

Type File Description
Added nodes/hswq_load_convrot_int8_controlnet.py Implementation of HSWQLoadConvRotINT8ControlNet
Added png/convrot_int8_controlnet.png Node workflow screenshot
Added zhmd/v3.4.4.md Chinese Release Notes
Modified __init__.py Node registration & bump version to 3.4.4
Modified pyproject.toml Bump package version to 3.4.4
Modified changelog.md / zhmd/CHANGELOG.md Version 3.4.4 changelog entries
Modified README.md / zhmd/README.md Documentation and compatibility guide

6. Links

Assets 2
Loading