[Research] Successful Load of Z-Image Turbo ControlNet (Flux) - Tensor Mapping & Geometry

[nikosleft]

Detailed Description

I managed to get the quantized Z-ControlNet (Z-Image Turbo) loading on the Vulkan backend by manually patching control.hpp. I'm sharing the tensor geometry I discovered in case it helps implement proper dynamic support:

Architecture: It uses 2-1 Embedder mapping (64 -> 3840).

Modulation: It uses the Z-Image 4x scheme (Input 256 -> Output 15360), not the standard Flux 6x.

Layers: 6 Control, 2 Refiner.

Bias: Must be explicitly disabled (false) for all Linears or it fails to find tensors.

Alternatives you considered

1. Using the Standard VERSION_FLUX Loader:
   I initially attempted to map the tensors to the existing Flux definition. This failed because standard Flux ControlNets expect a Modulation/AdaLN layer output of 6 * hidden_size (23040), whereas this Z-Image variant uses a compressed 4 * hidden_size (15360) scheme.

2. Using Img2Img Only:
   I considered bypassing ControlNet and relying solely on high-strength Image-to-Image. While this worked for general composition, it lacked the precise structural guidance required, necessitating a working ControlNet implementation.

3. Python/Torch Inference (ComfyUI/Forge):
   I evaluated running the .safetensors version via Python backends. However, given my hardware constraints (AMD 5700 XT, 8GB VRAM), the overhead was too high. The stable-diffusion.cpp Vulkan backend was the only viable path for acceptable inference speeds with this specific architecture.

Additional context

This works on an AMD 5700 XT (8GB) with Vulkan.

Update

I managed to load it by patching control.hpp, but the inference crashes or OOMs on 8GB cards because the Vulkan backend seems to dequantize the ControlNet to FP16, blowing up VRAM. We need a way to keep ControlNet quantized during inference.