[Quantization] SDNQ Minimax H3 loading

[asomoza]

The H3 transformer casts activations to a projection's weight dtype in seven places.

With SDNQ, the weight is stored as int8/uint8 or int4/uint4 and dequantized inside forward, so the cast turns the activations into integers:

RuntimeError: mat1 and mat2 must have the same dtype, but got Char and BFloat16

The same issue happens with layerwise casting, where the weight is stored as fp8 and upcast by a hook.

Code to test (this fails without this PR):

import torch

from diffusers import ModularPipeline
from diffusers.hooks import apply_group_offloading
from diffusers.utils import encode_video


MODEL = "OzzyGT/MiniMax_H3_sdnq_dynamic_4bit"
PROMPT = (
    "A red fox trotting through a snowy pine forest at dawn, snow crunching underfoot, "
    "breath visible in the cold air, low winter sun through the trees"
)

pipe = ModularPipeline.from_pretrained(MODEL)
pipe.load_components(workflow="t2va", dtype=torch.bfloat16)

offload = {
    "onload_device": torch.device("cuda"),
    "offload_device": torch.device("cpu"),
    "use_stream": True,
    "low_cpu_mem_usage": True,
}
pipe.transformer.enable_group_offload(offload_type="block_level", num_blocks_per_group=1, **offload)
apply_group_offloading(pipe.text_encoder.model, offload_type="leaf_level", **offload)
apply_group_offloading(pipe.vae, offload_type="leaf_level", **offload)
pipe.audio_vae.to("cuda")

state = pipe(
    prompt=PROMPT,
    num_frames=124,
    height=544,
    width=960,
    num_inference_steps=30,
    generator=torch.Generator("cpu").manual_seed(42),
)

encode_video(
    state.get("videos")[0],
    fps=24,
    audio=state.get("audio")[0],
    audio_sample_rate=state.get("sampling_rate"),
    output_path="h3_768p_group_offload.mp4",
)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.