Skip to content

v1.5.5

Latest

Choose a tag to compare

@DefTruth DefTruth released this 18 Nov 03:22
· 42 commits to main since this release
8f9b28c

🚀 Cache-DiT v1.5.5 Release Notes

Release Date: 2026-11-18
Comparison: v1.5.3...v1.5.5 (187 commits)
Full Changelog: GitHub Releases

📋 Overview

Cache-DiT v1.5.5 spans a month (2026-10-15 ~ 2026-11-18) and is organized around five threads: model support (25 new pipeline families, 🤗Diffusers coverage up to 0.43.0), the kernel and quantization stack (SVDQuant CuTe/NVFP4 with fused MLP, native FP8 and FP4 on CUTLASS collectives, fused memory-bound block operators, one _C_ops extension with no backend-selection layer), the distributed refactor (communicate / parallel split), SVD-Cache, and the CLI / package layout cleanup (_utils → cli).

✨ Core Highlights

1. 🧩 Model Support: 🤗Diffusers up to 0.43.0 and Built-in Pipelines

Model coverage now spans 🤗Diffusers releases from 0.12.0 to 0.43.0 (see the pipeline matrix in the README), plus built-in pipelines for models that have no 🤗Diffusers implementation. Every entry was ported with the full capability matrix — Cache, CP/TP, quantization and offload — and validated end to end.

Modality Model Notes
Video MiniMax-H3 (T2VA / FL2VA / Ref2VA) cache, CP, TP, AdaLN host residency
Video Fast-H3 few-step DMD2 students with VSA
Video SANA-Video 2.0 (720p) 5B
Video SANA-Video 2B (T2V/I2V) cache, CP, TP, quantization
Video SANA-WM (I2V) camera-controlled
Video LingBot-Video Dense 1.3B vendored 5-file pipeline
Video LongLive 2.0 (5B) cache, CP, TP, quantization
Video LongCat-Video-Avatar 1.5 (A2V/AI2V) cache, CP/TP, quantization
Video MAGI-2 Preview 114B MoE audio-video DiT, cache, CP/TP, offload, decoders
Video MOVA 360p (T2VA/I2VA) cache, CP, TP
Video HY-Nomas segment-AR camera-controlled, cache, CP, TP
Video Wan-Animate-2 base + distilled
Video Pika H3 Flex (T2VA/FL2VA/REF2VA) cache, CP, TP, fp8; plus a diffusers-native modular pipeline
Video CausVid-Wan Wan2.1-T2V-1.3B causal student
Video LTX-2.5 cache, TP, CP, fp8/SVDQuant, compile
Image HiDream-O1-Image dense CP mask, vision-conditioned
Image Ming-Image 0.1 design variants, MLLM condition extraction
Image PRXPixel cache, CP, TP, TE-P, quantization, compile
Image FireRed-Image-Edit 1.1 also fixes the Qwen-Image 1.x TP head_dim
Image Boogu-Image native, on the _C_ops -> cache_dit::models kernel family
Image MiniMax-H3-Image (T2I/Edit) exact AdaLN plan memoization
Image P-H3-Image (T2I/Edit) 8-step MiniMax-H3 based
Image FLUX.3 diffusers modular pipelines
Audio Stable Audio 3 text-to-audio
Audio LongCat-AudioDiT TTS, plus audio metrics

Built-in pipelines (those beyond 🤗Diffusers) are listed in the README with their minimum cache-dit version.

2. ⚡ Kernel & Quantization Stack

💎 SVDQuant W4A4

  • CuTe NVFP4 grew into a first-class path next to the native one: a unified runtime API (runtime + lora_layout), split-K producer with rank/M dispatch, a simplified FP4 mainloop, and a tuned row-major LoRA schedule.
  • Fused MLP and SwiGLU (#1332): a fused epilogue (qout/oscales, both D layouts, write_dense), producer-side SwiGLU on both producer families, and row_major as a first-class fused path.
  • Chain-level tuning (#1343, #1344): vectorized and paired fused-quantize stores, pipelined B-operand loads in the D epilogue, kFuseDown specialization, and a geometry-gated large-K fc2 path. Full-M fc2 now reaches 0.948x / 0.965x of the native kernel at M=4096 / 8192.
  • FlashInfer chain, opt-in (#1308, #1310): runtime_kernel="flashinfer+cute-dsl" or "flashinfer+cutlass-sm120"; ranks the chain cannot cover are rejected instead of silently downgraded. Replacing the exact amax(x * smooth) global scale with a safe upper bound cuts the per-layer cost from 127 µs to 48 µs.
  • --svdq-lora-layout is exposed in the generate CLI.

🎛️ FP8

  • Native FP8 backend (#1302): CUDA quantize plus a warp-specialized GEMM with split row/tensor kernels, and activation quantize fused through torch.compile.
  • CUTLASS SM120 collectives for every mode: per-row, per-tensor and per-block now share one mainloop with automatic padding/depad and a unified fp8_quantize_gemm. The use_ws knob is gone — CUTLASS selects the mainloop and the warp specialization; use_native=False still selects the torch._scaled_mm reference for the shapes it supports.

🔢 FP4

  • Native NVFP4 backend with torch._scaled_mm as the default GEMM, an SM120 CuTe GEMM family and a native NVFP4 Linear.
  • Then CUTLASS 4.8.1 collectives only: the hand-written FP4 GEMM and its use_ws were removed, entry points renamed to fp4_gemm_cutlass_2x (two stages quantize) / fp4_gemm_cutlass_1x (one stage quantize), with automatic padding for unsupported shapes.

🧬 Memory-bound block-operator fusion

Second-pass pipelines that keep the diffusers modules and state dict, and rebind only the block/attention seams to fused CUDA operators:

Model Measured (RTX PRO 5000, CUDA events)
FLUX.2-Klein (#1298) eager 228.79 → 196.44 ms/step (−14.1%), compile −0.3%; fused+compile vs stock+compile 46.75 dB / 0.9970 against compile's own drift of 46.70 dB / 0.9971
FLUX.3-Klein (#1299) eager 290.5 → 241.1 ms (−17.0%), compile ±0; e2e 4.83 → 4.03 s, --cache 3.78 → 3.17 s; qk_norm_rope 28.0 µs at 83.5% of DRAM peak vs inductor's 75.2 µs at 71.8%
Qwen-Image 2.1 (#1274) eager 403.0 → 355.3 ms/step (−11.8%), CP2 −7.3%; 43.74 dB / 0.9920 vs stock+compile

Every fused-op family is controlled by one switch, CACHE_DIT_FUSED_OPS (#1301).

📦 One extension, one backend

  • The four CUDA extensions were merged into a single _C_ops module (#1288) with one pybind entry point and per-family registrars; the build has one source list, one arch predicate (>=sm89, excluding sm90) and the default arch set ("89", "120a").
  • The backend-selection layer is gone: kernels/backend.py, kernels/cuda/, kernels/triton/, kernels/cutedsl/ and kernels/ops.py were removed. CUDA is the only implementation, and there is no runtime choice between Triton, CuTe DSL and CUDA.
  • svdq_extension_is_available was shortened to svdq_is_available.

3. 🔀 Distributed Module Refactor

  • distributed/core/ was split by responsibility and deleted, with no compatibility shim (#1290): communicate/primitives.py owns the all-to-all / Ring p2p collectives, head pad-unpad helpers, comm metadata and the FP8 comm kernel bindings; parallel/ owns the CP config/plan types, the hook registry, the CP runtime, ulysses/ring/usp and the diffusers bridge. Call sites import from distributed.parallel or distributed.communicate by symbol ownership.
  • async_ulysses moved under distributed/transformers/, which is the only place it ever patched.
  • Ulysses communication: per-chunk 16-byte alignment for the FP8 and BF16 all-to-all paths, the FP8 O-permute fused with quantize/dequantize, K-FP8 turned into an explicit opt-in, the communication kernels reorganized under csrc/kernels/ulysses as ulysses_fp8_*, read-only NUMA topology diagnostics, and Ring K/V receive buffers reused.
  • BF16 alignment padding is computed from the last dimension alone, so the common aligned case costs no list traversal.

4. 🧠 SVD-Cache Calibrator

A subspace-split calibrator (#1340): the cached block tensor is decomposed in the token × channel plane into a low-rank principal part and a high-frequency residual, the principal part is smoothed across steps and the residual is frozen. The basis is fitted online from the first full-compute feature of the current inference, so there is no offline calibration artifact (arXiv:2601.07396).

Two predictors are available: ema (the paper's per-step smoothing) and taylor (SVD plus Taylor extrapolation).

Measured status (FLUX.1-dev, 1024², 28 steps, seed 0, RTX PRO 5000) — treat SVD-Cache as reproduction/experimental support: on this harness neither predictor beats plain reuse often enough to recommend it.

Configuration (rdt 0.12, 14/28 cached, ~1.9x) latency PSNR (dB) SSIM
DBCache (plain reuse) 8.85 s 23.11 0.8848
+ TaylorSeer 8.86 s 24.21 0.9123
+ DMD 8.91 s 23.49 0.9074
+ SVD-Cache ema (β 0.9) 9.48 s 20.62 0.8051
+ SVD-Cache taylor (order 1) 9.47 s 23.38 0.8882
+ SVD-Cache taylor (order 1, τ 0.999) 9.51 s 23.81 0.8990

For production use --calibrator taylorseer (or dmd).

Calibrator selection is now unified: --calibrator {taylorseer,dmd,foca,svd} plus --calibrator-params "k=v,k=v". The legacy --taylorseer / --dmd / --foca flags remain as aliases, and conflicting selections fail instead of silently winning.

5. 🧰 CLI and Package Layout

  • src/cache_dit/_utils is now src/cache_dit/cli (#1291): argument parsing, example factories, example registration and the entrypoint. The old name hid the command-line surface behind a generic private name.
  • Module and package names lost their leading underscores (#1292, #1293) — for example attention/_interface.py → attention/interface.py, attention/_backend_selector.py → attention/backend_selector.py. Class-level _{NAME} naming is untouched: it still marks "subclass of diffusers' NAME".

🔧 Other Enhancements

  • Profiler: windowed Torch/Nsight collection plus NVTX stage scopes and module hooks, wired into the MiniMax-H3 steps (#1326).
  • Metrics: PSNR inputs are cast to float64, so a uint8 tensor no longer wraps around (#1335). Audio metrics were added with LongCat-AudioDiT.
  • Quantization: opt-in per-layer accuracy observers (#1216).
  • Registry: owned adapters are scoped to cache-dit's own pipelines, so a foreign adapter cannot be claimed (#1320).
  • Build: the redundant [tool.setuptools_scm] table was removed from pyproject.toml (it silently lost to setup.py) (#1317); the release tools share one workspace preparation step and one wheel inspector.
  • Docs and skills: a usage guide for the built-in pipelines, a model-integration runbook, the downstream-compatibility audit skill, the cache-algorithm integration skill, and an operator reference for the FP8/FP4 contracts.

⚠️ Upgrade Notes

Area Change Action
Package cache_dit._utils → cache_dit.cli update imports
Package leading underscores dropped from module/package names update imports (cache_dit.attention.interface, cache_dit.attention.backend_selector, …)
Extension four CUDA extensions merged into one _C_ops rebuild or reinstall the extension; a stale .so is not compatible
Kernels kernels/backend.py and the Triton / CuTe DSL backends were removed CUDA is the only implementation; drop backend selection
Kernels svdq_extension_is_available → svdq_is_available rename
FP8 fp8_gemm(..., use_ws=...) was removed drop the kwarg; CUTLASS selects the mainloop
FP4 hand-written GEMM removed use fp4_gemm_cutlass_2x / fp4_gemm_cutlass_1x
Distributed distributed/core/ deleted import from distributed.communicate / distributed.parallel
Extension the CUDA targets are now sm_89 + sm_120a, under one floor predicate (>=sm89, sm90 excluded) Ampere (sm_80/sm_86) and Hopper (sm_90) no longer load the CUDA operators; add sm_100 with CACHE_DIT_CUDA_ARCH_LIST when building
CLI --taylorseer / --dmd / --foca still parse, but conflicting selections now fail migrate to --calibrator
Cache SVD-Cache is experimental prefer --calibrator taylorseer or dmd

📦 Install

pip install -U cache-dit                       # PyPI, pure Python
pip install -U cache-dit --extra-index-url https://vipshop.github.io/cache-dit/whl/cu13/   # +cu13, with the CUDA extension

The CUDA channel carries the local version +cu13 (cache-dit==1.5.5+cu13) for CPython 3.11–3.14, built for sm_89 and sm_120a. Those wheels are manylinux_2_34_x86_64 (the floor dropped from 2_38 because the extension no longer needs the newer GLIBC symbols).