CalibAtt — Accelerating Text-to-Video Generation with Calibrated Sparse Attention (ECCV 2026)

Accelerating Text-to-Video Generation with Calibrated Sparse Attention
Shai Yehezkel, Shahar Yadin, Noam Elata, Yaron Ostrovsky-Berman, Bahjat Kawar

Abstract: Text-to-video diffusion models produce impressive results but are bottlenecked by the quadratic cost of self-attention over long spatiotemporal token sequences. We introduce CalibAtt, a calibration-based sparse attention method that accelerates inference without retraining. CalibAtt operates in two passes: a lightweight calibration pass that profiles attention patterns across a small set of prompts to produce aggregated sparse masks, followed by an optimized inference pass that applies these masks to skip unnecessary attention computations. Our approach is model-agnostic and compatible with existing video diffusion pipelines including Wan and Mochi. Experiments show that CalibAtt achieves significant speedups while maintaining video quality across standard benchmarks.

Installation

Requirements: Hopper H100/H200 GPU, CUDA >= 12.8, Python 3.10, PyTorch 2.8+, Linux.

Option A: Automatic setup (recommended)

Clone with submodules and run the setup script:

git clone --recursive https://github.com/apple/CalibAtt.git
cd CalibAtt
bash setup.sh

If you already cloned without --recursive, initialize submodules first:

git submodule update --init --recursive
bash setup.sh

Option B: Manual setup

If you prefer to install step by step (or need to customize individual steps), follow the instructions below.

0. Initialize submodules

All external C++/CUDA dependencies are pinned as git submodules under third_party/:

git submodule update --init --recursive

This checks out the exact versions of CUTLASS, Flash Attention, LiteAttention, and Composable Kernel that CalibAtt is tested against.

1. Create conda environment and install dependencies

conda env create -f environment.yaml
conda activate calibatt

# CUDA toolkit and FFmpeg
conda install "ffmpeg<8" -y
conda install -c nvidia cuda-toolkit=12.8 -y

# PyTorch
pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128

# Python dependencies
pip install -r requirements.txt

# Flash Attention 2
pip install flash-attn --no-build-isolation

2. Install Flash Attention 3

cd third_party/flash-attention/hopper
mkdir -p flash_attn_3
# CalibAtt only needs sm90, forward, hdim128 — disable the rest to speed up the build
MAX_JOBS=16 \
FLASH_ATTENTION_DISABLE_SM80=TRUE \
FLASH_ATTENTION_DISABLE_BACKWARD=TRUE \
FLASH_ATTENTION_DISABLE_HDIM64=TRUE \
FLASH_ATTENTION_DISABLE_HDIM96=TRUE \
FLASH_ATTENTION_DISABLE_HDIM192=TRUE \
FLASH_ATTENTION_DISABLE_HDIM256=TRUE \
FLASH_ATTENTION_DISABLE_FP8=TRUE \
FLASH_ATTENTION_DISABLE_SPLIT=TRUE \
FLASH_ATTENTION_DISABLE_PAGEDKV=TRUE \
FLASH_ATTENTION_DISABLE_SOFTCAP=TRUE \
python setup.py install
cd ../../..

3. Install kernel

Copy the LiteAttention submodule, apply the CalibAtt patch, symlink CUTLASS/CK, and build:

cp -r third_party/LiteAttention CalibAtt/kernel
cd CalibAtt/kernel
git apply ../../CalibAtt/kernel.patch
ln -sf ../../../third_party/cutlass csrc/cutlass
ln -sf ../../../third_party/composable_kernel csrc/composable_kernel
cd hopper
pip install -e . --no-build-isolation --config-settings editable_mode=compat
cd ../../..

4. Install LightX2V (optional, for distilled models only)

cd third_party/LightX2V
pip install -v -e .
cd ../..

Inference

Inference requires precomputed skip lists and similarity maps. You can either download pre-calibrated files or run calibration yourself (see Calibration).

Pre-calibrated files

Download the pre-calibrated skip lists and similarity maps and extract them under the calibration/ directory:

curl -fsSL "https://ml-site.cdn-apple.com/models/calibatt/pre_calibrated_files.zip" -o pre_calibrated_files.zip
unzip pre_calibrated_files.zip

This produces the following layout:

calibration/
├── wan_480p/
│   ├── skip_lists.pt
│   └── similarity_map.pt
├── wan_720p/
│   ├── skip_lists.pt
│   └── similarity_map.pt
└── mochi_480p/
    ├── skip_lists.pt
    └── similarity_map.pt

Running inference

torchrun --nproc_per_node=<NUM_GPUS> wan_run.py --config_path configs/wan_infer_720p.yaml

Other available configs: wan_infer_480p, wan_distilled_infer_480p, wan_distilled_infer_720p, mochi_infer. For distilled models use wan_distilled_run.py, for Mochi use mochi_run.py.

The inference configs point to calibration outputs via calibatt.skip_lists_path and calibatt.similarity_map_path. Edit your prompts in data/example.txt or point data_path to any text file with one prompt per line.

Calibration

Calibration profiles attention patterns on a small set of prompts and produces skip lists for inference. We use prompts from MovieGenBench:

mkdir -p data
curl -fsSL "https://raw.githubusercontent.com/facebookresearch/MovieGenBench/main/benchmark/MovieGenVideoBench.txt" \
    -o data/MovieGenVideoBench.txt

Running calibration

torchrun --nproc_per_node=<NUM_GPUS> wan_run.py --config_path configs/wan_calib_720p.yaml

Other calibration configs are available under configs/ for different models and resolutions.

Calibration configs enable calibrate_masks and calibrate_similarity under the calibatt section. After calibration completes, sparse masks are automatically aggregated into skip lists and saved to the aggregation.output_dir specified in the config (e.g., calibration/wan_720p/).

Extending to Other Models

CalibAtt is model-agnostic. To add support for a new video diffusion model, create a processor module in CalibAtt/processors/. Each processor must implement:

• make_processor(backend) — Returns a callable that replaces the model's native attention. It should perform QKV projections and route self-attention through backend.compute_attention(q, k, v).
• install(backend, pipeline) — Iterates over the model's transformer blocks, replaces each block's attention processor, and returns a list of kernel modules.
• after_step(backend) — Advances the layer, pass, and timestep counters according to the model's execution order.

See CalibAtt/processors/wan.py for a reference implementation and wan_run.py for how the processor is wired into the inference script. For a joint-attention (MMDiT) example, see CalibAtt/processors/mochi.py.

Configuration

All parameters are configured via YAML files in configs/. See the calibration configs (e.g., configs/wan_calib_480p.yaml) and inference configs (e.g., configs/wan_infer_480p.yaml) for all available options.

License

This project is licensed under the terms of the Apple License.

Citation

@inproceedings{yehezkel2026accelerating,
      title={Accelerating Text-to-Video Generation with Calibrated Sparse Attention},
      author={Shai Yehezkel and Shahar Yadin and Noam Elata and Yaron Ostrovsky-Berman and Bahjat Kawar},
      booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
      year={2026},
}

Acknowledgments

This work builds on the LiteAttention project, whose insights on efficient attention kernels were instrumental to our sparse attention implementation.

We thank Apple for supporting this research.