A cuda error when run your model

[Teiboku]

Traceback (most recent call last):
File "/root/autodl-tmp/code/FlashVSR/examples/WanVSR/infer_flashvsr_full.py", line 218, in
main()
File "/root/autodl-tmp/code/FlashVSR/examples/WanVSR/infer_flashvsr_full.py", line 204, in main
video = pipe(
^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/pipelines/flashvsr_full.py", line 407, in call
noise_pred_posi, pre_cache_k, pre_cache_v = model_fn_wan_video(
^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/pipelines/flashvsr_full.py", line 570, in model_fn_wan_video
x, last_pre_cache_k, last_pre_cache_v = block(
^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/models/wan_video_dit.py", line 461, in forward
self_attn_output, self_attn_cache_k, self_attn_cache_v = self.self_attn(
^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/models/wan_video_dit.py", line 362, in forward
x = self.attn(reorder_q, reorder_k, reorder_v, attention_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/models/wan_video_dit.py", line 296, in forward
x = flash_attention(q=q, k=k, v=v, num_heads=self.num_heads, attention_mask=attention_mask)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/code/FlashVSR/diffsynth/models/wan_video_dit.py", line 189, in flash_attention
x = block_sparse_attn_func(
^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/block_sparse_attn/block_sparse_attn_interface.py", line 399, in block_sparse_attn_func
return func.apply(
^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/torch/autograd/function.py", line 575, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/block_sparse_attn/block_sparse_attn_interface.py", line 192, in forward
out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = _block_sparse_attn_forward(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/autodl-tmp/envs/conda_envs/flashvsr/lib/python3.11/site-packages/block_sparse_attn/block_sparse_attn_interface.py", line 98, in _block_sparse_attn_forward
out, q, k, v, out_padded, softmax_lse, S_dmask, rng_state = block_sparse_attn_cuda.fwd_block(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

CUDA Version: 12.4
GPU: 1 x H800
The envrionment is by your readme, can you tell me is there any miss for my env?

[zhuang2002]

Thanks for your feedback. Based on the error log, this issue is not caused by FlashVSR, but by Block-Sparse Attention not being compiled correctly for your environment.

I recommend cloning and compiling Block-Sparse Attention in a clean folder. After compilation, please run the official correctness test to verify the installation:

python block_sparse_tests/fwd/test_correctness/full_test.py

If this test passes, Block-Sparse Attention is correctly built and FlashVSR should work normally. I have verified that this library can be successfully compiled on A100/A800/H200.

For more information, please refer to the official Block-Sparse Attention documentation:
https://github.com/mit-han-lab/Block-Sparse-Attention

[Teiboku]

I noticed that in your readme has this paragraph:

The Block-Sparse Attention backend currently achieves ideal acceleration only on NVIDIA A100 or A800 GPUs (Ampere architecture). On H100/H800 (Hopper) GPUs, due to differences in hardware scheduling and sparse kernel behavior, the expected speedup may not be realized, and in some cases performance can even be slower than dense attention.

So you have not tested your repo in H800? I think the Block-Sparse Attention seems doesn't support sm89 device like H800.

[zhuang2002]

Yes, that is correct. I have not tested on H800 yet, but I was able to run it successfully on A100/A800/H200. If you manage to make it work on H800, it would be great if you could share your experience.

[SekiroRong]

I tried on H100 and can't pass block_sparse_tests/fwd/test_correctness/full_test.py

[Teiboku]

I tried on H100 and can't pass block_sparse_tests/fwd/test_correctness/full_test.py

The Block-Sparse Attention only support sm80 and 90 device and H800 is sm89 device. I think if we can't use H800 besides the Block-Sparse Attention update(but seems no plan for this).

[zhuang2002]

I tried on H100 and can't pass block_sparse_tests/fwd/test_correctness/full_test.py

The Block-Sparse Attention only support sm80 and 90 device and H800 is sm89 device. I think if we can't use H800 besides the Block-Sparse Attention update(but seems no plan for this).

@Teiboku @SekiroRong Thanks for the discussion. Just for reference — in this related thread:
kijai/ComfyUI-WanVideoWrapper#1441

@lihaoyun6 shared a precompiled wheel for Block-Sparse Attention that claims support for SM_80 to SM_120 GPUs.
It might help users on H100/H800 who cannot compile the library from source.

@Teiboku @SekiroRong maybe you can give it a try:
🔗 https://github.com/lihaoyun6/Block-Sparse-Attention/releases

Thanks again @lihaoyun6 for sharing this!

[SekiroRong]

Unfortunately, the pre-build wheel is compiled on Windows rather than Linux

[SekiroRong]

BTW, Block-Sparse Attention build success on L20(sm86), and pass the block_sparse_tests/fwd/test_correctness/full_test.py

[lihaoyun6]

Unfortunately, the pre-build wheel is compiled on Windows rather than Linux

Since Block-Sparse Attention's setup.py is not optimized for Windows platform, it will be difficult to compile on Windows with the default configuration. On Linux, you only need to be patient. That's why I only made wheels for Windows 😁