getting "For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions." #11

swpatrick · 2023-12-12T08:39:11Z

I can run the program flawlessly upon Running on local URL: http://127.0.0.1:7860, but once it start animate the error occure.

Traceback (most recent call last):
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\gradio\queueing.py", line 456, in call_prediction
    output = await route_utils.call_process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\gradio\route_utils.py", line 232, in call_process_api
    output = await app.get_blocks().process_api(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\gradio\blocks.py", line 1522, in process_api
    result = await self.call_function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\gradio\blocks.py", line 1144, in call_function
    prediction = await anyio.to_thread.run_sync(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\anyio\to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\anyio\_backends\_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\anyio\_backends\_asyncio.py", line 807, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\gradio\utils.py", line 674, in wrapper
    response = f(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\demo\gradio_animate.py", line 30, in animate
    return animator(
           ^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\demo\animate.py", line 236, in __call__
    sample = self.pipeline(
             ^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\magicanimate\pipelines\pipeline_animation.py", line 654, in __call__
    ref_image_latents = self.images2latents(source_image[None, :], latents_dtype).cuda()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\magicanimate\pipelines\pipeline_animation.py", line 394, in images2latents
    latents.append(self.vae.encode(images[frame_idx:frame_idx+1])['latent_dist'].mean * 0.18215)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\diffusers\utils\accelerate_utils.py", line 46, in wrapper
    return method(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\diffusers\models\autoencoder_kl.py", line 260, in encode
    h = self.encoder(x)
        ^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\diffusers\models\vae.py", line 144, in forward
    sample = self.mid_block(sample)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\diffusers\models\unet_2d_blocks.py", line 562, in forward
    hidden_states = attn(hidden_states, temb=temb)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\diffusers\models\attention_processor.py", line 417, in forward
    return self.processor(
           ^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\diffusers\models\attention_processor.py", line 1036, in __call__
    hidden_states = F.scaled_dot_product_attention(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: the launch timed out and was terminated
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\gradio\queueing.py", line 501, in process_events
    response = await self.call_prediction(awake_events, batch)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\DisplacementMap\MagicAnimate_forwin\magic-animate-for-windows\venv\Lib\site-packages\gradio\queueing.py", line 465, in call_prediction
    raise Exception(str(error) if show_error else None) from error
Exception: None

I am runing gtx980 with Python 3.11.7 torch-2.0.1cu118-cp311-cp311-win_amd64

C:\Users\PatZ>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Wed_Feb__8_05:53:42_Coordinated_Universal_Time_2023
Cuda compilation tools, release 12.1, V12.1.66
Build cuda_12.1.r12.1/compiler.32415258_0

The text was updated successfully, but these errors were encountered:

sdbds · 2023-12-12T08:50:49Z

your torch with cuda 118,but your install cuda 12.1 in locally...
plz change requiments-windows about --extra-index-url https://download.pytorch.org/whl/cu118
to --extra-index-url https://download.pytorch.org/whl/cu121

swpatrick · 2023-12-12T08:57:56Z

C:\Users\PatZ>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:41:10_Pacific_Daylight_Time_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

C:\Users\PatZ>time
The current time is: 16:53:20.28
Enter the new time:
C:\Users\PatZ>date
The current date is: 12/12/2023 Tue
Enter the new date: (mm-dd-yy)

should be this, I didnt reopen the cmd window after i install the 11.8 cuda, sorry.
I had 12.1 at first

sdbds · 2023-12-12T08:59:56Z

C:\Users\PatZ>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:41:10_Pacific_Daylight_Time_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

C:\Users\PatZ>time
The current time is: 16:53:20.28
Enter the new time:
C:\Users\PatZ>date
The current date is: 12/12/2023 Tue
Enter the new date: (mm-dd-yy)

should be this, I didnt reopen the cmd window after i install the 11.8 cuda, sorry.

we recommend python 3.10 because some deps happend errors in 3.11

swpatrick · 2023-12-12T09:41:07Z

C:\Users\PatZ>python --version
Python 3.10.11

C:\Users\PatZ>time
The current time is: 17:23:53.55
Enter the new time:

C:\Users\PatZ>date
The current date is: 12/12/2023 Tue
Enter the new date: (mm-dd-yy)

C:\Users\PatZ>

--extra-index-url https://download.pytorch.org/whl/cu118/torch-2.0.1%2Bcu118-cp310-cp310-win_amd64.whl

I installed python 3.10 but got same error

sdbds · 2023-12-12T09:58:21Z

if your local cuda is 12.1，plz install cu121.
it is cuda environment problems...

swpatrick · 2023-12-12T10:13:57Z

what do you mean local cuda? , isnt cuda version is depend on what CUDA Toolkit you have installed on you machine? I had cuda 11.1 at the begining(from nvidia control pannel - system info), then i saw the project need 11.3 or above, so i install the 12.3 version, but torch does not support the 12.3 yet so I installed 12.1, sooner I figured out the program require v0.15 torchvision which only gonna work with v2.0 torch, and v2.0 torch does not support cuda 12.1 so finally I installed 11.8.

sdbds · 2023-12-12T10:22:20Z

check your environment path in windows, if you have multi cuda, it will use lastest version.

sdbds · 2023-12-12T10:22:59Z

what do you mean local cuda? , isnt cuda version is depend on what CUDA Toolkit you have installed on you machine? I had cuda 11.1 at the begining(from nvidia control pannel - system info), then i saw the project need 11.3 or above, so i install the 12.3 version, but torch does not support the 12.3 yet so I installed 12.1, sooner I figured out the program require v0.15 torchvision which only gonna work with v2.0 torch, and v2.0 torch does not support cuda 12.1 so finally I installed 11.8.

we use venv torch with cuda version like cu118 or cu121
but your system should have a cuda installed.

swpatrick · 2023-12-12T10:36:41Z

this is my enviorment path and use pip list after venv/script/active followed with nvcc --version

sdbds · 2023-12-12T12:49:23Z

this is my enviorment path and use pip list after venv/script/active followed with nvcc --version

as i known, GTX980 use CC with SM5.2
I guess cuda 11.8 or xformers ==0.0.22 deprecated SM5.2

(xformers 0.0.22 maybe)

You can down level these version for your GPU gtx980

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

getting "For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions." #11

getting "For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions." #11

swpatrick commented Dec 12, 2023

sdbds commented Dec 12, 2023 •

edited

Loading

swpatrick commented Dec 12, 2023 •

edited

Loading

sdbds commented Dec 12, 2023

swpatrick commented Dec 12, 2023

sdbds commented Dec 12, 2023

swpatrick commented Dec 12, 2023

sdbds commented Dec 12, 2023

sdbds commented Dec 12, 2023

swpatrick commented Dec 12, 2023

sdbds commented Dec 12, 2023

getting "For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions." #11

getting "For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with TORCH_USE_CUDA_DSA to enable device-side assertions." #11

Comments

swpatrick commented Dec 12, 2023

sdbds commented Dec 12, 2023 • edited Loading

swpatrick commented Dec 12, 2023 • edited Loading

sdbds commented Dec 12, 2023

swpatrick commented Dec 12, 2023

sdbds commented Dec 12, 2023

swpatrick commented Dec 12, 2023

sdbds commented Dec 12, 2023

sdbds commented Dec 12, 2023

swpatrick commented Dec 12, 2023

sdbds commented Dec 12, 2023

getting "For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions." #11

getting "For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions." #11

sdbds commented Dec 12, 2023 •

edited

Loading

swpatrick commented Dec 12, 2023 •

edited

Loading