Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Got cutlass error: Error Internal at: 214 #998

Open
syntrive opened this issue Mar 26, 2024 · 1 comment
Open

Got cutlass error: Error Internal at: 214 #998

syntrive opened this issue Mar 26, 2024 · 1 comment

Comments

@syntrive
Copy link

Hi, when I run 05_stable_diffusion# python3 src/benchmark.py, there is a error:

pt output: torch.Size([1, 77, 1024])
[gemm_rcr_bias_add_25.cu] Got cutlass error: Error Internal at: 214
[20:21:02] model_interface.cu:221: Error: [gemm_rcr_bias_add_25.cu] Got cutlass error: Error Internal at: 214
Traceback (most recent call last):
File "src/benchmark.py", line 312, in
benchmark_diffusers()
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "src/benchmark.py", line 288, in benchmark_diffusers
benchmark_clip(
File "src/benchmark.py", line 176, in benchmark_clip
exe_module.run_with_tensors(inputs, ys)
File "/usr/local/lib/python3.8/dist-packages/aitemplate/compiler/model.py", line 597, in run_with_tensors
outputs_ait = self.run(
File "/usr/local/lib/python3.8/dist-packages/aitemplate/compiler/model.py", line 495, in run
return self._run_impl(
File "/usr/local/lib/python3.8/dist-packages/aitemplate/compiler/model.py", line 434, in _run_impl
self.DLL.AITemplateModelContainerRun(
File "/usr/local/lib/python3.8/dist-packages/aitemplate/compiler/model.py", line 196, in _wrapped_func
raise RuntimeError(f"Error in function: {method.name}")
RuntimeError: Error in function: AITemplateModelContainerRun

How can I solve it? thank you! env cuda11.7 cutlass3.4.1

@syntrive
Copy link
Author

device is A800

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant