Skip to content

Conversation

@swang109
Copy link

@swang109 swang109 commented Nov 9, 2024

docker image里的cuda好像要升级一下,不然有这个报错: server log:2024-11-09 12:31:36 2024-11-09 20:31:36,085 INFO send inference response 2024-11-09 12:31:36
0%| | 0/1 [00:00<?, ?it/s]2024-11-09 20:31:36,093 INFO synthesis text 你好,我是通义生成式语音大模型,请问有什么可以帮您的吗?
2024-11-09 12:31:44
0%| | 0/1 [00:08<?, ?it/s]
2024-11-09 12:31:44 2024-11-09 20:31:44,105 ERROR Exception iterating responses: cuFFT error: CUFFT_INTERNAL_ERROR
2024-11-09 12:31:44 Traceback (most recent call last):
2024-11-09 12:31:44 File "/opt/conda/lib/python3.10/site-packages/grpc/_server.py", line 589, in _take_response_from_response_iterator
2024-11-09 12:31:44 return next(response_iterator), True
2024-11-09 12:31:44 File "/opt/CosyVoice/CosyVoice/runtime/python/grpc/server.py", line 62, in Inference
2024-11-09 12:31:44 for i in model_output:
2024-11-09 12:31:44 File "/opt/CosyVoice/CosyVoice/runtime/python/grpc/../../../cosyvoice/cli/cosyvoice.py", line 61, in inference_sft
2024-11-09 12:31:44 for model_output in self.model.tts(**model_input, stream=stream, speed=speed):
2024-11-09 12:31:44 File "/opt/CosyVoice/CosyVoice/runtime/python/grpc/../../../cosyvoice/cli/model.py", line 191, in tts
2024-11-09 12:31:44 this_tts_speech = self.token2wav(token=this_tts_speech_token,
2024-11-09 12:31:44 File "/opt/CosyVoice/CosyVoice/runtime/python/grpc/../../../cosyvoice/cli/model.py", line 138, in token2wav
2024-11-09 12:31:44 tts_speech, tts_source = self.hift.inference(speech_feat=tts_mel, cache_source=hift_cache_source)
2024-11-09 12:31:44 File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2024-11-09 12:31:44 return func(*args, **kwargs)
2024-11-09 12:31:44 File "/opt/CosyVoice/CosyVoice/runtime/python/grpc/../../../cosyvoice/hifigan/generator.py", line 410, in inference
2024-11-09 12:31:44 generated_speech = self.decode(x=speech_feat, s=s)
2024-11-09 12:31:44 File "/opt/CosyVoice/CosyVoice/runtime/python/grpc/../../../cosyvoice/hifigan/generator.py", line 350, in decode
2024-11-09 12:31:44 s_stft_real, s_stft_imag = self._stft(s.squeeze(1))
2024-11-09 12:31:44 File "/opt/CosyVoice/CosyVoice/runtime/python/grpc/../../../cosyvoice/hifigan/generator.py", line 334, in _stft
2024-11-09 12:31:44 spec = torch.stft(
2024-11-09 12:31:44 File "/opt/conda/lib/python3.10/site-packages/torch/functional.py", line 641, in stft
2024-11-09 12:31:44 return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
2024-11-09 12:31:44 RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR

docker image里的cuda好像要升级一下,不然有这个报错: server log:2024-11-09 12:31:36 2024-11-09 20:31:36,085 INFO send inference response
2024-11-09 12:31:36 
  0%|          | 0/1 [00:00<?, ?it/s]2024-11-09 20:31:36,093 INFO synthesis text 你好,我是通义生成式语音大模型,请问有什么可以帮您的吗?
2024-11-09 12:31:44 
  0%|          | 0/1 [00:08<?, ?it/s]
2024-11-09 12:31:44 2024-11-09 20:31:44,105 ERROR Exception iterating responses: cuFFT error: CUFFT_INTERNAL_ERROR
2024-11-09 12:31:44 Traceback (most recent call last):
2024-11-09 12:31:44   File "/opt/conda/lib/python3.10/site-packages/grpc/_server.py", line 589, in _take_response_from_response_iterator
2024-11-09 12:31:44     return next(response_iterator), True
2024-11-09 12:31:44   File "/opt/CosyVoice/CosyVoice/runtime/python/grpc/server.py", line 62, in Inference
2024-11-09 12:31:44     for i in model_output:
2024-11-09 12:31:44   File "/opt/CosyVoice/CosyVoice/runtime/python/grpc/../../../cosyvoice/cli/cosyvoice.py", line 61, in inference_sft
2024-11-09 12:31:44     for model_output in self.model.tts(**model_input, stream=stream, speed=speed):
2024-11-09 12:31:44   File "/opt/CosyVoice/CosyVoice/runtime/python/grpc/../../../cosyvoice/cli/model.py", line 191, in tts
2024-11-09 12:31:44     this_tts_speech = self.token2wav(token=this_tts_speech_token,
2024-11-09 12:31:44   File "/opt/CosyVoice/CosyVoice/runtime/python/grpc/../../../cosyvoice/cli/model.py", line 138, in token2wav
2024-11-09 12:31:44     tts_speech, tts_source = self.hift.inference(speech_feat=tts_mel, cache_source=hift_cache_source)
2024-11-09 12:31:44   File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
2024-11-09 12:31:44     return func(*args, **kwargs)
2024-11-09 12:31:44   File "/opt/CosyVoice/CosyVoice/runtime/python/grpc/../../../cosyvoice/hifigan/generator.py", line 410, in inference
2024-11-09 12:31:44     generated_speech = self.decode(x=speech_feat, s=s)
2024-11-09 12:31:44   File "/opt/CosyVoice/CosyVoice/runtime/python/grpc/../../../cosyvoice/hifigan/generator.py", line 350, in decode
2024-11-09 12:31:44     s_stft_real, s_stft_imag = self._stft(s.squeeze(1))
2024-11-09 12:31:44   File "/opt/CosyVoice/CosyVoice/runtime/python/grpc/../../../cosyvoice/hifigan/generator.py", line 334, in _stft
2024-11-09 12:31:44     spec = torch.stft(
2024-11-09 12:31:44   File "/opt/conda/lib/python3.10/site-packages/torch/functional.py", line 641, in stft
2024-11-09 12:31:44     return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
2024-11-09 12:31:44 RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR
@aluminumbox
Copy link
Collaborator

这个好像有人提过一个pr,似乎是跟rtx显卡有关的问题,这个dockerfile是在v100上验证过的

@aluminumbox aluminumbox closed this Jun 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants