Skip to content

about CUDA out of memory,when test 3min audio #2

@zhanghongyong123456

Description

@zhanghongyong123456

Test two minutes audio prompt cuda memory is full, I look at the error found is audio feature extraction error, how to achieve the feature extraction of c long video
main()
File "main/demo_dir.py", line 135, in main
test(model, cfg.demo_wav_dir_path, save_folder, condition, subject)
File "main/demo_dir.py", line 179, in test
prediction = model.predict(audio_feature, template_face, one_hot)
File "D:\2023project\project\0win_os\05speech_Pro\MultiTalk\models\stage2.py", line 123, in predict
hidden_states = self.audio_encoder(audio, self.dataset).last_hidden_state
File "H:\Anaconda3\envs\multitalk\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "D:\2023project\project\0win_os\05speech_Pro\MultiTalk\models\lib\wav2vec.py", line 147, in forward
encoder_outputs = self.encoder(
File "H:\Anaconda3\envs\multitalk\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "H:\Anaconda3\envs\multitalk\lib\site-packages\transformers\models\wav2vec2\modeling_wav2vec2.py", line 645, in forward
layer_outputs = layer(
File "H:\Anaconda3\envs\multitalk\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "H:\Anaconda3\envs\multitalk\lib\site-packages\transformers\models\wav2vec2\modeling_wav2vec2.py", line 493, in forward
hidden_states, attn_weights, _ = self.attention(
File "H:\Anaconda3\envs\multitalk\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "H:\Anaconda3\envs\multitalk\lib\site-packages\transformers\models\wav2vec2\modeling_wav2vec2.py", line 372, in forward
attn_weights = F.softmax(attn_weights, dim=-1)
File "H:\Anaconda3\envs\multitalk\lib\site-packages\torch\nn\functional.py", line 1834, in softmax
ret = input.softmax(dim)
RuntimeError: CUDA out of memory. Tried to allocate 13.41 GiB (GPU 0; 48.00 GiB total capacity; 96.36 GiB already allocated; 0 bytes free; 97.79 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions