CUDA out of memory

[sophia1488]

Hi! First, thanks for your contribution to music source separation!
Since I'd like to separate vocals & accompaniment using only 1 model, and the pre-trained model you provided has only one target, I decided to train it from scratch.
I also modify the code so that a validation set is added.

However, I cannot train it on 3 GTX 1080 with small batch size (<12), and I thought that the ResUNet_ismir doesn't require that much memory? Or does it really need a lot of GPU memory?
Could you provide some insight, like how long it takes for you to train a model and how many GPU memory it requires?

Thank you a lot!

[qiuqiangkong]

Hi, please use ResUNet_subband model, it is 4x smaller and the performance is close to the ResUNet system.

…

On Tue, 9 Nov 2021 at 14:30, Sophia Y. Chou ***@***.***> wrote: Hi! First, thanks for your contribution to music source separation! Since I'd like to separate vocals & accompaniment using only 1 model, and the pre-trained model you provided has only one target, I decided to train it from scratch. I also modify the code so that a validation set is added. However, I cannot train it on 3 GTX 1080 with small batch size (<12), and I thought that the ResUNet_ismir doesn't require that much memory? Or does it really need a lot of GPU memory? Could you provide some insight, like how long it takes for you to train a model and how many GPU memory it requires? Thank you a lot! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#27>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADFXTSLSQPEESZFFCH7T6X3ULC5YTANCNFSM5HUPDLSA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[sophia1488]

Thank you so much, though I later found that it was my preprocessing problem...
One other thing I'd like to ask is that is it normal to have negative SDR at epoch 8?

[qiuqiangkong]

Usually the SDR is positive after 1 epochs.

…

On Sat, 13 Nov 2021 at 21:00, Sophia Y. Chou ***@***.***> wrote: Thank you so much, though I later found that it was my preprocessing problem... One other thing I'd like to ask is that is it normal to have negative SDR at epoch 8? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#27 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADFXTSJRBROSYV4X4I3A6NLULZOPJANCNFSM5HUPDLSA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[sophia1488]

Got it, thanks!