Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Training setting problem #25

Open
Morank88 opened this issue Jun 27, 2020 · 6 comments
Open

Training setting problem #25

Morank88 opened this issue Jun 27, 2020 · 6 comments

Comments

@Morank88
Copy link

Hi,

Thank you for publishing your code!
I am encountering a training problem. As an initial phase I have tried to train only on 1000 samples from LibriSpeech train-clean-100 dataset. I am using the default configuration as was published in your VoiceFilter repo. The only difference is that I used batch size of 6 due to memory limitations. Is it possible that the problem is related to the small batch size that I use?

Another question is related to the generation of the training and testing sets. I have noticed that there is an option to use a VAD for generating the training set but by default it is not used. What is the best practice? to use the VAD or not?

I appreciate your help!

@Morank88
Copy link
Author

I manage to get some progress. Now I training on data from LibriSpeech train-clean-100 and train-clean-360 and testing on train-dev-clean. After 40k steps the SDR reached only to ~5. Is it possible that it is related to the batch size that I am using (6)?

Another question - what is the the learning rate policy? Did you fixed it on 1e-3 throughout the whole training or updated it?

Thanks.

@Morank88
Copy link
Author

Here are tensorboard results:
image

@natbuk2
Copy link

natbuk2 commented Sep 20, 2020

Hi Morank88 - did you get any improvement on your SDR? SDR I'm getting is much worse than even you're getting:
image

I've tried a number of runs (firstly, I had smaller batch size to run on lesser GFX card, but the run above was at as-downloaded batch size on amazon EC2 instance with NV100). Only differences are that I made the test sample 1000 (it's 100 in the code, but comment in the Readme mentioned 1000? maybe I'll change it back to the 100 as downloaded, and run again...) - and I have some likely more up-to-date python libraries (couldn't seem to find compatible torch 1.0.1 for example) - any suggestions?

Thanks in advance,

Nat

@jianew
Copy link

jianew commented Nov 22, 2020

Hi,

Thank you for publishing your code!
I am encountering a training problem. As an initial phase I have tried to train only on 1000 samples from LibriSpeech train-clean-100 dataset. I am using the default configuration as was published in your VoiceFilter repo. The only difference is that I used batch size of 6 due to memory limitations. Is it possible that the problem is related to the small batch size that I use?

Another question is related to the generation of the training and testing sets. I have noticed that there is an option to use a VAD for generating the training set but by default it is not used. What is the best practice? to use the VAD or not?

I appreciate your help!

hi,Can you share your settings, I run the same situation , thanks

@yunzhongfei
Copy link

I have also the same question.
The best result is 8 of SDR.

@zardkk
Copy link

zardkk commented Mar 17, 2022

Here are tensorboard results: image

i meet same question with you,how did you solve this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants