Multi-GPU Support

[ddemilla]

Are there any plans to add multi-gpu support? I didn't see anything in the TODO list. Thanks!

[r9y9]

No plans, because I have only 1 GPU:( Contributions are very welcome. FWIW, see #1 (comment)

[ddemilla]

Ok thanks, I haven't done much work in PyTorch but I will take a stab at it.

[azraelkuan]

@ddemilla for multi gpu training, you will need to fix the code to use cpu to run the eval_model function in train.py or you can do not run it in training process.

[ddemilla]

@azraelkuan Thanks, I will keep that in mind. I just had my gpu quot increased on gcloud so I will start experimenting tonight!

[azraelkuan]

@ddemilla
https://github.com/azraelkuan/wavenet_vocoder/blob/20452d5492bc94185216806251480da20813f80e/train.py#L649

[r9y9]

This should be fixed by #55. Feel free to reopen and leave comments if you have further requests. Thanks again, @azraelkuan,

[zhang-jian]

why we need "for multi gpu training, you will need to fix the code to use cpu to run the eval_model function in train.py or you can do not run it in training process."? Thanks

[azraelkuan]

@zhang-jian, because in eval mode, we only use one gpu to generate samples which will waste other resources. In my code, i use os.cmd() run the synthesis.py to eval a complete sentence.

[zhang-jian]

Clear now. Thanks.