Training on multiple gpu

[iedmrc]

Hi,
What would be the best way of training gpt-2-simple on multiple GPUs? Is there any built-in way to do multi-gpu training? If not, would you consider adding this feature as an enhancement? At least, splitting batches on to different GPUs would be nice especially in the '345M' case which needs a lot of memory and compute power.

Thanks.

[saippuakauppias]

@huntrontrakkr , Is your fork still in development? Or how to run it so that it uses all the GPU?

[huntrontrakkr]

@saippuakauppias Sorry for the delay - been gone for a bit. Fork works fine for my uses, may need refinement otherwise. There is a new setting called "Multi_GPU" that one can turn on to activate it - will go through and index all the gpus then split up the blocks appropriately. The CLIs still need to be extended to allow for the option, but otherwise it works great.

[huntrontrakkr]

Just pushed up, should be in a good place now.

I may put in a pull request.

[saippuakauppias]

I think @minimaxir will be happy to get PR!