One random number generator per GPU

[dominikgrewe]

Currently there's a single random number generator shared between all GPUs. The way it is set up seems to indicate that we really want one generator per GPU, but that's not what's happening:
In luaopen_libcutorch we call THCudaInit which itself calls THRandom_manualSeed for each GPU, each time creating a new generator and replacing the old one. We then call THCRandom_seed which again replaces the previously created generator with a new one.

Before #45 we'd only call THCRandom_manualSeed in cutorch_setDevice but that seems equally wrong.

If we want one generator per GPU we should create N generators at initialization and then pick the one for the currently chosen device when generating random numbers. Any thoughts?

[soumith]

You are right, our RNG code in cutorch might be broken. I haven't noticed it because I dont use it.

[soumith]

we might need a static array of generators and switch to the appropriate one on setDevice

[dominikgrewe]

Yes, that's what I had in mind too. We could even create them lazily in setDevice, thus only occurring the overhead if we really use multiple GPUs.

[soumith]

lazy sounds good too (though I doubt the overhead matters :) )

[dominikgrewe]

Fixed by #53