Repository navigation
Question re multi-gpu training and torch seeding #3070
Replies: 1 comment 3 replies
|
Good catch, and you're reading the code right: in the commit you linked, This has since been fixed on batchgenerators main: |
|
Good catch, and you're reading the code right: in the commit you linked, This has since been fixed on batchgenerators main: |
Uh oh!
There was an error while loading. Please reload this page.
Hi! I was trying to understand how nnunet sets up multi-gpu training on a cluster, and was reading on the importance of each process having its own seeds so that there is randomness in the fg locations sampled and augmentations each worker performs.
This is evident in the
producerfunction in batchgenerators/dataloading/nondet_multi_threaded_augmenter.py, which sets numpy's random seed, so that the fg locations sampled and such is not the same in all workers for dataloading.However, I've noticed that if seeds is None (which is what happens using the NonDetMultiThreadedAugmenter), torch does not set any seeds, i.e.:
I was wondering: since batchgeneratorsv2 augmentations use torch.rand a bunch, how does this not lead to the augmentations being equal across different batches on different GPUs? Is this handled somewhere else which I'm missing?
Thanks a lot!!
All reactions