Why should we set different seed per gpu with DDP?

[developer0hye]

deit/main.py

Line 182 in 35cd455

seed = args.seed + utils.get_rank()

Hi!

I recently referenced this project to implement DDP training code, and I can successfully implement and reproduce the performance of our model, thanks to this project.

I have one question related to DDP.

Why should we set different seed per gpu with DDP?

[TouvronHugo]

Hi @developer0hye,
Thanks for your question,
It is to be sure that the operations where we can be led to draw random values as for instance for data-augmentations will not be exactly the same between the different GPUs.
Best,
Hugo

[developer0hye]

@TouvronHugo
Hi @TouvronHugo
I got it! Thanks!