Implement PPO MPI (SB2 PPO1)

[araffin]

MPI can be quite useful to use multiprocessing full potential
but it is dependency that can be tricky to install.

[vwxyzjn]

Hey @araffin I prototyped multi-GPU support with torch.distributed vwxyzjn/cleanrl#162. Preliminary experiments seem successful when controlling torch thread number to 1 per process and use SyncVecEnv:

ppo_atari_multigpu_batch_reduce.py was able to obtain 20-30% speed up at no cost of sample efficiency by leveraging data parallelism given by torch.distributed.

[araffin]

@vwxyzjn ooh nice =)
I didn't know you could do that with PyTorch (it looks like they included MPI but for GPU).
I will try to have a look later this week.
Does this work also with cpu only?