[PPO2] What’s the difference between PPO1 and PPO2 ?

[stevenh-tw]

After reading some code,
I still not get the insight that why PPO2 is GPU optimized compared to PPO1 ?

Does anyone has any high-level insight?

[andytwigg]

Ppo2 uses vectorized envs to batch inference and training steps across multiple instances of an environment

…

On Sat, Jul 28, 2018, 9:43 AM Steven ***@***.***> wrote: After reading some code, I still not get the insight that why PPO2 is GPU optimized compared to PPO1 ? Does anyone has any high-level insight? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#485>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACXtta7Ie08soRxowLNd1kIcqpmGa4d5ks5uLJS1gaJpZM4VlGw9> .

[pzhokhov]

it is more that PPO1 is not gpu-optimized :) PPO1 uses 1 environment per MPI worker. In other words, when you run PPO1 with multiple MPI processes, each process creates its own copy of the environment, and its own neural net (NN). The gradients for NN updates are aggregated across the workers by virtue of using MpiAdamOptimizer class.
PPO2 implementation (while with a recent updates it can use MPI as well) uses different version of parallelism. Head process with a single neural net creates a bunch of subprocesses that run separate environments (run environments == take actions and produce next observations and rewards). As @andytwigg describes, the observations and rewards from these multiple environments in subprocesses are batched together in the head process. For visual observations, that creates a big enough batch so that computation of NN gradients on a GPU starts making sense (esp. if using convnets). Note that, by default, multiple environments in subprocesses are only used for atari and retro video games, but not, for instance, for mujoco (because observations there are not visual).
All that being said - ppo1 is now obsolete, and its functionality (including mpi) is fully covered by ppo2.
Closing this; please reopen if some unclarity remains.

[Bick95]

Hey!

@pzhokhov I saw your comment above, saying that PPO2's main advancement over PPO1 is that the former uses a more efficient form of parallelism than the latter.

However, on some other website I found that there shall be more undocumented changes.

Therefore, I was wondering whether a full list of all the small changes made in PPO2 compared to PPO1 exists, since I would be very interested in the details of the algorithm. :)