Skip to content
Companion to An Overview of Reinforcement Learning
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.

Deep Reinforcement Learning Algorithms

This repo serves as a companion to An Overview of Reinforcement Learning. It contains Pytorch implementations of the DQN, DDPG and PPO (for continuous action spaces) algorithms.

The DQN and DDPG algorithms are implemented in a parallel fashion in the style of APE-X with multiple CPU actor processes asynchronously communicating with a single CPU replay buffer process and a single GPU learner process. apex_png


To run any algorithm, run the command python --env <ENV_ID>.

Run python --help for further information about command line arguments.

Note on PPO

While it is straightforward to extend the implementation of PPO to be parallel as well, it was found that using decoupled actor processes did not perform very well due to differences in the running mean and standard deviation of the observations and rewards. Solving that by having all actor processes interact with a single process that kept track of the running means and standard deviations worked but caused a communication bottleneck that obviated the advantage of parallelism in the first place. OpenAI's baseline implementation solves this by using lower-level primitives that I chose to avoid for the sake of simplicity.

You can’t perform that action at this time.