Implement Proximal Policy Optimization

[seungjaeryanlee]

Like DQN (PR #617), Proximal Policy Optimization (PPO) is another widely used reinforcement learning algorithm. Proposed by Schulman et al. in 2017, PPO is an on-policy policy gradient algorithm that serves as a standard baselines for both environments with discrete and continuous action spaces.

There are two versions of PPO: PPO-Clip and PPO-Penalty. This code implements PPO-Clip, the more popular version.

TODO

• Fix existing issues
  • Optimize both actorNet and criticNet
  • Compute loss1 using the the minimum of surrogate losses surr1 and surr2
  • Fix gradients not being computed correctly and set to zero
• Find hyperparameters with consistent performance on CartPole
  • If performance is subpar, will implement GAE
• Refactor and document code
  • Refactor the Categorical distribution from swift-rl
  • Add documentation comments for hyperparameters
  • Add documentation comments for structs and classes
    • PPOMemory
    • ActorNetwork
    • CriticNetwork
    • ActorCritic
    • PPOAgent

[dan-zheng]

Some minor comments!

[BradLarson]

Functionally, this looks great. It reliably solves CartPole on my machines here. It looks good on my end to pull in.

There's some replicated code across our various RL examples, but I have ideas for how we can consolidate those once this is in.

The only other thing I'd add would be a small entry in the shared Readme for the Gym targets, but someone's already working on issue #657 to add further DQN documentation and they could add more about this target in that same update.

[seungjaeryanlee]

Awesome! I can review the relevant documentation (for both DQN and PPO) if needed.