Deep-RL-Toolkit

Overview

Deep RL Toolkit is a flexible and high-efficient reinforcement learning framework. RLToolkit is developed for practitioners with the following advantages:

• Reproducible. We provide algorithms that stably reproduce the result of many influential reinforcement learning algorithms.

• Extensible. Build new algorithms quickly by inheriting the abstract class in the framework.

• Reusable. Algorithms provided in the repository could be directly adapted to a new task by defining a forward network and training mechanism will be built automatically.

• Elastic: allows to elastically and automatically allocate computing resources on the cloud.

• Lightweight: the core codes <1,000 lines (check Demo).

• Stable: much more stable than Stable Baselines 3 by utilizing various ensemble methods.

Table of Content

• Deep-RL-Toolkit
  • Overview
  • Table of Content
  • Supported Algorithms
  • Supported Envs
  • Examples
    • Quick Start
  • References
    • Reference Papers
    • References code

Supported Algorithms

RLToolkit implements the following model-free deep reinforcement learning (DRL) algorithms:

Supported Envs

• OpenAI Gym
• Atari
• MuJoCo
• PyBullet

For the details of DRL algorithms, please check out the educational webpage OpenAI Spinning Up.

Examples

If you want to learn more about deep reinforcemnet learning, please read the deep-rl-class and run the examples.

• Classic Control
• Atari Benchmark
• Box2d Benchmark
• Mujuco Benchmark
• Petting Zoo

Quick Start

git clone https://github.com/jianzhnie/deep-rl-toolkit.git

# Run the DQN algorithm on the CartPole-v0 environment
python examples/cleanrl/cleanrl_runner.py --env CartPole-v0 --algo dqn
python examples/cleanrl/cleanrl_runner.py --env CartPole-v0 --algo ddqn
python examples/cleanrl/cleanrl_runner.py --env CartPole-v0 --algo dueling_dqn
python examples/cleanrl/cleanrl_runner.py --env CartPole-v0 --algo dueling_ddqn

# Run the C51 algorithm on the CartPole-v0 environment
python examples/cleanrl/cleanrl_runner.py --env CartPole-v0 --algo c51

# Run the DDPG algorithm on the Pendulum-v1 environment
python examples/cleanrl/cleanrl_runner.py --env Pendulum-v0 --algo ddpg

# Run the PPO algorithm on the CartPole-v0 environment
python examples/cleanrl/cleanrl_runner.py --env CartPole-v0 --algo ppo

References

Reference Papers

1. Deep Q-Network (DQN) _{^{(V. Mnih et al. 2015)}}
2. Double DQN (DDQN) _{^{(H. Van Hasselt et al. 2015)}}
3. Advantage Actor Critic (A2C)
4. Vanilla Policy Gradient (VPG)
5. Natural Policy Gradient (NPG) _{^{(S. Kakade et al. 2002)}}
6. Trust Region Policy Optimization (TRPO) _{^{(J. Schulman et al. 2015)}}
7. Proximal Policy Optimization (PPO) _{^{(J. Schulman et al. 2017)}}
8. Deep Deterministic Policy Gradient (DDPG) _{^{(T. Lillicrap et al. 2015)}}
9. Twin Delayed DDPG (TD3) _{^{(S. Fujimoto et al. 2018)}}
10. Soft Actor-Critic (SAC) _{^{(T. Haarnoja et al. 2018)}}
11. SAC with automatic entropy adjustment (SAC-AEA) _{^{(T. Haarnoja et al. 2018)}}

References code

• rllib

  • https://github.com/ray-project/ray
  • https://docs.ray.io/en/latest/rllib/index.html

• coach

  • https://github.com/IntelLabs/coach
  • https://intellabs.github.io/coach

• Pearl

  • https://github.com/facebookresearch/Pearl
  • https://pearlagent.github.io/

• tianshou

  • https://github.com/thu-ml/tianshou
  • https://tianshou.org/en/stable/

• stable-baselines3

  • https://github.com/DLR-RM/stable-baselines3
  • https://stable-baselines3.readthedocs.io/en/master/

• PARL

  • https://github.com/PaddlePaddle/PARL
  • https://parl.readthedocs.io/zh-cn/latest/

• openrl

  • https://github.com/OpenRL-Lab/openrl/
  • https://openrl-docs.readthedocs.io/zh/latest/

• cleanrl

  • https://github.com/vwxyzjn/cleanrl
  • https://docs.cleanrl.dev/