Skip to content

joshuaspear/d3rlpy

Β 
Β 

Repository files navigation

d3rlpy: An offline deep reinforcement learning library

test Documentation Status codecov Maintainability MIT

d3rlpy is an offline deep reinforcement learning library for practitioners and researchers.

import d3rlpy

dataset, env = d3rlpy.datasets.get_dataset("hopper-medium-v0")

# prepare algorithm
sac = d3rlpy.algos.SACConfig().create(device="cuda:0")

# train offline
sac.fit(dataset, n_steps=1000000)

# train online
sac.fit_online(env, n_steps=1000000)

# ready to control
actions = sac.predict(x)

⚠️ v2.x.x introduces breaking changes. If you still stick to v1.x.x, please explicitly install previous versions (e.g. pip install d3rlpy==1.1.1).

Key features

⚑ Most Practical RL Library Ever

  • offline RL: d3rlpy supports state-of-the-art offline RL algorithms. Offline RL is extremely powerful when the online interaction is not feasible during training (e.g. robotics, medical).
  • online RL: d3rlpy also supports conventional state-of-the-art online training algorithms without any compromising, which means that you can solve any kinds of RL problems only with d3rlpy.

πŸ”° User-friendly API

  • zero-knowledge of DL library: d3rlpy provides many state-of-the-art algorithms through intuitive APIs. You can become a RL engineer even without knowing how to use deep learning libraries.
  • extensive documentation: d3rlpy is fully documented and accompanied with tutorials and reproduction scripts of the original papers.

πŸš€ Beyond State-of-the-art

  • distributional Q function: d3rlpy is the first library that supports distributional Q functions in the all algorithms. The distributional Q function is known as the very powerful method to achieve the state-of-the-performance.

Installation

d3rlpy supports Linux, macOS and Windows.

PyPI (recommended)

PyPI version PyPI - Downloads

$ pip install d3rlpy

Anaconda

Anaconda-Server Badge Anaconda-Server Badge

$ conda install conda-forge/noarch::d3rlpy

Docker

Docker Pulls

$ docker run -it --gpus all --name d3rlpy takuseno/d3rlpy:latest bash

Supported algorithms

algorithm discrete control continuous control offline RL?
Behavior Cloning (supervised learning) βœ… βœ…
Neural Fitted Q Iteration (NFQ) βœ… β›” βœ…
Deep Q-Network (DQN) βœ… β›”
Double DQN βœ… β›”
Deep Deterministic Policy Gradients (DDPG) β›” βœ…
Twin Delayed Deep Deterministic Policy Gradients (TD3) β›” βœ…
Soft Actor-Critic (SAC) βœ… βœ…
Batch Constrained Q-learning (BCQ) βœ… βœ… βœ…
Bootstrapping Error Accumulation Reduction (BEAR) β›” βœ… βœ…
Conservative Q-Learning (CQL) βœ… βœ… βœ…
Advantage Weighted Actor-Critic (AWAC) β›” βœ… βœ…
Critic Reguralized Regression (CRR) β›” βœ… βœ…
Policy in Latent Action Space (PLAS) β›” βœ… βœ…
TD3+BC β›” βœ… βœ…
Implicit Q-Learning (IQL) β›” βœ… βœ…
Decision Transformer 🚧 βœ… βœ…

Supported Q functions

Benchmark results

d3rlpy is benchmarked to ensure the implementation quality. The benchmark scripts are available reproductions directory. The benchmark results are available d3rlpy-benchmarks repository.

Examples

MuJoCo

import d3rlpy

# prepare dataset
dataset, env = d3rlpy.datasets.get_d4rl('hopper-medium-v0')

# prepare algorithm
cql = d3rlpy.algos.CQLConfig().create(device='cuda:0')

# train
cql.fit(
    dataset,
    n_steps=100000,
    evaluators={"environment": d3rlpy.metrics.EnvironmentEvaluator(env)},
)

See more datasets at d4rl.

Atari 2600

import d3rlpy

# prepare dataset (1% dataset)
dataset, env = d3rlpy.datasets.get_atari_transitions(
    'breakout',
    fraction=0.01,
    num_stack=4,
)

# prepare algorithm
cql = d3rlpy.algos.DiscreteCQLConfig(
    observation_scaler=d3rlpy.preprocessing.PixelObservationScaler(),
    reward_scaler=d3rlpy.preprocessing.ClipRewardScaler(-1.0, 1.0),
).create(device='cuda:0')

# start training
cql.fit(
    dataset,
    n_steps=1000000,
    evaluators={"environment": d3rlpy.metrics.EnvironmentEvaluator(env, epsilon=0.001)},
)

See more Atari datasets at d4rl-atari.

Online Training

import d3rlpy
import gym

# prepare environment
env = gym.make('Hopper-v3')
eval_env = gym.make('Hopper-v3')

# prepare algorithm
sac = d3rlpy.algos.SACConfig().create(device='cuda:0')

# prepare replay buffer
buffer = d3rlpy.dataset.create_fifo_replay_buffer(limit=1000000, env=env)

# start training
sac.fit_online(env, buffer, n_steps=1000000, eval_env=eval_env)

Tutorials

Try cartpole examples on Google Colaboratory!

  • offline RL tutorial: Open In Colab
  • online RL tutorial: Open In Colab

More tutorial documentations are available here.

Contributions

Any kind of contribution to d3rlpy would be highly appreciated! Please check the contribution guide.

Community

Channel Link
Issues GitHub Issues

Family projects

Project Description
d4rl-atari A d4rl-style library of Google's Atari 2600 datasets

Roadmap

The roadmap to the future release is available in ROADMAP.md.

Citation

The paper is available here.

@article{d3rlpy,
  author  = {Takuma Seno and Michita Imai},
  title   = {d3rlpy: An Offline Deep Reinforcement Learning Library},
  journal = {Journal of Machine Learning Research},
  year    = {2022},
  volume  = {23},
  number  = {315},
  pages   = {1--20},
  url     = {http://jmlr.org/papers/v23/22-0017.html}
}

Acknowledgement

This work started as a part of Takuma Seno's Ph.D project at Keio University in 2020.

This work is supported by Information-technology Promotion Agency, Japan (IPA), Exploratory IT Human Resources Project (MITOU Program) in the fiscal year 2020.

About

An offline deep reinforcement learning library

Resources

License

Code of conduct

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 97.9%
  • Jupyter Notebook 1.8%
  • Other 0.3%