Description

The purpose of this repository is to summarise some of Offline RL's research to help build a knowledge framework for beginners in this topic (partly referenced hanjuku-kaso/awesome-offline-rl)

Reinforcement Learning Primer

• Notes on Reinforcement Learning Theory (ongoing)
  • Siyu Zhou

Papers

Review/Survey/Position Papers

• A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  • Rafael Figueiredo Prudencio, Marcos R. O. A. Maximo, and Esther Luna Colombini. arXiv, 2022.
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
  • Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. arXiv, 2020.

Offline RL: Theory/Methods

Policy Constraints

• [BCQ] Off-Policy Deep Reinforcement Learning without Exploration [slides] [code]
  • Scott Fujimoto, David Meger, and Doina Precup. ICML, 2019.
• [BEAR] Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction [website] [blog] [code]
  • Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. NeurIPS, 2019.
• [BRAC] Behavior Regularized Offline Reinforcement Learning
  • Yifan Wu, George Tucker, and Ofir Nachum. arXiv, 2019.
• [AWR] Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
  • Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. arXiv, 2019.

Regularization

• [CQL] Conservative Q-Learning for Offline Reinforcement Learning [website] [code] [blog]
  • Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. NeurIPS, 2020.
• [COMBO] COMBO: Conservative Offline Model-Based Policy Optimization
  • Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. NeurIPS, 2021.

Uncertainty Estimation

• [MOPO] MOPO: Model-based Offline Policy Optimization [code]
  • Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y. Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. NeurIPS, 2020.
• [MOReL] MOReL: Model-Based Offline Reinforcement Learning [podcast]
  • Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. NeurIPS, 2020.

Offline RL: Benchmarks/Experiments

• D4RL: Datasets for Deep Data-Driven Reinforcement Learning [website] [blog] [code]
  • Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. arXiv, 2020.
• d3rlpy: An Offline Deep Reinforcement Learning Library [software]
  • Takuma Seno and Michita Imai. arXiv, 2021.

Offline RL: Applications

• [Power Systems] Optimal Tap Setting of Voltage Regulation Transformers Using Batch Reinforcement Learning
  • Hanchen Xu, Alejandro D. Domínguez-García, and Peter W. Sauer. IEEE T POWER SYSTEMS, 2020.

Open Source Software/Implementations

Blog

Tutorials/Talks/Lectures

• Offline Reinforcement Learning: From Algorithms to Practical Challenges [NeurIPS 2020 Offline RL Tutorial Colab Exercise]
  • Aviral Kumar and Sergey Levine. NeurIPS2020.
• Introduction on Offline RL - Zhihu