Home

This is the wiki for the reinforcement learning Discord.

Contents

0. First Steps
1. Debugging Advice
2. Recommended Resources
   • Written Introductions
   • Lecture Videos
   • Blogs
   • Miscellaneous
3. Links
   • Lecture Videos
   • Blogs
   • Other Communities
   • Miscellaneous
   • RL Environments
   • RL Frameworks & Reference Implementations

First Steps

If you're entirely new to reinforcement learning, the best thing to do is to poke through the resources listed in Written Introductions and Video Introductions and see which ones suit your learning style.

All of these cover roughly the same material, and all have roughly the same prerequisites:

• you should be comfortable with probability, statistics, linear algebra and multivariable calculus at a first-year-undergrad level,
• you should be comfortable with programming in Python,
• and you should be familiar with deep learning concepts like convolutions and gradient descent.

If you're entirely new to probability/statistics/linear algebra/multivariable calculus, we'll make a weak recommendation for Khan Academy's stats/lin alg/calc courses, or for the Mathematics for Machine Learning book. These resources suit a lot of people well.

If you're entirely new to programming in Python, the /r/python wiki has a range of introductory resources and we'll make a weak recommendation for Automate the Boring Stuff With Python as a book that suits many people well.

If you're entirely new to deep learning, the /r/ML wiki has a range of introductory resources, and we'll make a weak recommendation for fast.ai as a course that suits many people well.

Debugging Advice

• Andy Jones's advice
• Matthew Rahtz's advice
• John Schulman's advice
• Joshua Achiam's advice
• Marcin Gumer's advice
• Antonin Raffin's advice
• Clemens Winter's advice
• Nato Lambert's advice on model-based RL debugging

If you go through these and you're still stuck, come ask for help in #practical on the Discord. Keep in mind that debugging RL is extraordinarily tricky even when you've direct access, so don't expect miracles! It helps a lot if you can share a minimal example, describe the problem in detail, and explain how you've tried to fix it so far. The more detail you put into your question, the more likely it'll be a good answer!

Recommended Resources

These resources are things we have first-hand experience of and can testify to the quality of.

If you've had a good experience with a resource, edit this page and increment the 👍. Feel free to move stuff from the things-we-found section up to here if you've personally had a good experience with it. If you're uncertain, come discuss it in #wiki first.

Written Introductions

• 6👍 Spinning Up is somewhere between an extended tutorial and a short course, with a practical bent.
• 7👍 Sutton & Barto's Introduction to Reinforcement Learning is the standard intro textbook.
• 1👍 Deep Reinforcement Learning Series by Jonathan Hui
• 1👍 Algorithms for Decision Making by Mykel J. Kochenderfer, Tim A. Wheeler, and Kyle H. Wray. Great introduction for RL. Lots of great resources on various subjects.

Video Introductions

• 7👍 Introduction to Reinforcement Learning with David Silver, with slides
• 2👍 Deep RL Course by Thomas Simonnini
• 2👍 Reinforcement Learning Specialization on Coursera from Martha and David White, professors at UAlberta
  • Unlike all the other materials listed here, this is not free. Our opinions on paywalled courses are divided, so take this as a qualified recommendation - only go down this route if you have a strong preference for MOOC-style learning. If you don't, any of the other free material is just as good.

Miscellaneous

• 3👍 Lilian Weng's Blog on RL, a phenomenal set of survey posts on different areas of RL.
• 3👍 TalkRL Podcast is all about Reinforcement Learning
• 1👍 Jordi Torres's article series, a very beginner friendly take on introduction to deep-rl.

Frameworks and Libraries

• 2👍 Stable Baselines in Pytorch (SB3), a set of robust, reliable implementations of RL algorithms.
• 2👍 RLlib by Ray, one of the better attempts at plug-n-play distributed RL.
• 1👍 CleanRL, a set of single-file implementations of basic RL algorithms.
• 1👍 AI Traineree, simple implementations with agents/envs being independent.

Other Communities

• General RL
  • /r/reinforcementlearning: the main hub for the RL community.
  • Artificial Intelligence: an extremely large general AI discussion server, with an RL channel.
  • ODS.ai: their Slack has an active Russian-language community.
• Specialist RL
  • L0 and Lc0: the Discords for Leela Zero and Leela Chess Zero, two open-source AlphaZero adaptations.
  • SC2 AI: Discord for StarCraft 2 bot development.
  • DonkeyCar League: a weekly friendly bot race league.
• General ML
  • EleutherAI: language-model focussed, but a lot of good general AI safety and scaling discussion too.
  • TPU Podcast: GAN focussed, but a lot of good general ML discussion. Origin of the EleutherAI discord and project.
  • ML Collective: More general than the others mentioned so far, while still explicitly research-focussed.

Links

This is a broad-strokes collection of things we've found in our around the internet that we think might be useful to others. It is much more lightly curated than the section above.

Lecture Videos

• Reinforcement Learning Course | DeepMind & UCL
• CS188@berkeley - Intro to AI (Weeks 3, 4 and 5 for Intro to MDPs and RL)
• CS285@berkeley - Deep Reinforcement Learning
• CS234@stanford - Reinforcement Learning Winter 2019
• Deep RL Bootcamp 2017
• RL course by Balaraman Ravindran (IIT Madras)
• CS 330: Deep Multi-Task and Meta Learning
• CS885@UWaterloo Spring 2018 - Reinforcement Learning

Blogs

• BAIR Berkeley Artificial Intelligence Research.
• RL Weekly is a weekly newsletter highlighting important progress in reinforcement learning in research or industry.

Miscellaneous

• University of Alberta simple RL algorithm flow chart

RL Frameworks & Reference Implementations

• OpenAI Baselines
• Stable Baselines
• SLM Lab: Modular Deep Reinforcement Learning framework in PyTorch
• Reinforcement Learning Coach
• Framework for training, evaluating, and deploying robust trading agents
• RLBench: Robot Learning Benchmark
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Denny Britz's implementations of basic RL algorithms.
• Garage: A toolkit for evaluating meta-RL and multi-task agents

RL Environments

The paperswithcode list is a good place to start.

• OpenAI Gym
• Unity ML-Agents
• PyBullet Gymperium: Open-source implementations of OpenAI Gym MuJoCo environments
• osim-rl: Reinforcement learning with musculoskeletal models in OpenSim
• Project Malmö: platform for Artificial Intelligence experimentation and research built on top of Minecraft
• Multi-agent particle environment used in the paper Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• Environment for reinforcement-learning algorithmic trading models
• Framework for training, evaluating, and deploying robust trading agents
• Doom-based AI Research Platform for Reinforcement Learning from Raw Visual Information
• OpenSpiel: Collection of RL environments and algorithms
• ns3-gym: The Playground for Reinforcement Learning
• Reco-gym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising
• RLBench: Robot Learning Benchmark
• OpenAI Gym trading environment
• Sairen: OpenAI Gym Reinforcement Learning Environment for the Stock Market
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Voyage Deepdrive: self-driving simulator
• CARLA: Open-source simulator for autonomous driving research
• Habitat: A simulation platform for research in embodied AI
• MineRL: Towards AI in Minecraft
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Multi-Agent-Learning-Environments
• PettingZoo: Multi-agent RL environments
• MiniGrid: Minimalistic Gridworld Environment
• endtoendAI: compilation of RL environments
• List of environments and competitions for RL and AI training
• Official OpenAI environments, and third party environments. Note that: the original link is no longer available.
• Awesome RL environment list