I designed this Challenge for you and me: Learn Deep Reinforcement Learning in Depth in 60 days!!

You heard about the amazing results achieved by Deepmind with AlphaGo Zero and by OpenAI in Dota 2! Don't you want to know how they work? This is the right opportunity for you and me to finally learn Deep RL and use it on new exciting projects.

The ultimate aim is to use these general-purpose technologies and apply them to all sorts of important real world problems. Demis Hassabis

This repository wants to guide you through the Deep Reinforcement Learning algorithms, from the most basic ones to the highly advanced AlphaGo Zero. You will find the main topics organized by week and the resources suggested to learn them. Also, every week I will provide practical examples implemented in python to help you better digest the theory. You are highly encouraged to modify and play with them!

During the whole challenge, I will update continuously this repository..

.. so stay tuned

#60DaysRLChallenge

Now we have also a Slack channel. To get an invitation, email me at andrea.lonza@gmail.com

This is my first project of this kind, so please, if you have any idea, suggestion or improvement contact me at andrea.lonza@gmail.com.

Prerequisites

• Basic level of Python and PyTorch
• Machine Learning
• Basic knowledge in Deep Learning (MLP, CNN and RNN)

Projects (Yet to decide)

• Q-learning
• DQN
• AC2
• ES
• AlphaGo Zero

Week 1 - Introduction

• An introduction to Reinforcement Learning by Arxiv Insights

• Introduction and course overview - CS294 by Levine

• Deep Reinforcement Learning: Pong from Pixels by Karpathy

Week 2 - RL Basics: MDP, Dynamic Programming and Model-Free Control

Those who cannot rember the part are condomned to repeat it - George Santayana

This week, we will learn about the basic blocks of reinforcement learning, starting from the definition of the problem all the way through the estimation and optimization of the functions that are used to express the quality of a policy or state.

---

Theoretical material

• Markov Decision Process - RL by David Silver

  Formalizing RL problem using MDP

  • Markov Processes
  • Markov Decision Processes

• Planning by Dynamic Programming - RL by David Silver

  How to solve known MDP

  • Policy iteration
  • Value iteration

• Model-Free Prediction - RL by David Silver

  Estimate the value function of unknown MDP

  • Monte Carlo Learning
  • Temporal Difference Learning
  • TD(λ)

• Model-Free Control - RL by David Silver

  Optimise the value function of an unknown MDP

  • Ɛ-greedy policy iteration
  • GLIE Monte Carlo Search
  • SARSA
  • Importance Sampling

---

Project of the Week

Q-learning applied to FrozenLake. For exercise, you can solve the game using SARSA or implement Q-learning by yourself. In the former case, only few changes are needed.

---

To know more

• Read chapters 3,4,5,6,7 of Reinforcement Learning An Introduction - Sutton, Barto
• Value functions introduction - DRL UC Berkley by Sergey Levine

Week 3 - Value Function Approximation and DQN

This week we'll learn more advanced concepts and apply deep neural network to Q-learning algorithms.

---

Theoretical material

Lectures

• Value functions approximation - RL by David Silver

  • Differentiable function approximators
  • Incremental methods
  • Batch methods (DQN)

• Advanced Q-learning algorithms - DRL UC Berkley by Sergey Levine

  • Replay Buffer
  • Double Q-learning
  • Continous actions (NAF,DDPG)
  • Pratical tips

Papers

Must Read

• Playing Atari with Deep Reinforcement Learning - 2013
• Human-level control through deep reinforcement learning - 2015
• Rainbow: Combining Improvements in Deep Reinforcement Learning - 2017

Extensions to DQN

• Deep Reinforcement Learning with Double Q-learning - 2015
• Prioritized Experience Replay - 2015
• Dueling Network Architectures for Deep Reinforcement Learning - 2016
• Noisy networks for exploration - 2017
• Distributional Reinforcement Learning with Quantile Regression - 2017

---

Project of the Week

DQN and some variants applied to Pong

This week the goal is to develop a DQN algorithm to play an Atari game. To make it more interesting I developed three extensions of DQN: Double Q-learning, Multi-step learning, Dueling networks and Noisy Nets. Play with them, and if you feel confident, you can implement Prioritized replay, Dueling networks or Distributional RL. To know more about these improvements read the papers!

---

Suggested

• Deep Reinforcement Learning in the Enterprise: Bridging the Gap from Games to Industry

Week 4 - Policy gradient methods - A2C and A3C

Week 5 - Advanced policy gradients - TRPO/PPO

Week 6 - Evolution Strategies and Genetic Algorithms

Week 7 - Model Based reinforcement learning - I2A

Week 8 - AlphaGoZero + Bonus

Last 4 days - Review + sharing

Best RL papers

Best resources

📺 Deep Reinforcement Learning - UC Berkeley class by Levine, check here their site.

📺 Reinforcement Learning course - by David Silver, DeepMind. Great introductory lectures by Silver, a lead researcher on AlphaGo. They follow the book Reinforcement Learning by Sutton & Barto.

📓 Reinforcement Learning: An Introduction - by Sutton & Barto. The "Bible" of reinforcement learning. Here you can find the PDF draft of the second version.

Additional resources

📚 Awesome Reinforcement Learning. A curated list of resources dedicated to reinforcement learning