ray-rllib/acm_recsys_tutorial_2022/01_intro_gym_and_rllib_optional.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bbd6492b",
   "metadata": {
    "tags": []
   },
   "source": [
    "© 2019-2022, Anyscale. All Rights Reserved\n",
    "\n",
    "![Anyscale Academy](../../images/AnyscaleAcademyLogo.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "742ec6ab",
   "metadata": {},
   "source": [
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/christy/AnyscaleDemos/blob/main/rllib_demos/recsys_conference/optional_01_intro_gym_and_rllib.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://github.com/christy/AnyscaleDemos/blob/main/rllib_demos/recsys_conference/optional_01_intro_gym_and_rllib.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7a5c9d0",
   "metadata": {},
   "source": [
    "#### Google Colab \n",
    "\n",
    "1. Look at top of notebook and click \"Copy to Drive\"\n",
    "2. Run cell below for required pip installs\n",
    "3. Adjust all the RLlib config() statements so total number of workers < 2 (Ray Tune) or <= 2 (RLlib algo.train() )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1a736279",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run this cell only for Google Colab\n",
    "# !pip install ray tensorflow_probability tensorboardX gym==0.21 lz4"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23708b75",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Optional - Introduction to the OpenAI Gym Environment and RLlib Algorithm top-level APIs\n",
    "\n",
    "### Learning objectives\n",
    "In this this tutorial, you will learn:\n",
    " * [What is Reinforcement Learning (RL)?](#intro_rl)\n",
    " * [Overview of RL terminology](#intro_rl)\n",
    " * [Introduction to OpenAI Gym environments](#intro_gym)\n",
    " * [High-level OpenAI Gym API calls](#intro_gym_api)\n",
    " * [Overview of RLlib](#intro_rllib)\n",
    " * [Train a policy using an algorithm from RLlib](#intro_rllib_api)\n",
    " * [Evaluate a RLlib policy](#eval_rllib)\n",
    " * [Reload RLlib policy from checkpoint and run inference](#reload_rllib)\n",
    " \n",
    " [Link to slides](https://github.com/anyscale/academy/blob/main/ray-rllib/acm_recsys_tutorial_2022/slides/rllib_acm_recsys_2022_slides.pdf)\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44f03ce0",
   "metadata": {
    "tags": []
   },
   "source": [
    "## What is Reinforcement Learning (RL)? <a class=\"anchor\" id=\"intro_rl\"></a> \n",
    "\n",
    "In the simplest definition, RL is a general framework where\n",
    "\n",
    "> **Agents** learn how to perform actions in an **environment** so as to maximize the cumulative sum of **rewards**.\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "<img src=\"./images/env_key_concept1.png\" width=\"50%\" />\n",
    "\n",
    "The agent and environment continuously interact in a feedback loop. \n",
    "- At each time step, from **State s(t)**, the agent takes an **action a(t)**.\n",
    "- The environment gives back a Reward **r(t+1)** and the next State **s(t+1)**.\n",
    "- Behind both the agent and environment is an algorithm.   \n",
    "<ul>\n",
    "    <ul>\n",
    "        <li>The algorithm uses data from past States, Actions, Rewards to train a policy, written <b>π</b>(s(t)).</li>\n",
    "        <li>The policy gives back the <b>Action a</b>(t).</li>\n",
    "    </ul>\n",
    "    </ul>\n",
    "\n",
    "<br>\n",
    "<b>The math concept behind RL is a Markov Decision Process</b><br>\n",
    "\n",
    "The sequence of States, Actions, Rewards <i>[S(0), A(0), R(1), S(1), A(1), ...]</i>, is a sequence of random variables.  \n",
    "> A sequence of random variables is a <b>stochastic process</b>.  \n",
    "\n",
    "The environment is fully known if we can predict the probability of a particular State at time t, given all the previous states and actions.  <i>Pr(S<sub>t+1</sub>=s<sub>t+1</sub> | A<sub>t</sub>=a<sub>t</sub>, S<sub>t</sub>=s<sub>t</sub>, A<sub>t-1</sub>=a<sub>t-1</sub>, ...S<sub>0</sub>=s<sub>0</sub>)</i>\n",
    "> <i>A stochastic process is a <b>Markov Decision Process</b> if the values at time t depend only on the values at time t-1</i>.  That is: <br>\n",
    "> > <i>Pr(S<sub>t+1</sub>=s<sub>t+1</sub> | A<sub>t</sub>=a<sub>t</sub>, S<sub>t</sub>=s<sub>t</sub>)</i>\n",
    "\n",
    "Markov Decision Processes (MDPs) can be solved computationally!  See the <a href=\"https://www.anyscale.com/blog/reinforcement-learning-with-deep-q-networks\">Anyscale tutorial blog</a> for an explanation of the <i>Bellman Equation</i>, which is a discretization of a Markov Decision Process, shown below.\n",
    "\n",
    "First, rewrite the MDP, expected reward, in terms of a Q function, where $\\gamma$ is a discount factor for future rewards.\n",
    "\n",
    "> $Q_π(s_t,a_t) = E_π \\left[\\sum_{j=0}^\\tau \\gamma^jr_{t+j+1}|S_t=s, A_t=a \\right]$\n",
    "\n",
    "<img src=\"./images/bellman_equation.png\" width=\"50%\" />\n",
    "\n",
    "> The ML problem behind RL is to minimize the mean squared error (MSE) difference of the 2 sides of the Bellman equation, just like in regression problems! \n",
    "\n",
    "Computationally, minimization problems can be solved practically using stochastic gradient descent (SGD) or Weighted Alternating Least Squares (WALS).\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "<b>Back to environments and agents...</b> <br>\n",
    "\n",
    "The **environment** is the agent's world, it is a simulation of the problem to be solved. The simulator might be of a:\n",
    "<ul>\n",
    "    <li>real, physical machine such as a gas turbine or autonomous vehicle</li>\n",
    "    <li>real, abstract system such as user behavior on a website or the stock market</li>\n",
    "    <li>virtual sytem on a computer such as a board game or a video game</li>\n",
    "    </ul>\n",
    "    \n",
    "The **agent** represents what is triggering the actions.  For example it could be:\n",
    "<ul>\n",
    "    <li>a software system that is triggering actions for machines</li>\n",
    "    <li>a type of user or investor</li>\n",
    "    <li>a game player or game system that is competing against real players </li>\n",
    "    </ul> \n",
    "    \n",
    "<br>\n",
    "<b>Comparison of RL to supervised learning</b> <br>\n",
    "<ul>\n",
    "    <li><u><i>Data</i></u>.  In supervised learning, you start with a labeled dataset.  In contrast, the <b>data in RL is not given up front; the environment acts as a data generator</b>.  One can also do RL on a pre-collected dataset (called offline RL), we will touch on offline RL later. </li> <br>\n",
    "    <li><u><i>Training</i></u>.  In supervised learning, a ML algorithm is trained on ALL the labeled training data AT ONCE.  <b>RL trains over a sequence of feedback loops.</b>  The RL algorithm optimizes the sum of individual rewards over repeated lifetimes (episodes) of sequential decisions: action -> feedback -> improved action -> repeat. </li><br>\n",
    "    <li><u><i>Evaluation</i></u>.  In supervised learning, a ML algorithm is evaluated on ALL the hold-out validation data AT ONCE. <b>RL REPEATEDLY evaluates a policy at different time steps</b>, typically whenever you save a checkpoint file. Evaluation at particular points in time in RL is similar in concept to \"backtesting\" in time series forecasting. RL evaluations are specific to a time step. </li>\n",
    "    </ul>\n",
    "\n",
    "<br>\n",
    "<b>In conclusion: why bother with an Agent, Environment, and RL?</b>  <br>\n",
    "\n",
    "Supervised learning can be too shortsighted or overlook important, changing user intents or business conditions.  \n",
    "\n",
    "<div class=\"alert alert-block alert-success\">    \n",
    "<b>💡 RL has become the de-facto ML approach for sequential decision-making processes, especially when there are multiple goals and long-term possibly delayed rewards. <br>\n",
    "    💡 RL can also work when there is no existing model to rely on or you want to improve over an existing decision-making strategy. </b> \n",
    "</div> \n",
    "\n",
    "<br> "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f0c899b",
   "metadata": {},
   "source": [
    "## Overview of RL terminology <a class=\"anchor\" id=\"intro_rl\"></a>\n",
    "\n",
    "An RL environment consists of: \n",
    "\n",
    "1. all possible actions (**action space**)\n",
    "2. a complete description of the environment, nothing hidden (**state space**)\n",
    "3. an observation by the agent of certain parts of the state (**observation space**)\n",
    "4. **reward**, which is the only feedback the agent receives after each action.\n",
    "\n",
    "The model that tries to maximize the expected sum over all future rewards is called a **policy**. The policy is a function mapping the environment's observations to an action to take, usually written **π** (s(t)) -> a(t).  <i>In deep reinforcement learning, this function is a neural network</i>.\n",
    "\n",
    "<b>Policy vs Model? </b>\n",
    "In traditional supervised learning, model means a trained algorithm, or a learned function.\n",
    "\n",
    "> <i>In RL, a model is roughly equivalent to a policy, but policy is more specific</i> because it is trained in a specific environment.  For deployment, we use the word \"model\" because more people understand the ML meaning of a trained model.\n",
    "\n",
    "Below is a high-level image of how the Agent and Environment work together to train a Policy in a RL simulation feedback loop in RLlib.\n",
    "\n",
    "<img src=\"./images/env_key_concept2.png\" width=\"98%\" />\n",
    "\n",
    "The **RL simulation feedback loop** repeatedly collects data, for one (single-agent case) or multiple (multi-agent case) policies, trains the policies on these collected data, and makes sure the policies' weights are kept in synch. \n",
    "\n",
    "During simulation loops, the environment collects observations, taken actions, receives rewards and so-called **done** flags, indicating the boundaries of different episodes the agents play through in the simulation.\n",
    "\n",
    "Each simulation iteration is called a <b>time step</b>.  The simulation iterations of action -> reward -> next state -> train -> repeat, until the end state, is called an **episode**, or in RLlib, a **rollout**.  At the end of the episode, when the <i>done</i> flag is True, we call RLlib method .reset(), which sets the <i>done</i> flag to False again.\n",
    "> 👉 Each episode consists of one or many time steps.\n",
    "\n",
    "<b>Per episode</b> (or between **done** flag == True), the RL simulation feedback loop repeats up to some specified end state (termination state or timesteps). Examples of termination are:\n",
    "<ul>\n",
    "    <li>the end of a maze (termination state)</li>  \n",
    "    <li>the player died in a game (termination state)</li>\n",
    "    <li>after 60 videos watched in a recommender system (timesteps).</li>\n",
    "    </ul>\n",
    "    \n",
    "<b>Why train for many episodes?</b>  When you are doing machine learning, you do not just do something once and report the result.  You do it many times, to make sure you did not just get \"lucky\" one time.  RL is similar.  By training for many episodes, you collect more data, which provides more variance, which is hopefully more realistic.  \n",
    "> 👉 Each training iteration consists of one or many episodes.\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "<b>💡 In RL, the policy is trained by repeating trials, or episodes (or rollouts), then reporting the calculated reward typically as an average of all achieved rewards per episode.  <br>\n",
    "   💡 The cumulative sum of all mean episode rewards is called the Return.</b> \n",
    "</div>\n",
    "    \n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93f5f99e",
   "metadata": {},
   "source": [
    "## Introduction to OpenAI Gym example: frozen lake <a class=\"anchor\" id=\"intro_gym\"></a>\n",
    "\n",
    "[OpenAI Gym](https://gym.openai.com/) is a well-known reference library of RL environments. \n",
    "\n",
    "#### 1. import gym\n",
    "\n",
    "Below is how you would import gym and view all available environments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ab66a503",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gym: 0.21.0\n",
      "Num Gym Environments: 1055\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[EnvSpec(FrozenLake-v1), EnvSpec(FrozenLake8x8-v1)]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import libraries\n",
    "import gym\n",
    "print(f\"gym: {gym.__version__}\")\n",
    "\n",
    "# List all available gym environments\n",
    "all_env  =  list(gym.envs.registry.all())\n",
    "print(f'Num Gym Environments: {len(all_env)}')\n",
    "\n",
    "# You could loop through and list all environments if you wanted\n",
    "# [print(e) for e in all_env]\n",
    "envs_starting_with_f = [e for e in all_env if str(e).startswith(\"EnvSpec(Frozen\")]\n",
    "envs_starting_with_f"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c02befc3",
   "metadata": {},
   "source": [
    "#### 2. Instatiate your Gym object\n",
    "\n",
    "The way you instantiate a Gym environment is with the **make()** function.\n",
    "\n",
    "The .make() function takes arguments:\n",
    "- **name of the Gym environment**, type: str, Required.\n",
    "- **runtime parameter values**, Optional.\n",
    "\n",
    "For the required string argument, you need to know the Gym name.  You can find the Gym name in the Gym documentation for environments, either:\n",
    "<ol>\n",
    "    <li>The doc page in <a href=\"https://www.gymlibrary.dev/environments/toy_text/frozen_lake/\">Gym's website</a></li>\n",
    "    <li>The environment's <a href=\"https://github.com/openai/gym/blob/master/gym/envs/toy_text/frozen_lake.py\">source code </a></li>\n",
    "    <li>\n",
    "        <a href=\"https://www.gymlibrary.ml/environments/classic_control/cart_pole/#description\">Research paper (if one exists)</a> referenced in the environment page </li>\n",
    "    </ol>\n",
    "    \n",
    "Below is an example of how to create a basic Gym environment, [frozen lake](https://www.gymlibrary.dev/environments/toy_text/frozen_lake/).  We can see below that the termination condition of an episode will be <b>TimeLimit</b> (the environment automatically ends an episode and sets done=True after this many timesteps).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "84aca5a7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: <TimeLimit<FrozenLakeEnv<FrozenLake-v1>>>\n",
      "env_spec: EnvSpec(FrozenLake-v1)\n"
     ]
    }
   ],
   "source": [
    "env_name = \"FrozenLake-v1\"\n",
    "\n",
    "# Instantiate gym env object with a runtime parameter value (is_slippery).\n",
    "# is_slippery=True specifies the environment is stochastic\n",
    "# is_slippery=False is the same as \"deterministic=True\"\n",
    "env = gym.make(\n",
    "    env_name,\n",
    "    is_slippery=False,  # whether the environment behaves deterministically or not\n",
    ")\n",
    "\n",
    "# inspect the gym spec for the environment\n",
    "print(f\"env: {env}\")\n",
    "env_spec = env.spec\n",
    "print(f\"env_spec: {env_spec}\")\n",
    "\n",
    "# Note: \"TimeLimit\" means termination condition for an episode will be time steps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e948aa36",
   "metadata": {},
   "source": [
    "#### 3. Inspect the environment action and observations spaces\n",
    "\n",
    "Gym Environments can be deterministic or stochastic.\n",
    "\n",
    "<ul>\n",
    "    <li>\n",
    "        <b>Deterministic</b> if the current state + selected action determines the next state of the environment.  <i>Chess is an example of a deterministic environment</i>, since all possible states/action combinations can be described as a discrete set of rules with states bounded by the pieces and size of the board.</li>\n",
    "    <li>\n",
    "        <b>Stochastic</b> if the policy output action is a probability distribution over a set of possible actions at time step t. In this case, the agent needs to compute its action from the policy in two steps. i) sample actions from the policy according to the probability distribution, ii) compute log likelihoods of the actions. <i>Random visitors to a website is an example of a stochastic environment</i>. </li>\n",
    "    </ul>\n",
    "\n",
    "<b>Gym actions.</b> The action_space describes the numerical structure of the legitimate actions that can be applied to the environment. \n",
    "\n",
    "For example, if we have 4 possible discrete actions, we could encode them as:\n",
    "<ul>\n",
    "    <li>0: LEFT</li>\n",
    "    <li>1: DOWN</li>\n",
    "    <li>2: RIGHT</li>\n",
    "    <li>3: UP</li>\n",
    "</ul>\n",
    "\n",
    "<b>Gym observations.</b>  The observation_space defines the structure as well as the legitimate values for the observation of a state of the environment.  \n",
    "\n",
    "For example, if we have a 4x4 grid, we could encode them as {0,1,2,3, 4, … ,15} for grid positions ((0,0), (0,1), (0,2), (0,3), …. (3,3)).\n",
    "\n",
    "From the Gym [documentation](https://www.gymlibrary.dev/environments/toy_text/frozen_lake/) about the frozen lake environment, we see: <br>\n",
    "\n",
    "|Frozen Lake      | Gym space   |\n",
    "|---------------- | ----------- |\n",
    "|Action Space     | Discrete(4) |\n",
    "|Observation Space| Discrete(16)|\n",
    " \n",
    "<b><a href=\"https://github.com/openai/gym/tree/master/gym/spaces\">Gym spaces</a></b> are gym data types.  The main types are `Discrete` for discrete numbers and `Box` for continuous numbers.  \n",
    "\n",
    "Gym Space `Discrete` elements are Python type `int`, and Gym Space `Box` are Python type `float32`.\n",
    "\n",
    "Below is an example how to inspect the environment action and observations spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "bf77077e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is a gym environment.\n",
      "\n",
      "gym action space: Discrete(4)\n",
      "gym observation space: Discrete(16)\n"
     ]
    }
   ],
   "source": [
    "# check if it is a gym instance\n",
    "if isinstance(env, gym.Env):\n",
    "    print(\"This is a gym environment.\")\n",
    "    print()\n",
    "\n",
    "    # print gym Spaces\n",
    "    if isinstance(env.action_space, gym.spaces.Space):\n",
    "        print(f\"gym action space: {env.action_space}\")\n",
    "    if isinstance(env.observation_space, gym.spaces.Space):\n",
    "        print(f\"gym observation space: {env.observation_space}\") \n",
    "        \n",
    "# Note: the action space is discrete with 4 possible actions.\n",
    "# Note: the observation space is 4x4 and thus runs from 0 to 15.\n",
    "# Note: if we chose 8x8, the observation space would change to Discrete(64)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "62a122a1",
   "metadata": {},
   "source": [
    "#### 4. Inspect gym environment default & runtime parameters\n",
    "\n",
    "Gym environments contain 2 sets of parameters that are set after the environment object is instantiated.\n",
    "<ul>\n",
    "    <li><b>Default parameters</b> are fixed in the Gym environment code itself.</li>\n",
    "    <li><b>Runtime parameters</b> are passed into the make() function as **kwargs.</li>\n",
    "    </ul>\n",
    "\n",
    "Below is an example of how to inspect the environment parameters.  Notice we can tell from the parameters that our frozen lake environment is: <br>\n",
    "1) <i>Deterministic</i>, and <br>\n",
    "2) Episode terminates with time step condition <i>max_episode_steps</i> = 100."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c8a7f9f9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Default spec params...\n",
      "id: FrozenLake-v1\n",
      "reward_threshold: 0.7\n",
      "nondeterministic: False\n",
      "max_episode_steps: 100\n",
      "order_enforce: True\n",
      "\n",
      "Runtime spec params...\n",
      "map_name: 4x4\n",
      "is_slippery: False\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# inspect env.spec parameters\n",
    " \n",
    "# View default env spec params that are hard-coded in Gym code itself\n",
    "# Default parameters are fixed\n",
    "print(\"Default spec params...\")\n",
    "print(f\"id: {env_spec.id}\")\n",
    "# rewards above this value considered \"success\"\n",
    "print(f\"reward_threshold: {env_spec.reward_threshold}\")\n",
    "# env is deterministic or stochastic\n",
    "print(f\"nondeterministic: {env_spec.nondeterministic}\")\n",
    "# number of time steps per episode\n",
    "print(f\"max_episode_steps: {env_spec.max_episode_steps}\")\n",
    "# must reset before step or render\n",
    "print(f\"order_enforce: {env_spec.order_enforce}\") \n",
    "\n",
    "# View runtime **kwargs .spec params.  These params set after env instantiated.\n",
    "# print(f\"type(env_spec._kwargs): {type(env_spec._kwargs)}\") #dict\n",
    "print()\n",
    "print(\"Runtime spec params...\")\n",
    "# Note: gym > v21 use just .kwargs instead of ._kwargs\n",
    "[print(f\"{k}: {v}\") for k,v in env_spec._kwargs.items()]\n",
    "print()\n",
    "\n",
    "# Note:  We can tell that our frozen lake environment is: \n",
    "# 1) Success criteria is rewards >= 0.7\n",
    "# 2) Deterministic\n",
    "# 3) Episode terminates when number time_steps = 100\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7659f606",
   "metadata": {
    "tags": []
   },
   "source": [
    "## High-level OpenAI Gym API calls <a class=\"anchor\" id=\"intro_gym_api\"></a>\n",
    "\n",
    "The most basic Gym API methods are: <br>\n",
    "\n",
    "- <b>env.reset()</b>\n",
    ">Reset the environment to an initial state.  Returns the initial observation.  <b>You should call this method every time at the start of a new episode.</b>\n",
    "\n",
    "- <b>env.step(action)</b> <br>\n",
    "> Using an action as input, applies that action to the environment and <b><i>returns the 4-tuple (next-observation, reward, done, info)</i></b>.\n",
    "\n",
    "- <b>action_space.sample()</b> <br>\n",
    "> Get a random action from the environment.  Used typically to loop through environment, calculating an environment \"Random Poicy baseline\".\n",
    "\n",
    "- <b>env.render()</b>  <br>\n",
    "> Visually inspect the environment. This is for human/debugging purposes; it is not seen by the agent/algorithm.  Note you cannot inspect an environment before it has been initialized with env.reset().\n",
    "    \n",
    "<div class=\"alert alert-block alert-success\">\n",
    "💡 <b>To play an episode, call reset() first!  <br>\n",
    "💡 After that, continue to call step() until the environment automatically returns done=True.</b> \n",
    "</div>\n",
    "\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3df6b15d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "\n",
      "\u001b[41mS\u001b[0mFFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n"
     ]
    }
   ],
   "source": [
    "# Print the starting observation.  \n",
    "# Recall possible observations are between 4x4 grid.\n",
    "print(env.reset())\n",
    "env.render()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "210a3b44",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "obs: 1, reward: 0.0, done: False\n",
      "  (Right)\n",
      "S\u001b[41mF\u001b[0mFF\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n",
      "obs: 5, reward: 0.0, done: True\n",
      "  (Down)\n",
      "SFFF\n",
      "F\u001b[41mH\u001b[0mFH\n",
      "FFFH\n",
      "HFFG\n"
     ]
    }
   ],
   "source": [
    "# Take an action\n",
    "# Recall the possible actions are: 0: LEFT, 1: DOWN, 2: RIGHT, 3: UP\n",
    "\n",
    "new_obs, reward, done, _ = env.step(2) #Right\n",
    "print(f\"obs: {new_obs}, reward: {reward}, done: {done}\")\n",
    "env.render()\n",
    "new_obs, reward, done, _ = env.step(1) #Down\n",
    "print(f\"obs: {new_obs}, reward: {reward}, done: {done}\")\n",
    "env.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b36534cd",
   "metadata": {},
   "source": [
    "We can also try to run an action in the frozen lake environment which is outside the defined number range."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c866c272",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Comment this cell if you want whole notebook to run without errors\n",
    "\n",
    "# Try to take an invalid action\n",
    "\n",
    "#env.step(4) # invalid\n",
    "\n",
    "# should see KeyError below"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61a34133",
   "metadata": {},
   "source": [
    "To test out your environment, typically you will loop through a few episodes to make sure it works.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2493ec57",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "episode: 4\n",
      "obs: 3, reward: 0.0, done: False\n",
      "  (Up)\n",
      "SFF\u001b[41mF\u001b[0m\n",
      "FHFH\n",
      "FFFH\n",
      "HFFG\n"
     ]
    }
   ],
   "source": [
    "from ipywidgets import Output\n",
    "from IPython import display\n",
    "import time\n",
    "\n",
    "# The following three lines are for rendering purposes only.\n",
    "# They allow us to render the env frame-by-frame in-place\n",
    "# (w/o creating a huge output which we would then have to scroll through).\n",
    "out = Output()\n",
    "display.display(out)\n",
    "with out:\n",
    "\n",
    "    # Putting the Gym simple API methods together.\n",
    "    # Here is a pattern for running a bunch of episodes.\n",
    "    num_episodes = 5 # Number of episodes you want to run the agent\n",
    "    total_reward = 0.0  # Initialize reward to 0\n",
    "\n",
    "    # Loop through episodes\n",
    "    for ep in range(num_episodes):\n",
    "\n",
    "        # Reset the environment at the start of each episode\n",
    "        obs = env.reset()\n",
    "        done = False\n",
    "\n",
    "        # Loop through time steps per episode\n",
    "        while True:\n",
    "            # take random action, but you can also do something more intelligent \n",
    "            action = env.action_space.sample()\n",
    "\n",
    "            # apply the action\n",
    "            new_obs, reward, done, info = env.step(action)\n",
    "            total_reward += reward\n",
    "\n",
    "            # If the epsiode is up, then start another one\n",
    "            if done:\n",
    "                break\n",
    "\n",
    "            # Render the env (in place).\n",
    "            time.sleep(0.3)\n",
    "            out.clear_output(wait=True)\n",
    "            print(f\"episode: {ep}\")\n",
    "            print(f\"obs: {new_obs}, reward: {total_reward}, done: {done}\")\n",
    "            env.render()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1531d661",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Overview of RLlib <a class=\"anchor\" id=\"intro_rllib\"></a>\n",
    "\n",
    "<img width=\"7%\" src=\"images/rllib-logo.png\"> is currently the most comprehensive open-source Python Reinforcement Learning framework. **RLlib** is <b>distributed by default</b> since it is built on top of **[Ray](https://docs.ray.io/en/latest/)**, an easy-to-use, open-source, distributed computing framework for Python that can handle complex, heterogeneous applications. Ray and RLlib run on compute clusters on any cloud without vendor lock.  RLlib Resources:\n",
    "<ol>\n",
    "    <li>The doc page on <a href=\"https://docs.ray.io/en/master/rllib/index.html\">ray.io website</a></li>\n",
    "    <li><a href=\"https://github.com/ray-project/ray/tree/master/rllib\">RLlib source code</a></li>\n",
    "    </ol>\n",
    "\n",
    "RLlib includes <b>25+</b> available [algorithms](https://docs.ray.io/en/master/rllib/rllib-algorithms.html), converted to both <img width=\"3%\" src=\"./images/tensorflow-logo.png\">_TensorFlow_ and <img width=\"3%\" src=\"./images/pytorch-logo.png\">_PyTorch_, covering different sub-categories of RL: _model-free_, _offline RL_, _model-based_, and _gradient-free_. Almost any RLlib algorithm can learn in a <b>multi-agent</b> setting. Many algorithms support <b>RNNs</b> and <b>LSTMs</b>.\n",
    "\n",
    "On a very high level, RLlib is organized by **environments**, **algorithms**, **examples**, **tuned_examples**, and **models**.  \n",
    "\n",
    "    ray\n",
    "    |- rllib\n",
    "    |  |- env \n",
    "    |  |- algorithms\n",
    "    |  |  |- alpha_zero \n",
    "    |  |  |- appo \n",
    "    |  |  |- ppo \n",
    "    |  |  |- ... \n",
    "    |  |- examples \n",
    "    |  |- tuned_examples\n",
    "    |  |- models\n",
    "\n",
    "Within **_env_** you will find [classes](https://docs.ray.io/en/latest/rllib/package_ref/env.html) that allow RLlib to handle e.g. the multi-agent cases (which gym does NOT cover).  RLlib automatically supports any **OpenAI Gym environment** (which supports most user cases). RLlib also handle external environments that have strict performance or hosting requirements. <i>(In the next notebook, we will use the **RLlib MultiAgentEnv** base class to create a **multi agent** environment).</i>\n",
    "\n",
    "Within **_examples_** you will find some examples of common custom rllib use cases.  \n",
    "\n",
    "Within **_tuned\\_examples_**, you will find, sorted by algorithm, suggested hyperparameter value choices within .yaml files. Ray RLlib team ran simulations/benchmarks to find suggested hyperparameter value choices.  These files are used for daily testing, and weekly hard-task testing to make sure they all run at speed, for both TF and Torch. Helps give you a leg-up with initial parameter choices!\n",
    "\n",
    "Within **_models_**, you will find building blocks for NNs, default models that RLlib will use (for either <img width=\"3%\" src=\"./images/tensorflow-logo.png\">_TensorFlow_ or <img width=\"3%\" src=\"./images/pytorch-logo.png\">_PyTorch_). For example, here are building blocks for DNN, CNN, RNN, and LSTM. \n",
    "\n",
    "In this tutorial, we will mainly focus on the **_algorithms_** package, where we will find RLlib algos to train policies on environments.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1748ae4c",
   "metadata": {},
   "source": [
    "## Train a policy using an algorithm from RLlib <a class=\"anchor\" id=\"intro_rllib_api\"></a>\n",
    "\n",
    "Once you have an environment, next you need to decide which RL algorithm to use.  There are many factors to consider when selecting which algorithm to use on your environment.  Following are some high-level best practices.\n",
    "\n",
    "1. <b>Choose an algorithm compatible with the action space.</b>  Do you have discrete actions (example: LEFT, RIGHT, …) or continuous actions (example: drive at a certain speed)? \n",
    "> To check high-level if an algorithm will work, look at the <a href=\"https://docs.ray.io/en/master/rllib/rllib-algorithms.html\">RLlib algorithms doc page</a>.  Algorithms are listed according to whether or not they support Discrete actions, or Continuous actions, or both.\n",
    "\n",
    "2. <b>Choose a stable algorithm.</b>  Look at the cumulative rewards per time step, they should rise steadily.  You do not want an algorithm where reward jumps up and down a lot.\n",
    "\n",
    "3. <b>Choose the most sample-efficient algorithm that works for your environment</b>.  Look at the cumulative rewards per time step, they should rise quickly. <i>PPO is extremely sample-efficient.  SAC is much less sample-efficient.</i>\n",
    "\n",
    "\n",
    "#### Step 1.  Import ray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f1f21c18",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of CPUs in this system: 16\n",
      "numpy: 1.23.3\n",
      "pandas: 1.4.4\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ray/anaconda3/lib/python3.8/site-packages/scipy/__init__.py:138: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.3)\n",
      "  warnings.warn(f\"A NumPy version >={np_minversion} and <{np_maxversion} is required for this version of \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ray: 3.0.0.dev0\n"
     ]
    }
   ],
   "source": [
    "# import commonly-used libraries\n",
    "import os\n",
    "import time\n",
    "import numpy as np\n",
    "print(f'Number of CPUs in this system: {os.cpu_count()}')\n",
    "import pandas as pd\n",
    "pd.set_option('display.max_columns', None)\n",
    "pd.set_option('display.max_rows', None)\n",
    "pd.set_option('display.max_colwidth', None)\n",
    "print(f\"numpy: {np.__version__}\")\n",
    "print(f\"pandas: {pd.__version__}\")\n",
    "\n",
    "# import ray\n",
    "import ray\n",
    "from ray import tune, rllib, air\n",
    "from ray.tune.logger import pretty_print\n",
    "print(f\"ray: {ray.__version__}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "617c422b",
   "metadata": {},
   "source": [
    "#### Step 2. Check environment for errors   \n",
    "\n",
    "Before you start training, it is a good idea to check the environment for errors.  RLlib provides a convenient [Environment pre-check function](https://github.com/ray-project/ray/blob/master/rllib/utils/pre_checks/env.py) for this.  It checks that the environment is compatible with OpenAI Gym and RLlib (and outputs a warning if necessary).\n",
    "\n",
    "Below, we check our Frozen Lake environment for errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "782aca4b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "checking environment ...\n",
      "All checks passed. No errors found.\n"
     ]
    }
   ],
   "source": [
    "from ray.rllib.utils.pre_checks.env import check_env\n",
    "\n",
    "# How to check you do not have any environment errors\n",
    "print(\"checking environment ...\")\n",
    "try:\n",
    "    check_env(env)\n",
    "    print(\"All checks passed. No errors found.\")\n",
    "except:\n",
    "    print(\"failed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a04320f",
   "metadata": {},
   "source": [
    "#### Step 3. Calculate an environment baseline\n",
    "\n",
    "Let's run through the environment, acting randomly, without rendering, and record the mean reward.  The purpose of this is to obtain a baseline before training a RLlib algorithm.\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "💡 If you are doing benchmarks, this random policy is often called a <b>\"baseline\".</b>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "40078197",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "**************\n",
      "Baseline Mean Reward=0.01+/-0.10 (out of success=0.7)\n",
      "Baseline won 31.0 times over 3000 episodes (23131 timesteps)\n",
      "Approx 0.01 wins per episode\n",
      "**************\n"
     ]
    }
   ],
   "source": [
    "# Putting the Gym simple API methods together.\n",
    "# Here is a pattern for running a bunch of episodes.\n",
    "num_episodes = 3000 # Number of episodes you want to run the agent\n",
    "num_timesteps = 0\n",
    "# Collect all episode rewards here\n",
    "episode_rewards = []\n",
    "\n",
    "# Loop through episodes\n",
    "for ep in range(num_episodes):\n",
    "\n",
    "    # Reset the environment at the start of each episode\n",
    "    obs = env.reset()\n",
    "    done = False\n",
    "    episode_reward = 0.0\n",
    "    \n",
    "    # Loop through time steps per episode\n",
    "    while True:\n",
    "        # take random action, but you can also do something more intelligent \n",
    "        action = env.action_space.sample()\n",
    "\n",
    "        # apply the action\n",
    "        new_obs, reward, done, info = env.step(action)\n",
    "        episode_reward += reward\n",
    "\n",
    "        # If the epsiode is up, then start another one\n",
    "        num_timesteps += 1\n",
    "        if done:\n",
    "            episode_rewards.append(episode_reward)\n",
    "            break\n",
    "\n",
    "# calculate mean_reward\n",
    "env_mean_random_reward = np.mean(episode_rewards)\n",
    "env_sd_reward = np.std(episode_rewards)\n",
    "# calculate number of wins\n",
    "total_reward = np.sum(episode_rewards)\n",
    "    \n",
    "print()\n",
    "print(\"**************\")\n",
    "print(f\"Baseline Mean Reward={env_mean_random_reward:.2f}+/-{env_sd_reward:.2f}\", end=\"\")\n",
    "print(f\" (out of success={env_spec.reward_threshold})\")\n",
    "print(f\"Baseline won {total_reward} times over {num_episodes} episodes ({num_timesteps} timesteps)\")\n",
    "print(f\"Approx {total_reward/num_episodes:.2f} wins per episode\")\n",
    "print(\"**************\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c9a00bf",
   "metadata": {},
   "source": [
    "#### Step 4.  Select an algorithm and find that algorithm's config class  \n",
    "\n",
    "Here is how to find an <b>RLlib algorithm's config class</b>.\n",
    "<ol>\n",
    "    <li>Open RLlib docs <a href=\"https://docs.ray.io/en/master/rllib/rllib-algorithms.html\">and navigate to the Algorithms page.</a></li>\n",
    "    <li>Scroll down and click url of algo you want to use, e.g. <i><b>DQN</b></i></li>\n",
    "    <li>On the <a href=\"https://docs.ray.io/en/master/rllib/rllib-algorithms.html#dqn\">algo docs page </a>, click on the <i><b>Implementation</b></i> link.  This will open the <a href=\"https://github.com/ray-project/ray/blob/master/rllib/algorithms/dqn/dqn.py\">algo code file on github</a>.</li>\n",
    "    <li>Scroll down to the <i>config class definition</i>.</li>\n",
    "    <li>Typically the docstring example will show: </li>\n",
    "    <ol>\n",
    "        <li>Example code using RLlib API, and </li>\n",
    "        <li>Example code using Ray Tune API.</li>\n",
    "    </ol>\n",
    "    </ol>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "cf7399e4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# config is an object instead of a dictionary since Ray version >= 1.13\n",
    "from ray.rllib.algorithms.dqn import DQNConfig\n",
    "\n",
    "# Default DQN config values\n",
    "# uncomment below to see the long list of specifically PPO default config values\n",
    "# print(pretty_print(DQNConfig().to_dict()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f0050d3",
   "metadata": {},
   "source": [
    "#### Step 5. Choose your training config settings and instantiate a config object with those settings\n",
    "\n",
    "As of Ray 1.13, RLlib configs been converted from primitive Python dictionaries into Objects. This makes them harder to print, but easier to set/pass.\n",
    "\n",
    "**Note about RLlib training parameter values precedence**\n",
    "<ol>\n",
    "    <li><i><b>Highest</b> precedence</i>: <b>user's config settings at time of training</b>.  These override all other config settings.</li>\n",
    "    <li><i><b>Lower</b> precedence</i>: <b>specific RLlib algorithm (e.g. DQN) config</b>:  \n",
    "        <ol>\n",
    "            <li>Open the <a href=\"https://github.com/ray-project/ray/blob/master/rllib/algorithms/dqn/dqn.py\">algo code file on github</a>.  </li>\n",
    "            <li>Scroll down to the config class <b>__init()__</b> method.</li>\n",
    "            <ol>\n",
    "            <li><i>Algorithm default hyperparameter values are here</i>.</li>\n",
    "            </ol>\n",
    "        </ol>\n",
    "    <li><i><b>Lowest</b></i> precedence: RLlib <b><a href\"https://github.com/ray-project/ray/blob/master/rllib/algorithms/algorithm_config.py\">generic algorithm config</a></b> settings.</li>\n",
    "    </ol>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "7b153924",
   "metadata": {},
   "outputs": [],
   "source": [
    "# RLlib generic (for all algorithms) config values\n",
    "\n",
    "from ray.rllib.algorithms.algorithm_config import AlgorithmConfig\n",
    "config = AlgorithmConfig()\n",
    "\n",
    "# # uncomment below to see the long list of default RLlib AlgorithmConfig values\n",
    "# print(f\"RLlib's general default training config values:\")\n",
    "# print(pretty_print(config.to_dict()))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "276bf275",
   "metadata": {},
   "source": [
    "**Total number of Ray workers(or actors) =**\n",
    "> number of rollout_workers (for streaming the data between learning steps) <br>\n",
    "> \\+ number of evaluation workers (for evaluation between learning steps)<br>\n",
    "> \\+ num_workers <br>\n",
    "> \\+ 1 for head node.<br>\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "    💡 <b>Ray Tune</b>: Total number Ray workers must be <b>(max) ONE LESS THAN</b> total number of available cores or processors. <br>\n",
    "    💡 <b>RLlib algo.train()</b>: Total number Ray workers must be <b>(max) EQUAL TO</b> Total number of available cores or processors.\n",
    "</div>\n",
    "\n",
    "**Note about eval configuration, scaling, and fault tolerance**\n",
    "    \n",
    "    \n",
    "All the above-mentioned scaling Ray num workers are set in the <b>Evaluation config</b>, except rollout_workers, which are set in a separate Rollout config.    \n",
    "\n",
    "    \n",
    "<ul>\n",
    "    <li><b><i>evaluation_interval</i></b> = number training <b>iterations</b> between evaluations</li>\n",
    "    <li><b><i>evaluation_duration</i></b> = number of evaluation <b>iterations</b> used for evaluation</li>\n",
    "    <li><b><i>evaluation_num_workers</i></b> = number extra parallel Ray workers just for evaluation</li>\n",
    "    <ul>\n",
    "        <li>These are in addition to rollout_workers.</li>\n",
    "        <li>Total number iterations used for eval = <i><b>evaluation_duration * evaluation_num_workers</b></i></li>\n",
    "        <li>These show up in the Ray Dashboard as extra \"RolloutWorker\"s.</li>\n",
    "    </ul>\n",
    "    <li><i><b>evaluation_config/num_workers</i></b> = number of extra parallel Ray workers for <b>training</b>.</li>\n",
    "        <ul>\n",
    "        <li>Only 1 possible for DQN (see documentation).  Any number you put here will be translated to 1 during runtime.</li>\n",
    "    </ul>\n",
    "    <li><i><b>evaluation_parallel_to_training</i></b> = whether or not to use the extra parallel Ray workers for evaluation.</li>\n",
    "    <ul>\n",
    "        <li>False by default.</li>\n",
    "    </ul>\n",
    "    <li><i><b>rollouts.num_rollout_workers</i></b> = number of extra parallel Ray workers used just for <b>streaming data</b> per gradient update.</li>\n",
    "    <ul>\n",
    "        <li>This parameter is set in <i>.rollouts</i> not in <i>.evaluation</i>.  </li>\n",
    "    </ul>\n",
    "</ul>\n",
    "\n",
    "\n",
    "Unless you overrode them, view other training parameters using <b><i>DQNConfig().to_dict():</b></i>\n",
    "    <ul>\n",
    "    <li><b><i>train_batch_size</i></b> = number of data samples from replay buffer every gradient update.</li>\n",
    "    <ul>\n",
    "        <li>Default batch_size for DQN = 32</li>\n",
    "    </ul>\n",
    "    <li><b><i>training iteration</i></b> = number of time steps per gradient update.</li>\n",
    "    <ul>\n",
    "        <li> Default timesteps per iteration for DQN = 1000</li>\n",
    "    </ul>\n",
    "        <li>Default learning rate <i><b>lr</i></b> = 5.0e-04</li>\n",
    "        <li><i><b>target_network_update_freq</i></b>: 500 means the frozen-in-time network gets updated only once every 500 time steps </li>\n",
    "    </ul>\n",
    "\n",
    "<b>View network architecture</b> <br>\n",
    "According to the paper\n",
    "> The input to the neural\n",
    "network consists is an 84 × 84 × 4 image produced by φ. The first hidden layer convolves 16 8 × 8\n",
    "filters with stride 4 with the input image and applies a rectifier nonlinearity [10, 18]. The second\n",
    "hidden layer convolves 32 4 × 4 filters with stride 2, again followed by a rectifier nonlinearity. The\n",
    "final hidden layer is fully-connected and consists of 256 rectifier units. The output layer is a fullyconnected linear layer with a single output for each valid action.\n",
    "\n",
    "Use dqn_algo.get_policy().model</li>\n",
    "> Below we can see the TensorFlow model architecture is 2 hidden layers with 256 Dense cells each\n",
    "\n",
    "<img src=\"./images/tf_dqn_algo_dnn_architecture.png\" width=\"80%\" />"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "713dfacc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Config type: <class 'ray.rllib.algorithms.dqn.dqn.DQNConfig'>\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-09-16 21:44:17,186\tWARNING deprecation.py:47 -- DeprecationWarning: `ray.rllib.algorithms.dqn.dqn.DEFAULT_CONFIG` has been deprecated. Use `ray.rllib.algorithms.dqn.dqn.DQNConfig(...)` instead. This will raise an error in the future!\n",
      "2022-09-16 21:44:17,189\tWARNING deprecation.py:47 -- DeprecationWarning: `config['multiagent']['replay_mode']` has been deprecated. config['replay_buffer_config']['replay_mode'] This will raise an error in the future!\n",
      "2022-09-16 21:44:17,194\tINFO worker.py:1221 -- Using address localhost:9031 set in the environment variable RAY_ADDRESS\n",
      "2022-09-16 21:44:17,195\tINFO worker.py:1331 -- Connecting to existing Ray cluster at address: 10.0.117.196:9031...\n",
      "2022-09-16 21:44:17,202\tINFO worker.py:1508 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32mhttps://console.anyscale.com/api/v2/sessions/ses_S99ZQzT4pS1G37w2HU5fADdM/services?redirect_to=dashboard \u001b[39m\u001b[22m\n",
      "2022-09-16 21:44:26,170\tWARNING util.py:66 -- Install gputil for GPU system monitoring.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Algorithm type: <class 'ray.rllib.algorithms.dqn.dqn.DQN'>\n",
      "\n",
      "DQN MODEL ARCHITECTURE:\n"
     ]
    }
   ],
   "source": [
    "# Create a DQNConfig object\n",
    "dqn_config = DQNConfig()\n",
    "\n",
    "# Only for Colab. Specify 1 gpu\n",
    "# dqn_config.num_gpus=1\n",
    "\n",
    "# Setup our config object to use our environment\n",
    "dqn_config.environment(env=\"FrozenLake-v1\")\n",
    "\n",
    "# Decide if you want torch or tensorflow DL framework.  Default is \"tf\"\n",
    "dqn_config.framework(framework=\"torch\")\n",
    "\n",
    "# Set the log level to DEBUG, INFO, WARN, or ERROR \n",
    "dqn_config.debugging(seed=415, log_level=\"ERROR\")\n",
    "\n",
    "# Setup evaluation\n",
    "dqn_config.evaluation(\n",
    "    \n",
    "    # Minimum number of training iterations between evaluations.\n",
    "    # Evaluations are blocking operations (if evaluation_parallel_to_training=False) \n",
    "    # set `evaluation_interval` larger for faster runtime.\n",
    "    evaluation_interval=15, \n",
    "\n",
    "    # Minimum number of evaluation iterations.\n",
    "    # If using multiple evaluation workers, we will run at least \n",
    "    # this many episodes * num_evalworkers total.\n",
    "    evaluation_duration=5,      \n",
    "\n",
    "    # Number of parallel evaluation workers. \n",
    "    # Zero by default, which means evaluation will run on the training resources. \n",
    "    # If you increase this, it will increase total Ray resource usage\n",
    "    # since evaluation workers are created separately from rollout workers \n",
    "    # Note: these show up on Ray Dashboard as extra \"RolloutWorker\"s\n",
    "    evaluation_num_workers=7,  #0 for Colab\n",
    "\n",
    "    # Use the parallel evaluation workers in parallel with training workers\n",
    "    evaluation_parallel_to_training=True,  #False for Colab\n",
    "    \n",
    "    evaluation_config = dict(\n",
    "        # Explicitly set \"explore\"=False to override default True\n",
    "        # Best practice value is False unless environment is stochastic\n",
    "        explore=False,\n",
    "        \n",
    "        # Number of parallel Training workers\n",
    "        # Override the num_workers from the training config \n",
    "        # Note: DQN only allows 1 Trainer worker, see documentation\n",
    "        num_workers=1,  #any number here will be reset = 1 for DQN\n",
    "    ),\n",
    ")\n",
    "\n",
    "# # Override default training parameters\n",
    "# dqn_config.training(target_network_update_freq=5000, \n",
    "#                     model=dict(\"fcnet_hiddens\" : [32, 32])\n",
    "#                    )\n",
    "\n",
    "# Setup sampling rollout workers for streaming the data \n",
    "dqn_config.rollouts(\n",
    "    num_rollout_workers=7,  #1 for Colab\n",
    "    \n",
    "    # for small environments this can be >1 based on size of your processor\n",
    "    num_envs_per_worker=1,)\n",
    "\n",
    "print(f\"Config type: {type(dqn_config)}\")\n",
    "\n",
    "# Use the config object's `build()` method for instantiating\n",
    "# an RLlib Algorithm instance that we can then train.\n",
    "# Note if using Tune, don't need algo object, but this is still a good debugging step.\n",
    "dqn_algo = dqn_config.build()\n",
    "print(f\"Algorithm type: {type(dqn_algo)}\")\n",
    "\n",
    "print()\n",
    "print(\"DQN MODEL ARCHITECTURE:\")\n",
    "# print(result['config']['model'])\n",
    "# # tf print keras model summary\n",
    "# print(dqn_algo.get_policy().model.base_model.summary())\n",
    "# # torch\n",
    "# from torchinfo import summary\n",
    "# summary(dqn_algo.get_policy().model)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "935d28a3",
   "metadata": {},
   "source": [
    "#### Step 6. Train an algorithm using the environment and algorithm config objects\n",
    "\n",
    "**Two ways to train RLlib policies***\n",
    "<ol>\n",
    "    <li><a href=\"https://docs.ray.io/en/master/rllib/package_ref/index.html\">RLlib API.</a> The main methods are:</li>\n",
    "    <ul><b>\n",
    "        <li>train()</li>\n",
    "        <li>save()</li>\n",
    "        <li>evaluate()</li>\n",
    "        <li>restore()</li>\n",
    "        <li>compute_single_action()</li></b>\n",
    "    </ul>\n",
    "    <li><a href=\"https://docs.ray.io/en/master/tune/api_docs/overview.html\">Ray Tune API.</a>  The main methods are:</li>\n",
    "        <ul>\n",
    "            <li><b>fit()</b></li>\n",
    "    </ul>\n",
    "    </ol>\n",
    "    \n",
    "*3rd way is RLlib CLI from command line using .yml file, but the .yml file is undocumented: <i>rllib train -f [myfile_name].yml</i><br>\n",
    "\n",
    "<b>RLlib API train()</b> will train for 1 <i>iteration</i> only.  Good for debugging since every single output will be shown for the single iteration.  \n",
    "\n",
    "<b>Ray Tune API run()</b> is usually more convenient since with 1 function call you get experiment management: hyperparameter tuning, save checkpoints, evaluate, and training up to a stopping criteria.\n",
    "\n",
    "✔Both methods will run the RLlib [environment pre-check function](https://github.com/ray-project/ray/blob/master/rllib/utils/pre_checks/env.py) you saw earlier in this notebook (Step 2. Check environment).\n",
    "\n",
    "<b>RLlib API restore()</b> will reload a checkpointed RLlib model for Serving and Offline learning, even if the model was trained using Tune.  Tune API methods will not work for this.\n",
    "\n",
    "<b>RLlib API compute_single_action()</b> will use the trained <i>`policy`</i> (RL word for trained model) and use that for inference on an environment.   \n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "In summary: <br>\n",
    "    💡 <b>Train</b> a RLlib algorithm with Ray Tune method <b>`.fit()`</b>  <br>\n",
    "    👉  <b>Develop</b> or debug a RLlib algorithm with RLlib method <b>`.train()`</b> <br>\n",
    "    👉  <b>Restore</b> a RLlib policy with RLlib  method <b>`.restore()`</b> <br>\n",
    "    👉  <b>Run inference</b> on an environment using a trained policy with RLlib method <b>`.compute_single_action()`</b>\n",
    "</div>\n",
    "\n",
    "💡 <b>Right-click on the cell below and choose \"Enable Scrolling for Outputs\"!</b>  This will make it easier to view, since model training output can be very long!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f7fbf6e4",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# # SINGLE .TRAIN() OUTPUT\n",
    "# # Check configs before submitting a long-running Tune job.\n",
    "\n",
    "# # Perform single `.train() iteration` call\n",
    "# # Result is a Python dict object\n",
    "# result = dqn_algo.train()\n",
    "\n",
    "# # Erase config dict from result (for better overview).\n",
    "# del result[\"config\"]\n",
    "# # Print out training iteration results.\n",
    "# print(pretty_print(result))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c0e05705",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Algo class for DQN: <class 'ray.rllib.algorithms.dqn.dqn.DQN'>\n",
      "Learning rate: 0.0005\n",
      "Train batch size: 32\n",
      "Gamma: 0.99\n",
      "Target network update freq: 500\n",
      "Eval_num_workers: 1\n",
      "Evaluation_parallel_to_training: True\n",
      "Num_rollout_workers: 3\n",
      "Num_envs_per_worker: 4\n",
      "Training num_workers: 1\n",
      "Model grayscale: False\n",
      "Model zero_mean: True\n"
     ]
    }
   ],
   "source": [
    "# Before setting up the Tune job hyperparam sweep,\n",
    "# Check current parameter settings\n",
    "\n",
    "print(f\"Algo class for DQN: {dqn_config.algo_class}\")\n",
    "print(f\"Learning rate: {dqn_config.lr}\")\n",
    "print(f\"Train batch size: {dqn_config.train_batch_size}\")\n",
    "print(f\"Gamma: {dqn_config.gamma}\")\n",
    "print(f\"Target network update freq: {dqn_config.target_network_update_freq}\")\n",
    "print(f\"Eval_num_workers: {dqn_config.evaluation_num_workers}\")\n",
    "print(f\"Evaluation_parallel_to_training: {dqn_config.evaluation_parallel_to_training}\")\n",
    "print(f\"Num_rollout_workers: {dqn_config.num_workers}\")\n",
    "print(f\"Num_envs_per_worker: {dqn_config.num_envs_per_worker}\")\n",
    "print(f\"Training num_workers: {dqn_config.to_dict()['evaluation_config']['num_workers']}\")\n",
    "print(f\"Model grayscale: {dqn_config.to_dict()['model']['grayscale']}\")\n",
    "print(f\"Model zero_mean: {dqn_config.to_dict()['model']['zero_mean']}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "3e6c7d8d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Default lr is: {'grid_search': [5e-05, 0.0002]}\n"
     ]
    }
   ],
   "source": [
    "# # Now let's change our existing config object and add a simple grid-search\n",
    "\n",
    "# grid search over training params\n",
    "dqn_config.training(\n",
    "    lr=tune.grid_search([0.00005, 0.0002]),\n",
    "    # train_batch_size=tune.grid_search([32, 100]),\n",
    ")\n",
    "print(f\"Default lr is: {dqn_config.lr}\")\n",
    "\n",
    "# other grid searches\n",
    "\n",
    "# # grid search over eval params\n",
    "# dqn_config.evaluation(\n",
    "#     evaluation_num_workers=tune.grid_search([0,1]),\n",
    "# )\n",
    "# print(f\"Default eval_num_workers for DQN is: {dqn_config.evaluation_num_workers}\")\n",
    "\n",
    "# # grid search over rollouts params\n",
    "# dqn_config.rollouts(\n",
    "#     num_rollout_workers=tune.grid_search([1,3,4])\n",
    "#     num_envs_per_worker=tune.grid_search([1,4]),\n",
    "# )\n",
    "# print(f\"Default num_envs_per_worker is: {dqn_config.num_envs_per_worker}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "24ed1326",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div class=\"tuneStatus\">\n",
       "  <div style=\"display: flex;flex-direction: row\">\n",
       "    <div style=\"display: flex;flex-direction: column;\">\n",
       "      <h3>Tune Status</h3>\n",
       "      <table>\n",
       "<tbody>\n",
       "<tr><td>Current time:</td><td>2022-09-16 21:45:35</td></tr>\n",
       "<tr><td>Running for: </td><td>00:01:09.05        </td></tr>\n",
       "<tr><td>Memory:      </td><td>6.3/62.0 GiB       </td></tr>\n",
       "</tbody>\n",
       "</table>\n",
       "    </div>\n",
       "    <div class=\"vDivider\"></div>\n",
       "    <div class=\"systemInfo\">\n",
       "      <h3>System Info</h3>\n",
       "      Using FIFO scheduling algorithm.<br>Resources requested: 0/16 CPUs, 0/0 GPUs, 0.0/34.23 GiB heap, 0.0/17.12 GiB objects\n",
       "    </div>\n",
       "    \n",
       "  </div>\n",
       "  <div class=\"hDivider\"></div>\n",
       "  <div class=\"trialStatus\">\n",
       "    <h3>Trial Status</h3>\n",
       "    <table>\n",
       "<thead>\n",
       "<tr><th>Trial name                   </th><th>status    </th><th>loc                </th><th style=\"text-align: right;\">    lr</th><th style=\"text-align: right;\">  iter</th><th style=\"text-align: right;\">  total time (s)</th><th style=\"text-align: right;\">   ts</th><th style=\"text-align: right;\">  reward</th><th style=\"text-align: right;\">  num_recreated_wor...</th><th style=\"text-align: right;\">  episode_reward_max</th><th style=\"text-align: right;\">  episode_reward_min</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>DQN_FrozenLake-v1_68815_00000</td><td>TERMINATED</td><td>10.0.117.196:564608</td><td style=\"text-align: right;\">5e-05 </td><td style=\"text-align: right;\">    37</td><td style=\"text-align: right;\">         35.9106</td><td style=\"text-align: right;\">37296</td><td style=\"text-align: right;\">    0.58</td><td style=\"text-align: right;\">                     0</td><td style=\"text-align: right;\">                   1</td><td style=\"text-align: right;\">                   0</td></tr>\n",
       "<tr><td>DQN_FrozenLake-v1_68815_00001</td><td>TERMINATED</td><td>10.0.117.196:564802</td><td style=\"text-align: right;\">0.0002</td><td style=\"text-align: right;\">    36</td><td style=\"text-align: right;\">         35.6155</td><td style=\"text-align: right;\">36288</td><td style=\"text-align: right;\">    0.25</td><td style=\"text-align: right;\">                     0</td><td style=\"text-align: right;\">                   1</td><td style=\"text-align: right;\">                   0</td></tr>\n",
       "</tbody>\n",
       "</table>\n",
       "  </div>\n",
       "</div>\n",
       "<style>\n",
       ".tuneStatus {\n",
       "  color: var(--jp-ui-font-color1);\n",
       "}\n",
       ".tuneStatus .systemInfo {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       ".tuneStatus td {\n",
       "  white-space: nowrap;\n",
       "}\n",
       ".tuneStatus .trialStatus {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "}\n",
       ".tuneStatus h3 {\n",
       "  font-weight: bold;\n",
       "}\n",
       ".tuneStatus .hDivider {\n",
       "  border-bottom-width: var(--jp-border-width);\n",
       "  border-bottom-color: var(--jp-border-color0);\n",
       "  border-bottom-style: solid;\n",
       "}\n",
       ".tuneStatus .vDivider {\n",
       "  border-left-width: var(--jp-border-width);\n",
       "  border-left-color: var(--jp-border-color0);\n",
       "  border-left-style: solid;\n",
       "  margin: 0.5em 1em 0.5em 1em;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(DQN pid=564608)\u001b[0m 2022-09-16 21:44:32,323\tWARNING deprecation.py:47 -- DeprecationWarning: `ray.rllib.algorithms.dqn.dqn.DEFAULT_CONFIG` has been deprecated. Use `ray.rllib.algorithms.dqn.dqn.DQNConfig(...)` instead. This will raise an error in the future!\n",
      "\u001b[2m\u001b[36m(DQN pid=564608)\u001b[0m 2022-09-16 21:44:32,325\tWARNING deprecation.py:47 -- DeprecationWarning: `config['multiagent']['replay_mode']` has been deprecated. config['replay_buffer_config']['replay_mode'] This will raise an error in the future!\n",
      "\u001b[2m\u001b[36m(DQN pid=564608)\u001b[0m 2022-09-16 21:44:32,325\tINFO simple_q.py:303 -- In multi-agent mode, policies will be optimized sequentially by the multi-GPU optimizer. Consider setting `simple_optimizer=True` if this doesn't work for you.\n",
      "\u001b[2m\u001b[36m(DQN pid=564608)\u001b[0m 2022-09-16 21:44:32,327\tINFO algorithm.py:357 -- Current log_level is ERROR. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.\n",
      "\u001b[2m\u001b[36m(DQN pid=564608)\u001b[0m 2022-09-16 21:44:42,681\tINFO trainable.py:163 -- Trainable.setup took 10.358 seconds. If your trainable is slow to initialize, consider setting reuse_actors=True to reduce actor creation overheads.\n",
      "\u001b[2m\u001b[36m(DQN pid=564608)\u001b[0m 2022-09-16 21:44:42,681\tWARNING util.py:66 -- Install gputil for GPU system monitoring.\n",
      "\u001b[2m\u001b[36m(DQN pid=564802)\u001b[0m 2022-09-16 21:44:47,916\tWARNING deprecation.py:47 -- DeprecationWarning: `ray.rllib.algorithms.dqn.dqn.DEFAULT_CONFIG` has been deprecated. Use `ray.rllib.algorithms.dqn.dqn.DQNConfig(...)` instead. This will raise an error in the future!\n",
      "\u001b[2m\u001b[36m(DQN pid=564802)\u001b[0m 2022-09-16 21:44:47,917\tWARNING deprecation.py:47 -- DeprecationWarning: `config['multiagent']['replay_mode']` has been deprecated. config['replay_buffer_config']['replay_mode'] This will raise an error in the future!\n",
      "\u001b[2m\u001b[36m(DQN pid=564802)\u001b[0m 2022-09-16 21:44:47,918\tINFO simple_q.py:303 -- In multi-agent mode, policies will be optimized sequentially by the multi-GPU optimizer. Consider setting `simple_optimizer=True` if this doesn't work for you.\n",
      "\u001b[2m\u001b[36m(DQN pid=564802)\u001b[0m 2022-09-16 21:44:47,919\tINFO algorithm.py:357 -- Current log_level is ERROR. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div class=\"trialProgress\">\n",
       "  <h3>Trial Progress</h3>\n",
       "  <table>\n",
       "<thead>\n",
       "<tr><th>Trial name                   </th><th style=\"text-align: right;\">  agent_timesteps_total</th><th>counters                                                                                                                                                                                      </th><th>custom_metrics  </th><th style=\"text-align: right;\">  episode_len_mean</th><th>episode_media  </th><th style=\"text-align: right;\">  episode_reward_max</th><th style=\"text-align: right;\">  episode_reward_mean</th><th style=\"text-align: right;\">  episode_reward_min</th><th style=\"text-align: right;\">  episodes_this_iter</th><th>info  </th><th style=\"text-align: right;\">  num_agent_steps_sampled</th><th style=\"text-align: right;\">  num_agent_steps_trained</th><th style=\"text-align: right;\">  num_env_steps_sampled</th><th style=\"text-align: right;\">  num_env_steps_sampled_this_iter</th><th style=\"text-align: right;\">  num_env_steps_trained</th><th style=\"text-align: right;\">  num_env_steps_trained_this_iter</th><th style=\"text-align: right;\">  num_faulty_episodes</th><th style=\"text-align: right;\">  num_healthy_workers</th><th style=\"text-align: right;\">  num_recreated_workers</th><th style=\"text-align: right;\">  num_steps_trained_this_iter</th><th>perf                                                 </th><th>policy_reward_max  </th><th>policy_reward_mean  </th><th>policy_reward_min  </th><th>sampler_perf                                                                                                                                                                                                     </th><th>sampler_results                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          </th><th>timers                                                                                                                                                                            </th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>DQN_FrozenLake-v1_68815_00000</td><td style=\"text-align: right;\">                  37296</td><td>{&#x27;num_env_steps_sampled&#x27;: 37296, &#x27;num_env_steps_trained&#x27;: 24224, &#x27;num_agent_steps_sampled&#x27;: 37296, &#x27;num_agent_steps_trained&#x27;: 24224, &#x27;last_target_update_ts&#x27;: 36912, &#x27;num_target_updates&#x27;: 69}</td><td>{}              </td><td style=\"text-align: right;\">             39.21</td><td>{}             </td><td style=\"text-align: right;\">                   1</td><td style=\"text-align: right;\">                 0.58</td><td style=\"text-align: right;\">                   0</td><td style=\"text-align: right;\">                  25</td><td>{&#x27;learner&#x27;: {&#x27;default_policy&#x27;: {&#x27;learner_stats&#x27;: {&#x27;allreduce_latency&#x27;: 0.0, &#x27;grad_gnorm&#x27;: 0.008527277037501335, &#x27;mean_q&#x27;: 0.4278366267681122, &#x27;min_q&#x27;: 0.14920161664485931, &#x27;max_q&#x27;: 0.6940673589706421, &#x27;cur_lr&#x27;: 5e-05}, &#x27;td_error&#x27;: array([-8.6868405e-03,  8.7241888e-02, -4.1888475e-02, -1.1860342e-01,\n",
       "        5.6287646e-04, -1.2221342e-01,  2.8638810e-02,  9.1963112e-03,\n",
       "        1.4920162e-01, -4.5209438e-02, -4.5209438e-02, -4.5209438e-02,\n",
       "        5.6287646e-04,  3.7136018e-02, -3.4540892e-05,  1.1093736e-02,\n",
       "        9.1963112e-03,  2.6681781e-02, -4.6372831e-02, -6.0570538e-03,\n",
       "       -4.1888475e-02, -4.5209438e-02,  3.7136018e-02,  8.7241888e-02,\n",
       "        7.5890422e-03,  7.5890422e-03, -2.2122264e-02,  3.2688934e-01,\n",
       "       -4.5209438e-02,  1.5362471e-02,  9.1963112e-03,  1.1093736e-02],\n",
       "      dtype=float32), &#x27;mean_td_error&#x27;: 0.007115486077964306, &#x27;model&#x27;: {}, &#x27;custom_metrics&#x27;: {}, &#x27;num_agent_steps_trained&#x27;: 32.0}}, &#x27;num_env_steps_sampled&#x27;: 37296, &#x27;num_env_steps_trained&#x27;: 24224, &#x27;num_agent_steps_sampled&#x27;: 37296, &#x27;num_agent_steps_trained&#x27;: 24224, &#x27;last_target_update_ts&#x27;: 36912, &#x27;num_target_updates&#x27;: 69}       </td><td style=\"text-align: right;\">                    37296</td><td style=\"text-align: right;\">                    24224</td><td style=\"text-align: right;\">                  37296</td><td style=\"text-align: right;\">                             1008</td><td style=\"text-align: right;\">                  24224</td><td style=\"text-align: right;\">                              672</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    3</td><td style=\"text-align: right;\">                      0</td><td style=\"text-align: right;\">                          672</td><td>{&#x27;cpu_util_percent&#x27;: 31.0, &#x27;ram_util_percent&#x27;: 13.2} </td><td>{}                 </td><td>{}                  </td><td>{}                 </td><td>{&#x27;mean_raw_obs_processing_ms&#x27;: 1.4178441437937044, &#x27;mean_inference_ms&#x27;: 1.5455208531656623, &#x27;mean_action_processing_ms&#x27;: 0.12568857315234733, &#x27;mean_env_wait_ms&#x27;: 0.14243347754283042, &#x27;mean_env_render_ms&#x27;: 0.0}</td><td>{&#x27;episode_reward_max&#x27;: 1.0, &#x27;episode_reward_min&#x27;: 0.0, &#x27;episode_reward_mean&#x27;: 0.58, &#x27;episode_len_mean&#x27;: 39.21, &#x27;episode_media&#x27;: {}, &#x27;episodes_this_iter&#x27;: 25, &#x27;policy_reward_min&#x27;: {}, &#x27;policy_reward_max&#x27;: {}, &#x27;policy_reward_mean&#x27;: {}, &#x27;custom_metrics&#x27;: {}, &#x27;hist_stats&#x27;: {&#x27;episode_reward&#x27;: [1.0, 1.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 1.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 1.0], &#x27;episode_lengths&#x27;: [70, 28, 26, 100, 28, 32, 27, 100, 37, 37, 41, 6, 49, 6, 38, 18, 30, 10, 14, 21, 23, 73, 21, 73, 16, 6, 41, 54, 27, 36, 100, 50, 7, 67, 62, 28, 4, 8, 15, 8, 19, 40, 51, 48, 7, 31, 28, 29, 100, 13, 100, 19, 27, 31, 16, 36, 49, 24, 69, 13, 39, 24, 31, 22, 11, 40, 20, 26, 13, 54, 17, 74, 51, 24, 65, 21, 44, 100, 57, 94, 47, 36, 100, 29, 34, 78, 25, 56, 34, 25, 13, 49, 67, 100, 76, 16, 29, 12, 58, 23]}, &#x27;sampler_perf&#x27;: {&#x27;mean_raw_obs_processing_ms&#x27;: 1.4178441437937044, &#x27;mean_inference_ms&#x27;: 1.5455208531656623, &#x27;mean_action_processing_ms&#x27;: 0.12568857315234733, &#x27;mean_env_wait_ms&#x27;: 0.14243347754283042, &#x27;mean_env_render_ms&#x27;: 0.0}, &#x27;num_faulty_episodes&#x27;: 0}</td><td>{&#x27;training_iteration_time_ms&#x27;: 46.405, &#x27;load_time_ms&#x27;: 0.177, &#x27;load_throughput&#x27;: 180472.943, &#x27;learn_time_ms&#x27;: 7.714, &#x27;learn_throughput&#x27;: 4148.348, &#x27;synch_weights_time_ms&#x27;: 1.598}</td></tr>\n",
       "<tr><td>DQN_FrozenLake-v1_68815_00001</td><td style=\"text-align: right;\">                  36288</td><td>{&#x27;num_env_steps_sampled&#x27;: 36288, &#x27;num_env_steps_trained&#x27;: 23552, &#x27;num_agent_steps_sampled&#x27;: 36288, &#x27;num_agent_steps_trained&#x27;: 23552, &#x27;last_target_update_ts&#x27;: 35856, &#x27;num_target_updates&#x27;: 67}</td><td>{}              </td><td style=\"text-align: right;\">             25.84</td><td>{}             </td><td style=\"text-align: right;\">                   1</td><td style=\"text-align: right;\">                 0.25</td><td style=\"text-align: right;\">                   0</td><td style=\"text-align: right;\">                  39</td><td>{&#x27;learner&#x27;: {&#x27;default_policy&#x27;: {&#x27;learner_stats&#x27;: {&#x27;allreduce_latency&#x27;: 0.0, &#x27;grad_gnorm&#x27;: 0.006625537294894457, &#x27;mean_q&#x27;: 0.5048487186431885, &#x27;min_q&#x27;: 0.3256007134914398, &#x27;max_q&#x27;: 0.659631073474884, &#x27;cur_lr&#x27;: 0.0002}, &#x27;td_error&#x27;: array([ 0.00502101, -0.18356207,  0.0422045 , -0.03005782,  0.00980699,\n",
       "        0.00460023, -0.14961559, -0.02186024, -0.07136136, -0.0737235 ,\n",
       "        0.04242083,  0.07258883,  0.06855002,  0.04006788, -0.04928786,\n",
       "        0.04242083, -0.03005782,  0.0422045 ,  0.00502101,  0.02249536,\n",
       "       -0.03005782,  0.0422045 ,  0.00502101,  0.35059726,  0.06855002,\n",
       "       -0.22094253,  0.07258883,  0.07258883,  0.04006788,  0.19351566,\n",
       "        0.025379  ,  0.04006788], dtype=float32), &#x27;mean_td_error&#x27;: 0.013983008451759815, &#x27;model&#x27;: {}, &#x27;custom_metrics&#x27;: {}, &#x27;num_agent_steps_trained&#x27;: 32.0}}, &#x27;num_env_steps_sampled&#x27;: 36288, &#x27;num_env_steps_trained&#x27;: 23552, &#x27;num_agent_steps_sampled&#x27;: 36288, &#x27;num_agent_steps_trained&#x27;: 23552, &#x27;last_target_update_ts&#x27;: 35856, &#x27;num_target_updates&#x27;: 67}       </td><td style=\"text-align: right;\">                    36288</td><td style=\"text-align: right;\">                    23552</td><td style=\"text-align: right;\">                  36288</td><td style=\"text-align: right;\">                             1008</td><td style=\"text-align: right;\">                  23552</td><td style=\"text-align: right;\">                              672</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    3</td><td style=\"text-align: right;\">                      0</td><td style=\"text-align: right;\">                          672</td><td>{&#x27;cpu_util_percent&#x27;: 27.95, &#x27;ram_util_percent&#x27;: 13.2}</td><td>{}                 </td><td>{}                  </td><td>{}                 </td><td>{&#x27;mean_raw_obs_processing_ms&#x27;: 1.4637645557302914, &#x27;mean_inference_ms&#x27;: 1.5683235257686001, &#x27;mean_action_processing_ms&#x27;: 0.12832067816466045, &#x27;mean_env_wait_ms&#x27;: 0.14487623458436125, &#x27;mean_env_render_ms&#x27;: 0.0}</td><td>{&#x27;episode_reward_max&#x27;: 1.0, &#x27;episode_reward_min&#x27;: 0.0, &#x27;episode_reward_mean&#x27;: 0.25, &#x27;episode_len_mean&#x27;: 25.84, &#x27;episode_media&#x27;: {}, &#x27;episodes_this_iter&#x27;: 39, &#x27;policy_reward_min&#x27;: {}, &#x27;policy_reward_max&#x27;: {}, &#x27;policy_reward_mean&#x27;: {}, &#x27;custom_metrics&#x27;: {}, &#x27;hist_stats&#x27;: {&#x27;episode_reward&#x27;: [1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0, 1.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0], &#x27;episode_lengths&#x27;: [28, 13, 8, 9, 59, 5, 68, 100, 26, 87, 87, 65, 27, 18, 19, 9, 52, 16, 30, 14, 13, 6, 4, 27, 30, 26, 72, 83, 95, 8, 9, 84, 36, 32, 5, 9, 15, 15, 20, 2, 6, 19, 12, 12, 5, 100, 69, 29, 13, 20, 100, 6, 4, 47, 9, 9, 27, 5, 19, 17, 10, 6, 7, 18, 30, 52, 23, 10, 8, 16, 22, 23, 27, 20, 7, 5, 3, 15, 45, 19, 52, 9, 39, 6, 9, 19, 7, 14, 53, 17, 14, 11, 9, 18, 6, 14, 9, 10, 16, 27]}, &#x27;sampler_perf&#x27;: {&#x27;mean_raw_obs_processing_ms&#x27;: 1.4637645557302914, &#x27;mean_inference_ms&#x27;: 1.5683235257686001, &#x27;mean_action_processing_ms&#x27;: 0.12832067816466045, &#x27;mean_env_wait_ms&#x27;: 0.14487623458436125, &#x27;mean_env_render_ms&#x27;: 0.0}, &#x27;num_faulty_episodes&#x27;: 0}                            </td><td>{&#x27;training_iteration_time_ms&#x27;: 44.841, &#x27;load_time_ms&#x27;: 0.226, &#x27;load_throughput&#x27;: 141430.693, &#x27;learn_time_ms&#x27;: 8.279, &#x27;learn_throughput&#x27;: 3865.328, &#x27;synch_weights_time_ms&#x27;: 1.625}</td></tr>\n",
       "</tbody>\n",
       "</table>\n",
       "</div>\n",
       "<style>\n",
       ".trialProgress {\n",
       "  display: flex;\n",
       "  flex-direction: column;\n",
       "  color: var(--jp-ui-font-color1);\n",
       "}\n",
       ".trialProgress h3 {\n",
       "  font-weight: bold;\n",
       "}\n",
       ".trialProgress td {\n",
       "  white-space: nowrap;\n",
       "}\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[2m\u001b[36m(DQN pid=564802)\u001b[0m 2022-09-16 21:44:58,239\tINFO trainable.py:163 -- Trainable.setup took 10.324 seconds. If your trainable is slow to initialize, consider setting reuse_actors=True to reduce actor creation overheads.\n",
      "\u001b[2m\u001b[36m(DQN pid=564802)\u001b[0m 2022-09-16 21:44:58,240\tWARNING util.py:66 -- Install gputil for GPU system monitoring.\n",
      "2022-09-16 21:45:35,727\tINFO tune.py:762 -- Total run time: 69.52 seconds (68.99 seconds for the tuning loop).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training completed.\n"
     ]
    }
   ],
   "source": [
    "# ##############\n",
    "# # EXAMPLE USING RAY TUNE API .fit() 1 UNTIL STOPPING CONDITION\n",
    "# ##############\n",
    "\n",
    "# Stopping criteria whichever occurs first: average (sum) reward, or ...\n",
    "stop_criteria = dict(\n",
    "        # stop after n seconds\n",
    "        time_total_s=35,\n",
    "    \n",
    "        # stop if reached n sampling timesteps\n",
    "        # timesteps_total=9000,  \n",
    "        # stop after n training iterations (calls to `Algorithm.train()`)\n",
    "        # training_iteration=30,\n",
    "        # stop if average (sum of) rewards in an episode is n or more\n",
    "        # episode_reward_mean=0.2,  # 0.2 out of max 0.7 \n",
    ")\n",
    "    \n",
    "\"\"\"\n",
    "Use tune.Tuner(\n",
    "    algo\n",
    "    param_space=..., \n",
    "    run_config=air.RunConfig(\n",
    "        local_dir='./rllib_notebook/offline_rl/bandits',\n",
    "        stop={'training_iteration': 1}\n",
    "    ))\n",
    "    tuner.fit()\n",
    "\"\"\"\n",
    "tuner = tune.Tuner(\n",
    "    # Alternatively, just put the string \"DQN\" here.\n",
    "    # All of RLlib's algos are pre-registered with Tune: e.g. \"PPO\", \"DQN\", \"SAC\", \"IMPALA\", etc..\n",
    "    dqn_config.algo_class,\n",
    "\n",
    "    # training config params (translated into a python dict!)\n",
    "    param_space=dqn_config.to_dict(),\n",
    "\n",
    "    run_config=air.RunConfig(\n",
    "        #redirect logs to relative path instead of default ~/ray_results/\n",
    "        local_dir = \"my_Tune_logs\",\n",
    "        \n",
    "        # Stopping criteria whichever occurs first: average reward over training episodes, or ...\n",
    "        stop=stop_criteria,\n",
    "\n",
    "        ###############\n",
    "        # Note about Ray Tune verbosity.\n",
    "        # Screen verbosity in Ray Tune is defined as verbose = 0, 1, 2, or 3, where:\n",
    "        # 0 = silent\n",
    "        # 1 = only status updates, no logging messages\n",
    "        # 2 = summary, status and brief trial results, includes logging messages\n",
    "        # 3 = summary, status and detailed trial results, includes logging messages\n",
    "        # Defaults to 3.\n",
    "        ###############\n",
    "        verbose=2,\n",
    "        )\n",
    ")\n",
    "experiment_results = tuner.fit()\n",
    "\n",
    "print(\"Training completed.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1c5a0a7",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "⬆️ Above, you will see a lot of Tune output.  Look for a summary table like this:\n",
    "\n",
    "<img src=\"./images/DQN_tune_lr_summary.png\" width=\"100%\" />   \n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bec4632d",
   "metadata": {},
   "source": [
    "## Evaluate a RLlib Policy <a class=\"anchor\" id=\"eval_rllib\"></a>\n",
    "\n",
    "Traditional Supervised ML splits data into train/valid/test, and runs evaluate on the entire valid dataset after the model has been trained.  RL, on the other hand, runs evaluation typically every time a checkpoint is saved.  \n",
    "\n",
    "RLlib policies can be evaluated by:\n",
    "<ul>\n",
    "    <li>Examining <b>Ray Tune experiment results</b>  </li>\n",
    "    <li>Calling RLlib Algorithm API <b>.evaluate()</b> typically every time <b>.save()</b> is called.</li>\n",
    "    <li>Visualize real-time training progress in <b>TensorBoard</b></li>\n",
    "    </ul>\n",
    "\n",
    "<b>If using Ray Tune .run()</b>\n",
    "\n",
    "> The `Tune.run()` method collects results from every iteration and puts them in the output directory where the other logging artifcats are stored. \n",
    "\n",
    "\n",
    "\n",
    "<b>If using RLlib .train()</b>\n",
    "\n",
    "> The `train()` method returns a dictionary. \n",
    "\n",
    "⬇️ Below we will examine the Ray Tune experiment results.\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "66b58780",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "df.shape: (2, 266)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>experiment_id</th>\n",
       "      <th>config/lr</th>\n",
       "      <th>config/gamma</th>\n",
       "      <th>episode_reward_mean</th>\n",
       "      <th>episode_len_mean</th>\n",
       "      <th>timesteps_total</th>\n",
       "      <th>training_iteration</th>\n",
       "      <th>done</th>\n",
       "      <th>time_total_s</th>\n",
       "      <th>timers/training_iteration_time_ms</th>\n",
       "      <th>timers/load_time_ms</th>\n",
       "      <th>timers/load_throughput</th>\n",
       "      <th>timers/learn_time_ms</th>\n",
       "      <th>timers/synch_weights_time_ms</th>\n",
       "      <th>num_rollout_workers</th>\n",
       "      <th>evaluation_num_workers</th>\n",
       "      <th>num_envs_per_eval_worker</th>\n",
       "      <th>num_train_workers</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5115e1ba42b345d7bfbf6dbd66817e82</td>\n",
       "      <td>0.00005</td>\n",
       "      <td>0.99</td>\n",
       "      <td>0.58</td>\n",
       "      <td>39.21</td>\n",
       "      <td>37296</td>\n",
       "      <td>37</td>\n",
       "      <td>True</td>\n",
       "      <td>35.910559</td>\n",
       "      <td>46.405</td>\n",
       "      <td>0.177</td>\n",
       "      <td>180472.943</td>\n",
       "      <td>7.714</td>\n",
       "      <td>1.598</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1c2ec92f08984363bfc3544d994ee439</td>\n",
       "      <td>0.00020</td>\n",
       "      <td>0.99</td>\n",
       "      <td>0.25</td>\n",
       "      <td>25.84</td>\n",
       "      <td>36288</td>\n",
       "      <td>36</td>\n",
       "      <td>True</td>\n",
       "      <td>35.615472</td>\n",
       "      <td>44.841</td>\n",
       "      <td>0.226</td>\n",
       "      <td>141430.693</td>\n",
       "      <td>8.279</td>\n",
       "      <td>1.625</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      experiment_id  config/lr  config/gamma  \\\n",
       "0  5115e1ba42b345d7bfbf6dbd66817e82    0.00005          0.99   \n",
       "1  1c2ec92f08984363bfc3544d994ee439    0.00020          0.99   \n",
       "\n",
       "   episode_reward_mean  episode_len_mean  timesteps_total  training_iteration  \\\n",
       "0                 0.58             39.21            37296                  37   \n",
       "1                 0.25             25.84            36288                  36   \n",
       "\n",
       "   done  time_total_s  timers/training_iteration_time_ms  timers/load_time_ms  \\\n",
       "0  True     35.910559                             46.405                0.177   \n",
       "1  True     35.615472                             44.841                0.226   \n",
       "\n",
       "   timers/load_throughput  timers/learn_time_ms  timers/synch_weights_time_ms  \\\n",
       "0              180472.943                 7.714                         1.598   \n",
       "1              141430.693                 8.279                         1.625   \n",
       "\n",
       "   num_rollout_workers  evaluation_num_workers  num_envs_per_eval_worker  \\\n",
       "0                    3                       1                         4   \n",
       "1                    3                       1                         4   \n",
       "\n",
       "   num_train_workers  \n",
       "0                  1  \n",
       "1                  1  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "TRAIN SETTINGS\n",
      "learning rate: 5e-05\n",
      "batch_size: 32\n",
      "eval_interval: 15\n",
      "Timesteps since last target update: 36912\n",
      "Num target updates: 69\n",
      "\n",
      "TIMINGS\n",
      "Total time (sec) 1st trial: 35.91055917739868\n",
      "Total time (sec) 2nd trial: 35.61547231674194\n"
     ]
    }
   ],
   "source": [
    "# Read tune results in a pandas dataframe\n",
    "df = experiment_results.get_dataframe()\n",
    "print(f\"df.shape: {df.shape}\")  # One row per experiment\n",
    "\n",
    "# pick off col numbers this way\n",
    "# temp = df.columns.tolist()\n",
    "# temp\n",
    "temp_columns = [\n",
    "    \"experiment_id\", \"config/lr\", \"config/gamma\", \"episode_reward_mean\",\n",
    "    \"episode_len_mean\", \"timesteps_total\", \"training_iteration\", \n",
    "    \"done\", \"time_total_s\", \n",
    "    \"timers/training_iteration_time_ms\", \"timers/load_time_ms\",\n",
    "    \"timers/load_throughput\", \"timers/learn_time_ms\", \"timers/synch_weights_time_ms\",\n",
    "    \"config/num_workers\", \"config/evaluation_num_workers\",\n",
    "    \"config/num_envs_per_worker\", \n",
    "    \"config/evaluation_config/num_workers\"\n",
    "]\n",
    "temp = df.loc[:, temp_columns].head()\n",
    "temp.rename(columns={'config/evaluation_config/num_workers':'num_train_workers'}, inplace=True)\n",
    "temp.rename(columns={'config/num_envs_per_worker':'num_envs_per_eval_worker'}, inplace=True)\n",
    "temp.rename(columns={'config/evaluation_num_workers':'evaluation_num_workers'}, inplace=True)\n",
    "temp.rename(columns={'config/num_workers':'num_rollout_workers'}, inplace=True)\n",
    "from IPython.display import display\n",
    "display(temp)\n",
    "\n",
    "print()\n",
    "print(\"TRAIN SETTINGS\")\n",
    "print(f\"learning rate: {df.iloc[0,:]['config/lr']}\")\n",
    "print(f\"batch_size: {df.iloc[0,:]['config/train_batch_size']}\")\n",
    "print(f\"eval_interval: {df.iloc[0,:]['config/evaluation_interval']}\")\n",
    "print(f\"Timesteps since last target update: {df.iloc[0,:]['info/last_target_update_ts']}\")\n",
    "print(f\"Num target updates: {df.iloc[0,:]['info/num_target_updates']}\")\n",
    "\n",
    "print()\n",
    "print(\"TIMINGS\")\n",
    "print(f\"Total time (sec) 1st trial: {df.iloc[0,:]['time_total_s']}\")\n",
    "print(f\"Total time (sec) 2nd trial: {df.iloc[1,:]['time_total_s']}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53ffd00e",
   "metadata": {},
   "source": [
    "<br>\n",
    "\n",
    "**Iterate based on Tune results**\n",
    "\n",
    "From above, only running for <1 minute, we can see that the best configuration is\n",
    "- learning rate = 0.00005\n",
    "\n",
    "Next, let's train longer with these settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "254bafeb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# To start fresh, restart Ray in case it is already running\n",
    "if ray.is_initialized():\n",
    "    ray.shutdown()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4804579d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-09-16 21:45:36,699\tINFO worker.py:1221 -- Using address localhost:9031 set in the environment variable RAY_ADDRESS\n",
      "2022-09-16 21:45:36,700\tINFO worker.py:1331 -- Connecting to existing Ray cluster at address: 10.0.117.196:9031...\n",
      "2022-09-16 21:45:36,705\tINFO worker.py:1508 -- Connected to Ray cluster. View the dashboard at \u001b[1m\u001b[32mhttps://console.anyscale.com/api/v2/sessions/ses_S99ZQzT4pS1G37w2HU5fADdM/services?redirect_to=dashboard \u001b[39m\u001b[22m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Config type: <class 'ray.rllib.algorithms.dqn.dqn.DQNConfig'>\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-09-16 21:45:46,192\tWARNING util.py:66 -- Install gputil for GPU system monitoring.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Algorithm type: <class 'ray.rllib.algorithms.dqn.dqn.DQN'>\n"
     ]
    }
   ],
   "source": [
    "# Change config settings\n",
    "# Create a DQNConfig object\n",
    "dqn_config2 = DQNConfig()\\\n",
    "    .environment(env=\"FrozenLake-v1\")\\\n",
    "    .framework(framework=\"torch\")\\\n",
    "    .debugging(seed=415, log_level=\"ERROR\")\\\n",
    "    .evaluation(\n",
    "        evaluation_interval=15, \n",
    "        evaluation_duration=5,      \n",
    "        evaluation_num_workers=7,  #0 for Colab\n",
    "        evaluation_parallel_to_training=True, #False for Colab\n",
    "        evaluation_config = dict(\n",
    "            explore=False,\n",
    "            num_workers=1,  #any number here will be reset = 1 for DQN\n",
    "        ),)\\\n",
    "    .rollouts(\n",
    "        num_rollout_workers=7, #1 for Colab\n",
    "        num_envs_per_worker=1,)\\\n",
    "    .training(\n",
    "        lr=0.00005,)\n",
    "\n",
    "print(f\"Config type: {type(dqn_config2)}\")\n",
    "\n",
    "# Use the config object's `build()` method for instantiating\n",
    "# an RLlib Algorithm instance that we can then train.\n",
    "dqn_algo = dqn_config2.build()\n",
    "print(f\"Algorithm type: {type(dqn_algo)}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "31697fd6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration=0, Mean Reward=0.03+/-0.00\n",
      "Checkpoints saved at saved_runs/dqn/checkpoint_000001\n",
      "Iteration=14, Mean Reward=0.38+/-0.10\n",
      "Checkpoints saved at saved_runs/dqn/checkpoint_000015\n",
      "Iteration=19, Mean Reward=0.59+/-0.24\n",
      "Checkpoints saved at saved_runs/dqn/checkpoint_000020\n",
      "DQN won 59.0 times over 2000 episodes (72940 timesteps)\n",
      "Approx 0.03 wins per episode\n",
      "Training took 29.36 seconds, 0.49 minutes\n"
     ]
    }
   ],
   "source": [
    "###############\n",
    "# EXAMPLE USING RLLIB API .train() IN A LOOP\n",
    "# To train for N number of episodes, you put .train() into a loop, \n",
    "# similar to the way we ran the Gym env.step() in a loop.\n",
    "###############\n",
    "\n",
    "start_time = time.time()\n",
    "\n",
    "# train the Algorithm instance for 20 iterations\n",
    "num_iterations = 20\n",
    "dqn_rewards  = []\n",
    "checkpoint_dir = \"saved_runs/dqn/\"\n",
    "\n",
    "for i in range(num_iterations):\n",
    "    # Call its `train()` method\n",
    "    result = dqn_algo.train()\n",
    "    \n",
    "    # Extract reward from results.\n",
    "    dqn_rewards.append(result[\"episode_reward_mean\"])\n",
    "    \n",
    "    # checkpoint and evaluate every 15 iterations\n",
    "    if ((i % 14 == 0) or (i == num_iterations-1)):\n",
    "        print(f\"Iteration={i}, Mean Reward={result['episode_reward_mean']:.2f}\",end=\"\")\n",
    "        try:\n",
    "            print(f\"+/-{np.std(dqn_rewards ):.2f}\")\n",
    "        except:\n",
    "            print()\n",
    "        # save checkpoint file\n",
    "        checkpoint_file = dqn_algo.save(checkpoint_dir)\n",
    "        print(f\"Checkpoints saved at {checkpoint_file}\")\n",
    "        # evaluate the policy\n",
    "        eval_result = dqn_algo.evaluate()\n",
    "\n",
    "# convert num_iterations to num_episodes\n",
    "num_episodes = len(result[\"hist_stats\"][\"episode_lengths\"]) * num_iterations\n",
    "# convert num_iterations to num_timesteps\n",
    "num_timesteps = sum(result[\"hist_stats\"][\"episode_lengths\"] * num_iterations)\n",
    "# calculate number of wins\n",
    "num_wins = np.sum(result[\"hist_stats\"][\"episode_reward\"])\n",
    "\n",
    "# train time\n",
    "secs = time.time() - start_time\n",
    "print(f\"DQN won {num_wins} times over {num_episodes} episodes ({num_timesteps} timesteps)\")\n",
    "print(f\"Approx {num_wins/num_episodes:.2f} wins per episode\")\n",
    "print(f\"Training took {secs:.2f} seconds, {secs/60.0:.2f} minutes\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc51fc0d",
   "metadata": {},
   "source": [
    "\n",
    "<b>Compare the DQN Training results to Random Baseline <br></b>\n",
    "- DQN Mean Reward=~0.67+/-0.26.  This is much higher than baseline!\n",
    "> Baseline Mean Reward=~0.02+/-0.13 (out of success=0.7) <br>\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "    ✔ <b>DQN mean reward is approx 30x higher than the random baseline! <br>\n",
    "</div>\n",
    "\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "5a9ec9da",
   "metadata": {},
   "outputs": [],
   "source": [
    "# To stop the Algorithm (and Env) and release its blocked resources, use:\n",
    "dqn_algo.stop()\n",
    "\n",
    "# TODO tensorboard plot Q-values for DQN instead of entropy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86f6d3ba",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Visualize the training progress in TensorBoard\n",
    "\n",
    "<b>Ray Tune</b> automatically creates logs for your trained RLlib models that can be visualized in TensorBoard.  Ray Tune logs are stored in the specified redirect `local_dir`; or if none specified then the logs are stored in `~/ray_results/`.\n",
    "\n",
    "<b>RLlib Algorithm .train() requires an explicit .save() step</b> in order to create logs.  The default format for .save() is Ray Tune .json logs compatible with TensorBoard.  Unlike Ray Tune, using .save(), it is only possible to store logs in `~/ray_results/`.  You cannot change the location of the TensorBoard logs.\n",
    "\n",
    "To visualize the performance of your RL policy:\n",
    "\n",
    "<ol>\n",
    "    <li>Open a terminal</li>\n",
    "    <li><i><b>cd</b></i> into the correct log directory.</li>\n",
    "    <li><i><b>ls</b></i></li>\n",
    "    <li>You should see files such as: <i>result.json, params.json, ... </i></li>\n",
    "    <li>To be able to compare all your experiments, cd one dir level up.\n",
    "    <li><i><b>cd ..</b></i>  \n",
    "    <li><i><b>tensorboard --logdir . </b></i></li>\n",
    "    <li>Look at the url in the message, and open it in a browser</li>\n",
    "        </ol>\n",
    "        \n",
    "Note Step 7 above: if running RLlib on a cluster, use <a href=\"https://blog.tensorflow.org/2019/12/introducing-tensorboarddev-new-way-to.html\">tensorboard.dev</a> instead.  Navigate to the directory on the head node where `ray_results/` directory is located.  From there, run \n",
    "`tensorboard dev upload --logdir .`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "edf58a74",
   "metadata": {},
   "source": [
    "#### Tensorboard\n",
    "\n",
    "TensorBoard will give you many pages of charts.  Most of the charts will be showing Train/Eval <b>sample efficiency</b>, <i>the number of training steps it took to achieve a certain level of performance</i>.\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "    <b>A few charts you will want to inspect:</b>\n",
    "<ol>\n",
    "    <li>View training mean episode reward with </li>\n",
    "        <ul> \n",
    "            <li><b>x-axis step</b>.  This shows the whole learning curve.</li>\n",
    "            <li><b>x-axis relative</b>. Look to the far right of the chart, here you can quickly pick off which model got the highest mean episode reward. \n",
    "            <li>Use the tensorboard menu on left-hand side to toggle between x-axis views.</li>\n",
    "        </ul>\n",
    "        <li>If comparing different training runs, check that <b>rank order of mean episode rewards per model matches between training and evaluation</b>.</li>\n",
    "        <ul>\n",
    "            <li>Training charts usually on page 1.</li>\n",
    "            <li>Evaluation charts usually on page 2.</li>\n",
    "        </ul>\n",
    "    <li>View <b>training entropy</b>.  It should be generally decreasing.  \n",
    "        <ul>\n",
    "            <li>Entropy charts usually on page 3.</li>\n",
    "            <li>If you see entropy decreasing but then at some point flatten or even increase, it means you should stop training earlier.</li>\n",
    "        </ul>\n",
    "    <li>Toggle the ✔ checkbox <i>`Ignore outliers in chart scaling`</i>, in case you have 1 model way outperforming other policies.  The checkbox is located on the top-left menu.</li>\n",
    "    <li>💡 When viewing final model outputs per chart, make sure you are hovering far enought to the right to see a filled-color-dot at the end of each line chart.  This means you are viewing the final, overall metrics for that chart. </li>\n",
    "</ol>\n",
    "</div>\n",
    "    \n",
    "   \n",
    "<b>TensorBoard Screenshots:</b> <br>  \n",
    "<img src=\"./images/frozen_lake_v1_tensorboard.png\" width=\"80%\" />    \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2cfaf0e",
   "metadata": {},
   "source": [
    "## Reload RLlib policy from checkpoint and run inference <a class=\"anchor\" id=\"reload_rllib\"></a>\n",
    "\n",
    "We want to reload the desired RLlib model from checkpoint file and then run the policy in inference mode on the environment it was trained on.  \n",
    "\n",
    "You will need:\n",
    "<ul>\n",
    "    <li>Your <b>algorithm's config class</b></li>\n",
    "    <li>Name of the <b>environment</b> you used to train the policy.</li>\n",
    "    <li>Path to the desired <b>checkpoint</b> file you want to use to restore the policy.</li>\n",
    "    </ul>\n",
    "\n",
    "#### Step 1. Find the best model checkpoint file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "8ab51c7b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "./saved_runs/dqn/checkpoint_000020\n"
     ]
    }
   ],
   "source": [
    "# EXAMPLE GETTING CHECKPOINT FROM RLLIB TRAIN\n",
    "\n",
    "# Enter the last checkpoint manually\n",
    "checkpoint = \"./saved_runs/dqn/checkpoint_000020\"\n",
    "print(f\"\\n{checkpoint}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10534ab0",
   "metadata": {},
   "source": [
    "#### Step 2. Re-initialize an already-trained algorithm object from the checkpoint file\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2808e08",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create new Algorithm and restore its state from the last checkpoint.\n",
    "\n",
    "# create an empty Algorithm\n",
    "algo = dqn_config2.build()\n",
    "\n",
    "# restore the agent from the checkpoint\n",
    "algo.restore(checkpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a722c389",
   "metadata": {},
   "source": [
    "#### Step 3. Play and render the game\n",
    "\n",
    "Now we want to play the trained policy doing inference in the environment it was trained on.\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "✔ During inference, call the RLlib API method <b>compute_single_action()</b>: <br>\n",
    "\n",
    "👍 Uses the trained <i>policy</i> (RL word for trained model) to calculate actions for the entire number of time steps in 1 <i>rollout</i> (RLlib word for episode during inference). \n",
    "</div>\n",
    "\n",
    "⬇️ Below we play the game 100 times using the DQN already-trained policy.\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4d4866b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#############\n",
    "## Create the env to do inference on\n",
    "#############\n",
    "env = gym.make(env_name)\n",
    "obs = env.reset()\n",
    "\n",
    "# Use the restored algorithm from checkpoint and run it in inference mode\n",
    "episode_reward = 0.0\n",
    "done = False\n",
    "num_episodes = 0\n",
    "num_steps = 0\n",
    "\n",
    "while num_episodes < 100:\n",
    "    # Compute an action (`a`).\n",
    "    a = algo.compute_single_action(observation=obs, explore=False)\n",
    "    # Send the computed action `a` to the env.\n",
    "    obs, reward, done, _ = env.step(a)\n",
    "    episode_reward += reward\n",
    "    num_steps += 1\n",
    "    \n",
    "    # Is the episode `done`? -> Reset.\n",
    "    if done:\n",
    "        obs = env.reset()\n",
    "        num_episodes += 1\n",
    "\n",
    "# calculate mean_reward\n",
    "print()\n",
    "print(\"**************\")\n",
    "mean_reward = episode_reward / num_episodes\n",
    "print(f\"DQN mean_reward: {mean_reward:.2f} out of success: {env_spec.reward_threshold} after {num_episodes} episodes or {num_steps} time steps\")\n",
    "print(f\"DQN won {episode_reward} times over {num_episodes} plays (episodes)\")\n",
    "print(\"**************\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc575697",
   "metadata": {},
   "source": [
    "<b>How does our inferenced policy compare to the Random baseline? <br></b>\n",
    "- DQN wins ~70 times over 100 plays.  This is much higher than baseline!\n",
    "> Baseline won ~53.0 times over 3000 plays (episodes) <br>\n",
    "\n",
    "\n",
    "⬇️ Below we render the game using the DQN policy, so we can visually inspect the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "403d2378",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ipywidgets import Output\n",
    "from IPython import display\n",
    "import time\n",
    "\n",
    "# The following three lines are for rendering purposes only.\n",
    "# They allow us to render the env frame-by-frame in-place\n",
    "# (w/o creating a huge output which we would then have to scroll through).\n",
    "out = Output()\n",
    "display.display(out)\n",
    "with out:\n",
    "\n",
    "    #############\n",
    "    ## Create the env to do inference on\n",
    "    #############\n",
    "    env = gym.make(env_name)\n",
    "    obs = env.reset()\n",
    "\n",
    "    #############\n",
    "    ## Use the restored policy and run it in inference mode\n",
    "    ## Run compute_single_action() in inference episodes loop\n",
    "    ## You will see an ASCII rendering in-place for about 10 seconds\n",
    "    #############\n",
    "    episode_reward = 0.0\n",
    "    done = False\n",
    "    num_episodes = 0\n",
    "\n",
    "    while num_episodes < 5:\n",
    "        # Compute an action (`a`).\n",
    "        a = algo.compute_single_action(observation=obs, explore=False)\n",
    "        # Send the computed action `a` to the env.\n",
    "        obs, reward, done, _ = env.step(a)\n",
    "        episode_reward += reward\n",
    "\n",
    "        # Is the episode `done`? -> Reset.\n",
    "        if done:\n",
    "            obs = env.reset()\n",
    "            num_episodes += 1\n",
    "\n",
    "        # Render the env (in place).\n",
    "        time.sleep(0.3)\n",
    "        out.clear_output(wait=True)\n",
    "        print(f\"episode: {num_episodes}\")\n",
    "        print(f\"obs: {obs}, reward: {episode_reward}, done: {done}\")\n",
    "        env.render()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ed12df2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# To stop the Algorithm (and Env) and release its blocked resources, use:            \n",
    "algo.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db7fffa4",
   "metadata": {},
   "source": [
    "### Summary\n",
    "\n",
    "In this notebook, we have learned:\n",
    "* What a gym Environment is, and how the gym.Env API is used define sequential decision making problems using python code\n",
    "* How RLlib looks like on the surface (where to find its algorithms and top-level APIs)\n",
    "* How to train a RLlib algorithm using `.train()` and a built-in gym.Env (\"frozen lake\")\n",
    "* Where to find checkpoint files, logs, tensorboard files, etc..\n",
    "* How to play and render some episodes from a gym.Env using a trained RLlib algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27a10c55",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "1. [Reinforcement Learning: an introduction, by Sutton and Barto, book free download](http://incompleteideas.net/book/the-book-2nd.html)\n",
    "2. [Anyscale tutorial blog explanation of Deep Q-Learning (DQN)](https://www.anyscale.com/blog/reinforcement-learning-with-deep-q-networks)\n",
    "3. [OpenAI Gym Environments](https://www.gymlibrary.dev/)\n",
    "4. [Ray doc page](https://docs.ray.io/en/master/)\n",
    "5. [Rllib github](https://github.com/ray-project/ray/tree/master/rllib)\n",
    "6. [RLlib Algorithms doc page](https://docs.ray.io/en/master/rllib/rllib-algorithms.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb6009d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Shut down Ray if you are done\n",
    "import ray\n",
    "if ray.is_initialized():\n",
    "    ray.shutdown()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e173a12",
   "metadata": {},
   "source": [
    "## Thank you!\n",
    "\n",
    "<a href=\"https://docs.google.com/forms/d/1pxsMIPMxTTd2HH6710UOApx_smPDPPO0fpVWYKzvOgI/edit\">Survey</a> - Let us know how useful you have found this tutorial.\n",
    "\n",
    "**We would love to connect with you!**\n",
    "\n",
    "**Twitter** - @anyscalecompute | @raydistributed <br>\n",
    "<b><a href=\"https://github.com/ray-project/ray\">Github</a></b> - 😜 give us a star!<br>\n",
    "<b><a href=\"https://www.ray.io/community\">Slack</a></b> - [+invitation link](https://docs.google.com/forms/d/e/1FAIpQLSfAcoiLCHOguOm8e7Jnn-JJdZaCxPGjgVCvFijHB5PLaQLeig/viewform)<br>\n",
    "<b><a href=\"https://discuss.ray.io/\">Discuss</a></b> - searchable questions <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6052577c",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}