4_callbacks_hyperparameter_tuning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "4_callbacks_hyperparameter_tuning.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/araffin/rl-tutorial-jnrr19/blob/master/4_callbacks_hyperparameter_tuning.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "i8lIXBiHRYb6"
      },
      "source": [
        "# Stable Baselines Tutorial - Callbacks and hyperparameter tuning\n",
        "\n",
        "Github repo: https://github.com/araffin/rl-tutorial-jnrr19\n",
        "\n",
        "Stable-Baselines: https://github.com/hill-a/stable-baselines\n",
        "\n",
        "Documentation: https://stable-baselines.readthedocs.io/en/master/\n",
        "\n",
        "RL Baselines zoo: https://github.com/araffin/rl-baselines-zoo\n",
        "\n",
        "\n",
        "## Introduction\n",
        "\n",
        "In this notebook, you will learn how to use [Callbacks](https://stable-baselines.readthedocs.io/en/master/guide/callbacks.html) which allow to do monitoring, auto saving, model manipulation, progress bars, ...\n",
        "\n",
        "\n",
        "You will also see that finding good hyperparameters is key to success in RL.\n",
        "\n",
        "## Install Dependencies and Stable Baselines Using Pip"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "owKXXp8rRZI7",
        "colab": {}
      },
      "source": [
        "# Stable Baselines only supports tensorflow 1.x for now\n",
        "%tensorflow_version 1.x\n",
        "!apt install swig\n",
        "!pip install tqdm==4.36.1\n",
        "!pip install stable-baselines[mpi]==2.10.0"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "18ivrnsaSWUn",
        "colab": {}
      },
      "source": [
        "import gym\n",
        "from stable_baselines import A2C, SAC, PPO2, TD3"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "PytOtL9GdmrE"
      },
      "source": [
        "# The importance of hyperparameter tuning\n",
        "\n",
        "When compared with Supervised Learning, Deep Reinforcement Learning is far more sensitive to the choice of hyper-parameters such as learning rate, number of neurons, number of layers, optimizer ... etc. \n",
        "Poor choice of hyper-parameters can lead to poor/unstable convergence. This challenge is compounded by the variability in performance across random seeds (used to initialize the network weights and the environment).\n",
        "\n",
        "Here we demonstrate on a toy example the [Soft Actor Critic](https://arxiv.org/abs/1801.01290) algorithm applied in the Pendulum environment. Note the change in performance between the default and \"tuned\" parameters. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "w5oVvYHwdnYv",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "from stable_baselines.common.evaluation import evaluate_policy"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "-a0v3fgwe54j",
        "colab": {}
      },
      "source": [
        "eval_env = gym.make('Pendulum-v0')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "5WRR7kmIeqEB",
        "colab": {}
      },
      "source": [
        "default_model = SAC('MlpPolicy', 'Pendulum-v0', verbose=1, seed=0).learn(8000)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "jQbDcbEheqWj",
        "outputId": "23732cca-9d4d-4cd9-cc44-116c9f1bd98b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "mean_reward, std_reward = evaluate_policy(default_model, eval_env, n_eval_episodes=100)\n",
        "print(f\"mean_reward:{mean_reward:.2f} +/- {std_reward:.2f}\")"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "mean_reward:-1236.50 +/- 231.99\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "smMdkZnvfL1g",
        "colab": {}
      },
      "source": [
        "tuned_model = SAC('MlpPolicy', 'Pendulum-v0', batch_size=256, verbose=1, policy_kwargs=dict(layers=[256, 256]), seed=0).learn(8000)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "DN05_Io8fMAr",
        "outputId": "1d85dd25-c0ea-4bb6-a02a-a846dba0bf61",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "source": [
        "mean_reward, std_reward = evaluate_policy(tuned_model, eval_env, n_eval_episodes=100)\n",
        "print(f\"mean_reward:{mean_reward:.2f} +/- {std_reward:.2f}\")"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "mean_reward:-143.51 +/- 86.19\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pi9IwxBYVMl8"
      },
      "source": [
        "Exploring hyperparameter tuning is out of the scope (and time schedule) of this tutorial. However, you need to know that we provide tuned hyperparameter in the [rl zoo](https://github.com/araffin/rl-baselines-zoo) as well as automatic hyperparameter optimization using [Optuna](https://github.com/pfnet/optuna).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "irHk8FXdRUnw"
      },
      "source": [
        "# Callbacks\n",
        "\n",
        "\n",
        "Please read the [documentation](https://stable-baselines.readthedocs.io/en/master/guide/callbacks.html). Although Stable-Baselines provides you with a callback collection (e.g. for creating checkpoints or for evaluation), we are going to re-implement some so you can get a good understanding of how they work.\n",
        "\n",
        "To build a custom callback, you need to create a class that derives from `BaseCallback`. This will give you access to events (`_on_training_start`, `_on_step()`) and useful variables (like `self.model` for the RL model).\n",
        "\n",
        "`_on_step` returns a boolean value for whether or not the training should continue.\n",
        "\n",
        "Thanks to the access to the models variables, in particular `self.model`, we are able to even change the parameters of the model without halting the training, or changing the model's code."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uE30k2i7kohh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from stable_baselines.common.callbacks import BaseCallback"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wjRvJ8zBftL3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class CustomCallback(BaseCallback):\n",
        "    \"\"\"\n",
        "    A custom callback that derives from ``BaseCallback``.\n",
        "\n",
        "    :param verbose: (int) Verbosity level 0: not output 1: info 2: debug\n",
        "    \"\"\"\n",
        "    def __init__(self, verbose=0):\n",
        "        super(CustomCallback, self).__init__(verbose)\n",
        "        # Those variables will be accessible in the callback\n",
        "        # (they are defined in the base class)\n",
        "        # The RL model\n",
        "        # self.model = None  # type: BaseRLModel\n",
        "        # An alias for self.model.get_env(), the environment used for training\n",
        "        # self.training_env = None  # type: Union[gym.Env, VecEnv, None]\n",
        "        # Number of time the callback was called\n",
        "        # self.n_calls = 0  # type: int\n",
        "        # self.num_timesteps = 0  # type: int\n",
        "        # local and global variables\n",
        "        # self.locals = None  # type: Dict[str, Any]\n",
        "        # self.globals = None  # type: Dict[str, Any]\n",
        "        # The logger object, used to report things in the terminal\n",
        "        # self.logger = None  # type: logger.Logger\n",
        "        # # Sometimes, for event callback, it is useful\n",
        "        # # to have access to the parent object\n",
        "        # self.parent = None  # type: Optional[BaseCallback]\n",
        "\n",
        "    def _on_training_start(self) -> None:\n",
        "        \"\"\"\n",
        "        This method is called before the first rollout starts.\n",
        "        \"\"\"\n",
        "        pass\n",
        "\n",
        "    def _on_rollout_start(self) -> None:\n",
        "        \"\"\"\n",
        "        A rollout is the collection of environment interaction\n",
        "        using the current policy.\n",
        "        This event is triggered before collecting new samples.\n",
        "        \"\"\"\n",
        "        pass\n",
        "\n",
        "    def _on_step(self) -> bool:\n",
        "        \"\"\"\n",
        "        This method will be called by the model after each call to `env.step()`.\n",
        "\n",
        "        For child callback (of an `EventCallback`), this will be called\n",
        "        when the event is triggered.\n",
        "\n",
        "        :return: (bool) If the callback returns False, training is aborted early.\n",
        "        \"\"\"\n",
        "        return True\n",
        "\n",
        "    def _on_rollout_end(self) -> None:\n",
        "        \"\"\"\n",
        "        This event is triggered before updating the policy.\n",
        "        \"\"\"\n",
        "        pass\n",
        "\n",
        "    def _on_training_end(self) -> None:\n",
        "        \"\"\"\n",
        "        This event is triggered before exiting the `learn()` method.\n",
        "        \"\"\"\n",
        "        pass"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OqpPtxaCfynB",
        "colab_type": "text"
      },
      "source": [
        "Here we have a simple callback that can only be called twice:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7ILY0AkFfzPJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class SimpleCallback(BaseCallback):\n",
        "    \"\"\"\n",
        "    a simple callback that can only be called twice\n",
        "\n",
        "    :param verbose: (int) Verbosity level 0: not output 1: info 2: debug\n",
        "    \"\"\"\n",
        "    def __init__(self, verbose=0):\n",
        "        super(SimpleCallback, self).__init__(verbose)\n",
        "        self._called = False\n",
        "    \n",
        "    def _on_step(self):\n",
        "      if not self._called:\n",
        "        print(\"callback - first call\")\n",
        "        self._called = True\n",
        "        return True # returns True, training continues.\n",
        "      print(\"callback - second call\")\n",
        "      return False # returns False, training stops.      "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "5gTXaNLARUnw",
        "colab": {}
      },
      "source": [
        "model = SAC('MlpPolicy', 'Pendulum-v0', verbose=1)\n",
        "model.learn(8000, callback=SimpleCallback())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "adsKMvDkRUn0"
      },
      "source": [
        "## First example: Auto saving best model\n",
        "In RL, it is quite useful to keep a clean version of a model as you are training, as we can end up with burn-in of a bad policy. This is a typical use case for callback, as they can call the save function of the model, and observe the training over time.\n",
        "\n",
        "Using the monitoring wrapper, we can save statistics of the environment, and use them to determine the mean training reward.\n",
        "This allows us to save the best model while training.\n",
        "\n",
        "Note that this is not the proper way of evaluating an RL agent, you should create an test environment and evaluate the agent performance in the callback (cf `EvalCallback`). For simplicity, we will be using the training reward as a proxy."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IDI3lKTiiKP9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import os\n",
        "\n",
        "import numpy as np\n",
        "\n",
        "from stable_baselines.bench import Monitor\n",
        "from stable_baselines.common.vec_env import DummyVecEnv\n",
        "from stable_baselines.common.cmd_util import make_vec_env\n",
        "from stable_baselines.results_plotter import load_results, ts2xy"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nzMHj7r3h78m",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class SaveOnBestTrainingRewardCallback(BaseCallback):\n",
        "    \"\"\"\n",
        "    Callback for saving a model (the check is done every ``check_freq`` steps)\n",
        "    based on the training reward (in practice, we recommend using ``EvalCallback``).\n",
        "\n",
        "    :param check_freq: (int)\n",
        "    :param log_dir: (str) Path to the folder where the model will be saved.\n",
        "      It must contains the file created by the ``Monitor`` wrapper.\n",
        "    :param verbose: (int)\n",
        "    \"\"\"\n",
        "    def __init__(self, check_freq, log_dir, verbose=1):\n",
        "        super(SaveOnBestTrainingRewardCallback, self).__init__(verbose)\n",
        "        self.check_freq = check_freq\n",
        "        self.log_dir = log_dir\n",
        "        self.save_path = os.path.join(log_dir, 'best_model')\n",
        "        self.best_mean_reward = -np.inf\n",
        "\n",
        "    def _init_callback(self) -> None:\n",
        "        # Create folder if needed\n",
        "        if self.save_path is not None:\n",
        "            os.makedirs(self.save_path, exist_ok=True)\n",
        "\n",
        "    def _on_step(self) -> bool:\n",
        "        if self.n_calls % self.check_freq == 0:\n",
        "\n",
        "          # Retrieve training reward\n",
        "          x, y = ts2xy(load_results(self.log_dir), 'timesteps')\n",
        "          if len(x) > 0:\n",
        "              # Mean training reward over the last 100 episodes\n",
        "              mean_reward = np.mean(y[-100:])\n",
        "              if self.verbose > 0:\n",
        "                print(\"Num timesteps: {}\".format(self.num_timesteps))\n",
        "                print(\"Best mean reward: {:.2f} - Last mean reward per episode: {:.2f}\".format(self.best_mean_reward, mean_reward))\n",
        "\n",
        "              # New best model, you could save the agent here\n",
        "              if mean_reward > self.best_mean_reward:\n",
        "                  self.best_mean_reward = mean_reward\n",
        "                  # Example for saving best model\n",
        "                  if self.verbose > 0:\n",
        "                    print(\"Saving new best model at {} timesteps\".format(x[-1]))\n",
        "                    print(\"Saving new best model to {}.zip\".format(self.save_path))\n",
        "                  self.model.save(self.save_path)\n",
        "\n",
        "        return True"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "1TuYLBEaRUn0",
        "colab": {}
      },
      "source": [
        "# Create log dir\n",
        "log_dir = \"/tmp/gym/\"\n",
        "os.makedirs(log_dir, exist_ok=True)\n",
        "\n",
        "# Create and wrap the environment\n",
        "env = make_vec_env('CartPole-v1', n_envs=1, monitor_dir=log_dir)\n",
        "# it is equivalent to:\n",
        "# env = gym.make('CartPole-v1')\n",
        "# env = Monitor(env, log_dir)\n",
        "# env = DummyVecEnv([lambda: env])\n",
        "\n",
        "# Create Callback\n",
        "callback = SaveOnBestTrainingRewardCallback(check_freq=20, log_dir=log_dir, verbose=1)\n",
        "\n",
        "model = A2C('MlpPolicy', env, verbose=0)\n",
        "model.learn(total_timesteps=10000, callback=callback)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Mx18FkEORUn3"
      },
      "source": [
        "## Second example: Realtime plotting of performance\n",
        "While training, it is sometimes useful to how the training progresses over time, relative to the episodic reward.\n",
        "For this, Stable-Baselines has [Tensorboard support](https://stable-baselines.readthedocs.io/en/master/guide/tensorboard.html), however this can be very combersome, especially in disk space usage. \n",
        "\n",
        "**NOTE: Unfortunately live plotting does not work out of the box on google colab**\n",
        "\n",
        "Here, we can use callback again, to plot the episodic reward in realtime, using the monitoring wrapper:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "c0Bu1HWKRUn4",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "%matplotlib notebook\n",
        "\n",
        "\n",
        "class PlottingCallback(BaseCallback):\n",
        "    \"\"\"\n",
        "    Callback for plotting the performance in realtime.\n",
        "\n",
        "    :param verbose: (int)\n",
        "    \"\"\"\n",
        "    def __init__(self, verbose=1):\n",
        "        super(PlottingCallback, self).__init__(verbose)\n",
        "        self._plot = None\n",
        "\n",
        "    def _on_step(self) -> bool:\n",
        "        # get the monitor's data\n",
        "        x, y = ts2xy(load_results(log_dir), 'timesteps')\n",
        "      if self._plot is None: # make the plot\n",
        "          plt.ion()\n",
        "          fig = plt.figure(figsize=(6,3))\n",
        "          ax = fig.add_subplot(111)\n",
        "          line, = ax.plot(x, y)\n",
        "          self._plot = (line, ax, fig)\n",
        "          plt.show()\n",
        "      else: # update and rescale the plot\n",
        "          self._plot[0].set_data(x, y)\n",
        "          self._plot[-2].relim()\n",
        "          self._plot[-2].set_xlim([self.locals[\"total_timesteps\"] * -0.02, \n",
        "                                   self.locals[\"total_timesteps\"] * 1.02])\n",
        "          self._plot[-2].autoscale_view(True,True,True)\n",
        "          self._plot[-1].canvas.draw()\n",
        "        \n",
        "# Create log dir\n",
        "log_dir = \"/tmp/gym/\"\n",
        "os.makedirs(log_dir, exist_ok=True)\n",
        "\n",
        "# Create and wrap the environment\n",
        "env = make_vec_env('MountainCarContinuous-v0', n_envs=1, monitor_dir=log_dir)\n",
        "\n",
        "plotting_callback = PlottingCallback()\n",
        "        \n",
        "model = PPO2('MlpPolicy', env, verbose=0)\n",
        "model.learn(20000, callback=plotting_callback)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "49RVX7ieRUn7"
      },
      "source": [
        "## Third example: Progress bar\n",
        "Quality of life improvement are always welcome when developping and using RL. Here, we used [tqdm](https://tqdm.github.io/) to show a progress bar of the training, along with number of timesteps per second and the estimated time remaining to the end of the training:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "pXa8f6FsRUn8",
        "colab": {}
      },
      "source": [
        "from tqdm.auto import tqdm\n",
        "\n",
        "class ProgressBarCallback(BaseCallback):\n",
        "    \"\"\"\n",
        "    :param pbar: (tqdm.pbar) Progress bar object\n",
        "    \"\"\"\n",
        "    def __init__(self, pbar):\n",
        "        super(ProgressBarCallback, self).__init__()\n",
        "        self._pbar = pbar\n",
        "\n",
        "    def _on_step(self):\n",
        "        # Update the progress bar:\n",
        "        self._pbar.n = self.num_timesteps\n",
        "        self._pbar.update(0)\n",
        "\n",
        "# this callback uses the 'with' block, allowing for correct initialisation and destruction\n",
        "class ProgressBarManager(object):\n",
        "    def __init__(self, total_timesteps): # init object with total timesteps\n",
        "        self.pbar = None\n",
        "        self.total_timesteps = total_timesteps\n",
        "        \n",
        "    def __enter__(self): # create the progress bar and callback, return the callback\n",
        "        self.pbar = tqdm(total=self.total_timesteps)\n",
        "            \n",
        "        return ProgressBarCallback(self.pbar)\n",
        "\n",
        "    def __exit__(self, exc_type, exc_val, exc_tb): # close the callback\n",
        "        self.pbar.n = self.total_timesteps\n",
        "        self.pbar.update(0)\n",
        "        self.pbar.close()\n",
        "        \n",
        "model = TD3('MlpPolicy', 'Pendulum-v0', verbose=0)\n",
        "with ProgressBarManager(2000) as callback: # this the garanties that the tqdm progress bar closes correctly\n",
        "    model.learn(2000, callback=callback)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "lBF4ij46RUoC"
      },
      "source": [
        "## Forth example: Composition\n",
        "Thanks to the functional nature of callbacks, it is possible to do a composition of callbacks, into a single callback. This means we can auto save our best model, show the progess bar and episodic reward of the training.\n",
        "\n",
        "The callbacks are automatically composed when you pass a list to the `learn()` method. Under the hood, a `CallbackList` is created."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "5hU3T9tkRUoD",
        "scrolled": false,
        "colab": {}
      },
      "source": [
        "from stable_baselines.common.callbacks import CallbackList\n",
        "\n",
        "# Create log dir\n",
        "log_dir = \"/tmp/gym/\"\n",
        "os.makedirs(log_dir, exist_ok=True)\n",
        "\n",
        "# Create and wrap the environment\n",
        "env = make_vec_env('CartPole-v1', n_envs=1, monitor_dir=log_dir)\n",
        "\n",
        "# Create callbacks\n",
        "auto_save_callback = SaveOnBestTrainingRewardCallback(check_freq=1000, log_dir=log_dir)\n",
        "\n",
        "model = PPO2('MlpPolicy', env, verbose=0)\n",
        "with ProgressBarManager(10000) as progress_callback:\n",
        "  # This is equivalent to callback=CallbackList([progress_callback, auto_save_callback])\n",
        "  model.learn(10000, callback=[progress_callback, auto_save_callback])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SRB4-qIxg_c9",
        "colab_type": "text"
      },
      "source": [
        "## Exercise: Code your own callback\n",
        "\n",
        "\n",
        "The previous examples showed the basics of what is a callback and what you do with it.\n",
        "\n",
        "The goal of this exercise is to create a callback that will evaluate the model using a test environment and save it if this is the best known model.\n",
        "\n",
        "To make things easier, we are going to use a class instead of a function with the magic method `__call__`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MOn0Sr3OhC2U",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class EvalCallback(BaseCallback):\n",
        "  \"\"\"\n",
        "  Callback for evaluating an agent.\n",
        "  \n",
        "  :param eval_env: (gym.Env) The environment used for initialization\n",
        "  :param n_eval_episodes: (int) The number of episodes to test the agent\n",
        "  :param eval_freq: (int) Evaluate the agent every eval_freq call of the callback.\n",
        "  \"\"\"\n",
        "  def __init__(self, eval_env, n_eval_episodes=5, eval_freq=20):\n",
        "    super(EvalCallback, self).__init__()\n",
        "    self.eval_env = eval_env\n",
        "    self.n_eval_episodes = n_eval_episodes\n",
        "    self.eval_freq = eval_freq\n",
        "    self.best_mean_reward = -np.inf\n",
        "  \n",
        "  def _on_step(self):\n",
        "    \"\"\"\n",
        "    This method will be called by the model.\n",
        "\n",
        "    :return: (bool)\n",
        "    \"\"\"\n",
        "    \n",
        "    # self.n_calls is automatically updated because\n",
        "    # we derive from BaseCallback\n",
        "    if self.n_calls % self.eval_freq == 0:\n",
        "      # === YOUR CODE HERE ===#\n",
        "      # Evaluate the agent:\n",
        "      # you need to do self.n_eval_episodes loop using self.eval_env\n",
        "      # hint: you can use self.model.predict(obs, deterministic=True)\n",
        "      \n",
        "      # Save the agent if needed\n",
        "      # and update self.best_mean_reward\n",
        "      \n",
        "      print(\"Best mean reward: {:.2f}\".format(self.best_mean_reward))\n",
        "      \n",
        "\n",
        "      # ====================== #    \n",
        "    return True"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IO0I81jAkQ0z",
        "colab_type": "text"
      },
      "source": [
        "### Test your callback"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_OMop3TlkTbx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Env used for training\n",
        "env = gym.make(\"CartPole-v1\")\n",
        "# Env for evaluating the agent\n",
        "eval_env gym.make(\"CartPole-v1\")\n",
        "\n",
        "# === YOUR CODE HERE ===#\n",
        "# Create the callback object\n",
        "callback = None\n",
        "\n",
        "# Create the RL model\n",
        "model = None\n",
        "\n",
        "# ====================== #\n",
        "\n",
        "# Train the RL model\n",
        "model.learn(int(100000), callback=callback)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5wS20a_NfMAh",
        "colab_type": "text"
      },
      "source": [
        "# Conclusion\n",
        "\n",
        "\n",
        "In this notebook we have seen:\n",
        "- that good hyperparameters are key to the success of RL, you should not except the default ones to work on every problems\n",
        "- what is a callback and what you can do with it\n",
        "- how to create your own callback\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uA4gCDtogIaD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}