Using the callback system in fastai.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The callback system in the fastai library allows you to execute any code at specific times in the training. This can be extremely useful if you want to\n",
    "\n",
    "- record some parameters (for instance metrics, gradients)\n",
    "- use a condition to change a parameter (for instance change the learning rate if the validation loss hasn't improved)\n",
    "- make a schedule of parameters (for instance changing the probabilities of the dropout layers as the training goes)\n",
    "\n",
    "In this notebook, I'll show how to use this to code your own callbacks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%reload_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's experiment on cifar10 and small networks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.conv_learner import *\n",
    "PATH = Path(\"../data/cifar10/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')\n",
    "stats = (np.array([ 0.4914 ,  0.48216,  0.44653]), np.array([ 0.24703,  0.24349,  0.26159]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data(sz,bs):\n",
    "    tfms = tfms_from_stats(stats, sz, aug_tfms=[RandomFlip()], pad=sz//8)\n",
    "    return ImageClassifierData.from_paths(PATH, val_name='test', tfms=tfms, bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "size = 32\n",
    "batch_size = 64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = get_data(size,batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I've built a simple fully convolutional neural net."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ConvBlock(ch_in, ch_out, ks = 3, stride=1, padding=1, re=True, bn=True):\n",
    "    layers = [nn.Conv2d(ch_in,ch_out,ks,stride=stride,padding=1, bias=False)]\n",
    "    if bn: layers.append(nn.BatchNorm2d(ch_out))\n",
    "    if re: layers.append(nn.ReLU(inplace=True))\n",
    "    return nn.Sequential(*layers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "layers = [ConvBlock(3,64), \n",
    "          ConvBlock(64,64,stride=2), #size 16x16\n",
    "          ConvBlock(64,128,stride=2), #size 8x8\n",
    "          ConvBlock(128,256,stride=2), #size 4x4\n",
    "          ConvBlock(256,10, stride=2), #size 2x2 \n",
    "          nn.AdaptiveAvgPool2d(1), \n",
    "          Flatten()]\n",
    "model = nn.Sequential(*layers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = ConvLearner.from_model_data(model, data)\n",
    "learn.crit = F.cross_entropy\n",
    "learn.opt_fn = partial(optim.SGD, momentum=0.9)\n",
    "learn.metrics = [accuracy]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('init') #To go back from the first model later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what's a callback? Let's have a look at the source code of fastai."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Callback:\n",
    "    def on_train_begin(self): pass\n",
    "    def on_batch_begin(self): pass\n",
    "    def on_phase_begin(self): pass\n",
    "    def on_epoch_end(self, metrics): pass\n",
    "    def on_phase_end(self): pass\n",
    "    def on_batch_end(self, metrics): pass\n",
    "    def on_train_end(self): pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the class that doesn't seem to do anything but it's just because it's a general wrapper. Those seven functions are all called at a certain point during training. The names are self explanatory but just in case:\n",
    "- on_train_begin is called at the very beginning of the training. Useful to initialize the variables.\n",
    "- on_phase_begin is called at the beginning of each training phase (if you don't use the training API, there is only one phase during training). It's useful if you plan on having a different behavior during each phase.\n",
    "- on_batch_begin is called before the current batch is passed through the network, it's where you will want to update training parameters.\n",
    "- on_batch_end is called at the end of the batch, with the result of the loss function. It's where you will want to save data or metrics.\n",
    "- on_epoch_end is called at the end of an epoch, after the validation, with the validation loss and metrics as an argument. Again, useful to save validation data, or if you want to change something before the next epoch.\n",
    "- on_phase_end is called at the end of a training phase.\n",
    "- on_train_end is called at the very end of the training.\n",
    "\n",
    "Your customized callback doesn't have to implement those seven functions since the Callback class has them all, so you can code only the ones you want. Note that you can stop the training during on_batch_end or on_epoch_end by returning True, which might be useful!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin with a simple class that will record the validation losses. To see what's going on, let's use the easiest way: use the debugger to show us what this metrics object contains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SaveValidationLoss(Callback):\n",
    "    def __init__(self):\n",
    "        self.val_losses = []\n",
    "    \n",
    "    def on_epoch_end(self, metrics):\n",
    "        pdb.set_trace()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First let's find a learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f64aa1ed62eb48a9b25dbdbc534a3c26",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                                                                              \n",
      "    0      2.137145   2.88337    0.1061    \n",
      "\n"
     ]
    }
   ],
   "source": [
    "learn.lr_find(wds=1e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEOCAYAAABmVAtTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd4lFXax/HvPZNCSCUQQgoxdALSg1QRQVHEtWJfUVfFiljWta6vrq5ldV1R14K9ALsqiAgIKgrSMYTQWyiB0EIIgYSQft4/ZsjGmIQEMnmm3J/rmouZZ87M8zuZMHeedo4YY1BKKaUAbFYHUEop5T60KCillKqgRUEppVQFLQpKKaUqaFFQSilVQYuCUkqpCloUlFJKVdCioJRSqoIWBaWUUhW0KCillKrgZ3WA+mrRooVJTEy0OoZSSnmUlStXZhtjok7WzuOKQmJiIikpKVbHUEopjyIiGXVpp7uPlFJKVdCioJRSqoIWBaWUUhW0KCillKqgRUEppVQFLQpKKaUq+ExRKCotY1pqJjr9qFJK1cxnisL0VXt48IvVfLduv9VRlFLKbflMUbiydzxdYsJ4cvo6vknbQ25BMcYYth/Mp7CkzOp4SinlFjzuiuZT5We3ccc5bRn/nzTG/ycNgMjgAHKOFRPfLIjHL0oiMjiADi1DaB4SaHFapZSyhs8UBYBzO7ekZ+sI0nbnAtA1NowOLUOZtDyDuyelAhDgZ2NoxyiOl5SRENmU0X3isdsEP5uNxenZrN1zhD/0iOW8pJYYAyIgIlZ2SymlGox42oHX5ORk09BjH2UcOsby7TmENvFjUXo2C7dm42cXth889ru2IYF+5BeVEh7kT2lZOc2CA/jrxV04Lyma/KJSmgbY8bc79sqVlpXjZ/eZPXRKKTcmIiuNMcknbadFoWZZRwv5Jm0vNptgF+jbJpKO0aHMXruPZdsP4WezMX9LFrtzjtPE30ZhSTlB/naGdopCBL5ff4DSckNooB9d48K4uHssZ7WJpImfHYMhLiJIi4ZSqlFoUWgkRaVlfLJkJxmHCggJ9CMrr4hl2w9RUlaO3Sac3SGKkEA/Zq7ZR3Z+0W9em9i8KfcO60BEkD9d48KICQ+yqBdKKW+nRcHNlJUbMg4dY3VmLuXlUFhaxkeLd5KelV/Rpnt8OBd0bUX/ts3pc0YzC9MqpbyNFgUPUFpWzo8bDxDoZ2fj/qPMSNvLpv15AJzbKYq3buhDUIDd4pRKKW+gRcEDlZaVs27vUZZuO8RLczYR6Gejc0wYOceKGNw+igHtmjOqWwx2m57tpJSqHy0KHm7Ouv18vSqTrLwiCorK2HzAsQUxqH1zxg/vSK+EiIqznJRS6mQsLwoi0hr4FGgFlAMTjTETqrS5FHjW+XwpcL8xZlFt7+srRaGqsnLDFym7eW7mBo4VO67APi+pJc9d1o2o0EDdelBK1codikIMEGOMSRWRUGAlcJkxZkOlNiHAMWOMEZHuwBfGmM61va+vFoUT8gpLWLQ1mwVbDjI9bQ+FJeX42YQ/9j+DJ0Yl6daDUqpadS0KLvsGMcbsM8akOu/nARuBuCpt8s3/qlIw4Fn7siwQ2sSfkd1iePHK7nw3fghDOkbRMjSQj5fs5IJ//cLMNXt1JFil1ClrlGEuRCQR6AUsr+a5y4EXgJbAqMbI4y3atAjm0z+dBcAXv+7m2VkbuHfyKhaflc2jFyYR3tTf4oRKKU/j8gPNzl1EC4C/G2Om1dJuCPCUMea8ap4bC4wFSEhI6JORkeGquB6tsKSMl+Zs4qPFOwkJ9OPyXnE8cH5HIoMDrI6mlLKY5ccUnCH8gZnAXGPMq3VovwPoa4zJrqmNrx9TqIuN+47y1vxtzFyzlyZ+drrFhxMVGkjP+Aiu65dASKBPjYOolMINioI4hg79BMgxxtxfQ5v2wDbngebewLdAvKkllBaFulu2/RCfLt3J1gP5HC4oJju/mNaRQdx7bntG92mtZywp5UPqWhRc+SfjIOBGYK2IpDmXPQ4kABhj3gGuBMaISAlwHLimtoKg6qd/2+b0b9scAGMMy7bn8PBXq3lk6lq+XrWHv17cha6x4RanVEq5E714zccYY/h8WQYvz93MseIyHjy/I2MGnEFoEz0orZQ3s/yUVOWeRIQbByTyzb2DOTM2jJfnbuaG95dTUlZudTSllBvQouCj2rQIZvo9g3j16h6syTzCaz9uIa+wxOpYSimLaVHwYSLCFb3jGdUthn//vI2z//EzS9JrPPFLKeUDtCgoXhrdnTuGtMXfbuOOz1fyyZKdlOruJKV8khYFRUigH49dlMS0uwaSENmU/5uxnvH/TdPCoJQP0qKgKrSObMrMcYP5y4WdmLVmH/dMTiW3oNjqWEqpRqSXtqrfEBHuHtqeALuNv8/eyNasJbxxXS+9nkEpH6FbCqpat53dlkm39mNfbiGjXl/EZ8t0vCmlfIEWBVWjge1bMOu+wfRsHcFT36zj8a/X8v36/VbHUkq5kBYFVau2USFMuq0ff+gey5QVuxj72Uom/rKNsnLPuhJeKVU3WhTUSQUH+vH6db3Y/OxIzkuK5vnZm7jqnSVs2HtUJ/RRystoUVB1FuBn470xfXh5dHdSd+Vy0esLGTdlFdn5RVZHU0o1ED37SNWLiHBVcmtiI4KYlrqHb9L2sGx7Dted1ZohHaPomxhpdUSl1GnQUVLVaVm35wh/n7WRpdsPAfDwBZ2465x22HSuBqXciuWT7LiKFgX3lHW0kHFTVrF8Rw7tW4bwxnW9SIoJszqWUspJh85WjaplWBMm396fRy7szLaD+YycsJAL/vUL01ft0YPRSnkQ3VJQDW793iP8uCGL6Wl72JF9DIAJ1/bk0p5xFidTynfploKyTNfYcMaf14EfHzyHUd1jaB4cwJ+/XM38zVlWR1NKnYQWBeUydpvw7+t788OD59AuKoTbPklhsc7XoJRb06KgXC4yOID/jO1PfLMgbv80hZ3OXUpKKfejRUE1ioimAUy+vT92m3DR6wv5z4pd5BzTYbmVcjdaFFSjiY0I4uu7B9IxOpRHp61l6Ms/k3W00OpYSqlKtCioRtW+ZSjvjUnmsp6xHC0sZexnK3WGN6XciBYF1eiiQgN57dpeTLi2J2m7c7nynaU6fpJSbsJlRUFEWovIzyKyUUTWi8j4atrcICJrnLclItLDVXmU+7m0ZxyvXt2DzfuPcvNHKygsKbM6klI+z5VbCqXAQ8aYJKA/cI+IdKnSZgdwjjGmO/AsMNGFeZQbuqJ3PG9c15t1e47S45nvmb12n9WRlPJpLisKxph9xphU5/08YCMQV6XNEmPMYefDZUC8q/Io93V+l2jevbEP7VuGcPekVN6ev02HxlDKIo1yTEFEEoFewPJamt0KfNcYeZT7uaBrK6bdPZCLu8fw0pxNXPn2EvIKS6yOpZTPcfnYRyISAiwA/m6MmVZDm3OBt4DBxphD1Tw/FhgLkJCQ0CcjQyeR91bl5YaPluzkhdkbaRYcwIgu0XSOCePG/mdYHU0pj1bXsY9cOsmOiPgDU4FJtRSE7sD7wMjqCgKAMWYizuMNycnJul/Bi9lswq2D25AQ2ZQ3f05n0vJdAPjbhNF94vGz6wlzSrmSy7YURESAT4AcY8z9NbRJAH4CxhhjltTlfXWUVN+yO6eAWz/5lS0H8hnSMYqnLk6ifctQq2Mp5XEsn2RHRAYDC4G1wImrkx4HEgCMMe+IyPvAlcCJ/UGlJwutRcH3FJeW8/6i7bz6/RZKyw3jh3fg3mHt8detBqXqzPKi4CpaFHzX/iOF3PH5SlbvziUuIoiXR3dnYPsWVsdSyiPofArK67QKb8IHNyXz/OXdKCot54YPlrNgy0GrYynlVbQoKI/SIiSQ6/slMO+hc0hsHswD/03j3z+ns3p3rtXRlPIKWhSURwoP8ufDm/sSHdaEl+du5tJ/L+bJ6WspL/es3aFKuRuXnpKqlCu1aRHMrHGDWbnrMF/8upvPl+3C327jqYu74Dj5TSlVX1oUlEez2YS+iZEkn9GM0Cb+fLh4B8eKSnn2sjMJ9LNbHU8pj6NFQXkFEeHJUUkE+tt4e/428gpL+fvl3YgMDrA6mlIeRY8pKK9hswmPXNiZ0X3i+W7dfs59ZT77jhy3OpZSHkWLgvI6r1zVg4k39uF4cRn3/yeN/KJSqyMp5TG0KCivNKJrK/52aVeW78hh3ORUHYpbqTrSoqC81rVnJfDkqCR+3nywYmA9pVTttCgor/anQW04u0MLnvl2Pc98u16vY1DqJLQoKK9mswmvXdOTYZ1b8tHinYz9LIXjxToXtFI10aKgvF7zkEDe+WMf7hvWnh83ZnH5W4uZlpppdSyl3JIWBeUTRIQHR3TiyVFJbNqfx4NfrOZfP2yxOpZSbkeLgvIpt53dli3PjeTK3vFMmLeVl+du0jOTlKpEi4LyOQF+Nl68shsXdm3Fv3/exkWvL+LI8RKrYynlFrQoKJ/kb7fx9h97M7xzSzbuO6q7kpRy0qKgfJaIMHFMMtf2bc1nyzJYkp5tdSSlLKdFQfk0u014+IJORIUEMubDFcxZt8/qSEpZSouC8nnNQwKZ+8AQzowL55Gpa8k6Wmh1JKUso0VBKRwzub08ujvHS8o479UFrMnU6T2Ve5mWmtkoo/5qUVDKqUN0KN/eO5iQQD9ueG85Ww7kWR1JKQC2HMjj4a/W8O6C7S5flxYFpSrp1CqU/94xgAA/G2M+WEF2fpHVkZTis6UZ+NmE8cM7uHxdWhSUqqJ1ZFPeuymZnIJiLnxtIUcK9BoGZZ3jxWV8u2YvF3RtRbNGmElQi4JS1eid0IyPb+5Ldn4RHy7eYXUc5aMKiku55eMVHDlewg39EhplnS4rCiLSWkR+FpGNIrJeRMZX06aziCwVkSIR+bOrsih1Kga2b8Hwzi2ZtHyXDrmtLDFlxW6Wbc/hX1f3pF/b5o2yTlduKZQCDxljkoD+wD0i0qVKmxzgPuAVF+ZQ6pRd1iuO7Pwi3l6wzeooysccLSzh7fnbSIoJ47JecY22XpcVBWPMPmNMqvN+HrARiKvSJssY8yugO22VW7qoWwwjukQzYd5Wlm0/ZHUc5UMWbc0mO7+Ihy/o2KjrbZRjCiKSCPQClp/i68eKSIqIpBw8eLAhoylVK7tNeOnK7iRENuXeyankFhRbHUn5iO/W7Sck0I+zO0Q16npdXhREJASYCtxvjDl6Ku9hjJlojEk2xiRHRTXuD0ipZsEBTLi2J4cLSnjxu01Wx1E+YPvBfGat2csN/RPwtzfu+UAuXZuI+OMoCJOMMdNcuS6lXKlrbDg3DUjkvym7yTxcYHUc5eXeX7QDP7uN289u2+jrduXZRwJ8AGw0xrzqqvUo1Vj+NDgRuwgvzNatBeU6h48VMy01kyt6xdEiJLDR1+/KLYVBwI3AMBFJc94uEpE7ReROABFpJSKZwIPAkyKSKSJhLsyk1CmLb9aU+4Z3YNbafXy/fr/VcZSX+nLlbgpLyrllUBtL1u/nqjc2xiwC5CRt9gPxrsqgVEO7/ey2fLlyN/dOXsWs+wbTITrU6kjKy8zffJCkmDA6tbLmd0uvaFaqHoIC7Hx5x0CaBtr585erKSkrtzqS8iJFpWWszDjMgEa6UK06WhSUqqdW4U145pKurM48wsNfrtarnVWDWbUrl6LScga006KglEf5Q/dYuseHMz1tr46NpBrM0m2HsAmc1SbSsgxaFJQ6BTab8M09gxjWuSXPzdrIqNcXcry4zOpYysMtSs/mzLhwwoP8LcugRUGpUyQiPHfZmXRuFcr6vUcZ9cZCft6cZXUs5aEWp2eTuuswwztHW5qjTkVBRMaLSJg4fCAiqSIywtXhlHJ3sRFBzLl/COOGtScz5zj3TkqluFQPPqv6+ef3m7nh/eW0btaUG/o3zhDZNanrlsKfnENUjACigFuAF12WSikP89CITrx2bU+OFZfx6g9bMEYPPqu6KSot4+MlO+nXJpJpdw+05IK1yupaFE5cb3AR8JExZjUnuQZBKV8zsF1z2kYF886CbUyYt9XqOMpDLEk/RF5hKXec09byggB1LworReR7HEVhroiEArqNrFQlEU0DmPfgOYzuE89rP25l3sYDVkdSHmD22n2EBvoxqH0Lq6MAdS8KtwKPAn2NMQWAP45dSEqpSkSEF67oxhnNm+puJHVS2w/mM2P1Xs7vEk2gn93qOEDdi8IAYLMxJldE/gg8CRxxXSylPJe/3cY957Zn/d6j/LRJz0ZSNZu1Zh9FpeU8cH7jTqRTm7oWhbeBAhHpAfwFyAA+dVkqpTzc5b3iaB0ZxIR5W3VrQdVo8bZskmLCaB3Z1OooFepaFEqN4zf7UmCCMWYCoCOBKVUDf7uNe4a2Z03mEeZv0dkC1e8dLy4jNSOXwe2tG9KiOnUtCnki8hiOobBniYgdx3EFpVQNrugdT1xEEBN+1K0F9XspGTkUl5Uz0E0OMJ9Q16JwDVCE43qF/UAc8LLLUinlBQL8bIwb1p603bm8Pi/d6jjKzfy0KQt/u3BWonXjHFWnTkXBWQgmAeEicjFQaIzRYwpKncQ1fVtzSY9Y3vhpK4fyi6yOo9yEMYZvV+/jvKRoggNdNq3NKanrMBdXAyuAq4CrgeUiMtqVwZTyBiLCPee2p7Tc8Mr3W6yOo9zEtoP5ZOcXcW6nllZH+Z26lqgncFyjkAUgIlHAj8BXrgqmlLfo1CqUXgkRTFmxi76Jzbiit0426OsmL9+N3SYM6uBexxOg7scUbCcKgtOherxWKZ836bZ+xEUEMWP1XqujKIvlFhTz+fIMrugVR1xEkNVxfqeuX+xzRGSuiNwsIjcDs4DZroullHdpGuDHqO4xLNqaTcrOHD0byYctTj9EcWk5157V2uoo1arrgeaHgYlAd6AHMNEY84grgynlbe4Y0pbYiCBGv7OU9xfqbG2+alF6NiGBfvSIj7A6SrXqvAvIGDPVGPOgMeYBY8zXrgyllDdqHhLIp386C4DX520l83CBxYmUFRanZ9O/bXP87O65B77WVCKSJyJHq7nlicjRxgqplLdIbBHM9w8MobTccOMHK3RCHh+z61ABu3IK3O4q5spqLQrGmFBjTFg1t1BjTFhjhVTKm3SMDuXVq3uwI/sYc9bvtzqOakSLt2UDMNgNzzo6wWXbLyLSWkR+FpGNIrJeRMZX00ZE5HURSReRNSLS21V5lHInw5JaEhcRxNMz1pNfVGp1HNVIFqVn0yqsCe2iQqyOUiNX7tQqBR4yxiQB/YF7RKRLlTYjgQ7O21gco7Eq5fUC/exMuLYnOceKuXdyqp6N5APKyw1L0rMZ1L4FIu47caXLioIxZp8xJtV5Pw/YiGPMpMouBT41DsuACBGJcVUmpdxJcmIkj47szPzNB1m3Rw/RebuVuw5zuKCEIR3dd9cRNNIFaCKSCPQClld5Kg7YXelxJr8vHEp5rWuSW+NvFyavyLA6inKxV+ZuJjjAzvCkaKuj1MrlRUFEQoCpwP3GmKp/DlW3DfW77WgRGSsiKSKScvCgjk2vvEez4ABG94nn61V7KCwpszqOcpE56/azfEcOD43oRIibDYBXlUuLgoj44ygIk4wx06ppkglUvqwvHvjdOADGmInGmGRjTHJUVJRrwiplkQvPjKGwpJwfNhywOopykY+X7CA6LNBtr2KuzJVnHwnwAbDRGPNqDc1mAGOcZyH1B44YY/a5KpNS7mhw+xa0iwrmzZ/SKS/XA87eKD3rGEM6RNE0wL23EsC1WwqDcMzUNkxE0py3i0TkThG509lmNrAdSAfeA+52YR6l3JLdJtw3vAObD+Tx3Tq9bsHbHMwrIju/iA7R7nsaamUuK1vGmEVUf8ygchsD3OOqDEp5iou7x/KvH7bwwaLtjOquJ+B5k7nOCxTP6eh+cydUxz0H31DKx9htwk0DE0ndlcuazFyr46gGtHTbIWLDm9CpVajVUepEi4JSbmJ0n3iCA+x8vGSn1VFUA1myLZsfNx5gQDv3vjahMi0KSrmJ0Cb+jO4Tz8zV+ziYp/M5e7rSsnIenbqWuGZBPHZRZ6vj1JkWBaXcyJiBiZSUl/PSnE1WR1Gnacqvu9mVU8BjI5NoERJodZw606KglBtpFxXCbYPbMDU1ky0H8qyOo05RQXEp/5izif5tIzkvyTMOMJ+gRUEpN3PX0PaENfHn/75ZrwPleagFmw+SV1jK+OEd3Xrwu+poUVDKzUQGB3Df8A4s3X6I9Xt1oDxPsyYzl7smpQKQnNjM4jT1p0VBKTd0ac9YAvxsXDtxGb/uzLE6jqqHGWmOkXr+Mbo7/m465WZtPC+xUj6gRUggU27vj59duOqdpazerdcueIrNB/I4My6Mq5Pdf5yj6mhRUMpN9TmjGZ/f2g+Au527I5R7M8awcV8enaI9d7ZiLQpKubEz48K5oV8Ce3KPMy010+o46iQyDx8nO7+IngkRVkc5ZVoUlHJz44Z1IC4iiCe+XseO7GNWx1G1SHPu5uvVWouCUspFWoU3YepdAxGBN39KtzqOqsW6PUcIsNvoGO0Z4xxVR4uCUh6gVXgTruoTz4zVe8g8XGB1HFWDNZlHSIoJJcDPc79aPTe5Uj7mtrPbEmC38cf3l1NSVm51HFVFWblh3Z4jnBkXbnWU06JFQSkP0TqyKY+PSmLnoQLW7jlidRxVxbTUTPKKSjm7g2dPGaxFQSkPMvLMGGwCT89Yz5rMXEp1i8EtZBw6xt9mbqBHfDgXdI22Os5p0aKglAeJDA5gRJdWrMk8wiVvLuaZbzdYHcmnGWMoKC7l1k9SyCss5a8Xd/G4sY6q0qKglId5/bpevHFdLwA+W5bB1JV6/YIVjDHc/NGvdHlqLulZ+bx1Q2+SEyOtjnXatCgo5WEC/Gz8oUcsm569kK6xYbw2b4vuRrLAC99tYsGWgwD0bB3ByDNbWZyoYWhRUMpDNfG3M354B3bnHGfaqj1Wx/EppWXlfL4sg7MSI/n4lr58fls/j99tdIIWBaU82PldomkbFcykZRk690Ij2rQ/j4LiMq7vl8DQTi0JCfSzOlKD0aKglAcTEa5Obs3qzCO8qFN4NpoTu40GtmtucZKGp0VBKQ93Y/8zsAl8tGgn6Vn5VsfxCat25dK+ZQgtw5pYHaXBuawoiMiHIpIlIutqeL6ZiHwtImtEZIWInOmqLEp5s+BAP766ayCB/jbunZxKWbnuRnK19Kw8Onnw+Ea1ceWWwsfAhbU8/ziQZozpDowBJrgwi1JerXdCM56/vBub9ucxabkeX3ClY0Wl7MopoH3LEKujuITLioIx5hegtnkEuwDznG03AYki4tmXAiploVHdYugaG8ZT36znsWlrrY7jtVZmHKbcOCZB8kZWHlNYDVwBICJnAWcA8RbmUcqj2WzCa9f0JL5ZENPT9lBYUmZ1JK/08+YsAvxsWhRc4EWgmYikAeOAVUBpdQ1FZKyIpIhIysGDBxszo1IepUN0KK9c1YPCknKe+XaD7kZygfmbDzK4fQuCveg01MosKwrGmKPGmFuMMT1xHFOIAnbU0HaiMSbZGJMcFeXZIxAq5Wr92zbn7qHtmLJiF3PXH7A6jlc5UlDCjuxjXruVABYWBRGJEJEA58PbgF+MMUetyqOUN3loRCfimwXx/OyN7Mk9TkFxtRvhqp7W7HFMt9kj3nOn2zwZV56SOgVYCnQSkUwRuVVE7hSRO51NkoD1IrIJGAmMd1UWpXyN3Sa8clUP9uQeZ9CLPzHslQWkZ+VZHcvjrcl0zGPRLd6zJ9Kpjct2ihljrjvJ80uBDq5av1K+rn/b5jx9SVcm/rKN3TnHeevnbbx6TU+rY3m0TfvzaB0ZRHiQv9VRXEavaFbKi93Y/wwW/mUYo7rH8MvWbIpLdTTV05F5uICEyKZWx3ApLQpK+YDRfeLJzi9izvr9VkfxWBv3HWXVrlziI7QoKKU83JAOUUQGB/C9FoVTUlZuuObdpQD0aO29B5lBi4JSPsFuEy7vFcfstftYkp5tdRyP8+nSnRwtLOWec9tx3VmtrY7jUloUlPIRD57fkajQQD5YVO3lQKoWs9fuIykmjD+P6OQ1k+nURIuCUj4iONCPS3vGMW9TFsu2H7I6jsfIOHSMVbtyGdKhhdcXBNCioJRPuf+8DoQG+vHVykyro3iMD51bVjcPSrQ2SCPxzsE7lFLVahrgx4iurfhqZSbGwK6cY5SWGw7lFzPt7oG0CAm0OqJbSc/K45OlGQztFEVMeJDVcRqFbiko5WNG93EMRjw1NZNfdx5m1a5cduUU8MmSndYGc0Oz1zrO1nr6D10tTtJ4dEtBKR8zoF1znr+8G+v2HiE8yB9/m/Dtmn288VM6Ww/k88b1vfC369+L+48U8uoPW+iVEEFii2Cr4zQaLQpK+aDr+yX85vGNAxJ56pt1fLduPzPS9nJlH53a5NlZGwC4Otm7T0GtSv8cUEoRFRrIWzf0pm1UMP9N2W11HMsdLy7jxw0HuHlgItedlXDyF3gRLQpKKQBEhCt7x7NiRw67cwqsjmOpScszKCotZ+SZrayO0ui0KCilKlzROw4/m/DhYt+9wK2krJw3fkpnSMco+rVtbnWcRqdFQSlVISY8iMt6xfHR4p1c8+5SFvvgkBgPfbGaI8dLuN7Lh7OoiRYFpdRv3HlOOwCW78jhwS/SKCotszhR48k8XMCM1XsBGNzBN6f+1aKglPqN9i1D+P6BIbx7Yx8OHC3iqenrMcZYHcvljDG8u2A7dpuw+NFhhAT65smZvtlrpVStOkaH0qFlCNf3S2Dy8l1sO5jPiK7R3Dq4LXabd47/M2fdfj5blkGLkEDiInzj6uXqaFFQSlVLRPjbJV1ZtSuXlIzDpGQcJiY8iIu6xQB4VXEoLze8s2AbAH+5oJPFaaylu4+UUjXys9uYOW4wm569kMjgAMZNWUXSU3O4+aMV5BeVWh2vwSzfkcPqzCM8dH5Hru7rmweYT9CioJSqld0mNPG3c9vZbbDbhOLSchZuzWb020soLze8+N0mxny4grJyzzzuYIzh4yU78LcLtwxuY3Ucy+nuI6VUndx1TjvGDEgkPSufmz5cwab9ebR9fHY5xykeAAAOj0lEQVTF8z9vyiI6rAnd4sMtTFl/X63MZO76A1zSI9ZnDy5XplsKSqk6ERFCAv3o2TqClU+eR6foUESoOCh726cp/OHNRTw7c4PFSeumuLSct+dv4+Gv1hAa6Mdzl59pdSS3oGVRKVVvfnYb344bjL9dEBG+SNnNX75aA8AHi3ZQVFpG19hwtx436NOlO3lpziYAHhzRkbAm/tYGchMuKwoi8iFwMZBljPldCRaRcOBzIMGZ4xVjzEeuyqOUalgBfv/b0XB1cmsu6RHL1gP5jJuSyufLdgHgb7dxWc9Y/NxsKO6N+47y3KyNhAf588IV3biwq++NcVQTV35SHwMX1vL8PcAGY0wPYCjwTxEJcGEepZQLNfG30y0+nLdu6FOx7M9fruaDRe41jtKR4yWMnLAQgA9uSuaibjHYvOj02tPlsqJgjPkFyKmtCRAqjpmwQ5xtveccN6V8VJfYMNL/PpIJ1/YEYOIv29mbe9ziVA75RaX0fvYHAG4ZlEhyYqTFidyPldt0bwJJwF5gLTDeGFNuYR6lVAPxs9u4tGccs+4bTF5hKRN/2W51JMBx1XJZueHCrq34Px+aYrM+rCwKFwBpQCzQE3hTRMKqaygiY0UkRURSDh482JgZlVKnoWtsOEkxoaRn5VsdBYBv0vYQ3yyIt//Y2+oobsvKonALMM04pAM7gM7VNTTGTDTGJBtjkqOifHPkQqU8VbPgABalZ3PJm4v4YcMBSzLM35xF4qOzWLg1mwu6tsKx11pVx8qisAsYDiAi0UAnwD22MZVSDWaUc6ykNZlHuP3TFGav3dfoGV77cWvF/U7RoY2+fk/isqIgIlOApUAnEckUkVtF5E4RudPZ5FlgoIisBeYBjxhjfG9GD6W83FXJrdn54ij+Mbo7AHdPSm3U3UnTUjNJ251L51aOYtC3jR5cro142jjpycnJJiUlxeoYSql6MsYwc80+xk1ZBcCW50b+5loHVygsKWPAC/M4o3kwU27vT6CfzWdPPxWRlcaY5JO1c68rSpRSXktEuLh7DNFhgQBc8fZiso4WumRdxhg278/jr9PXcbighEdHdiYowO6zBaE+tCgopRqNiLD88fP41zU92HIgn9d/2nryF52Cv8/ayAWv/cKXKzMZ1T2GfrrLqM60KCilGt3lveIZ1qkl8zZmNfhUn8Wl5Uxa7hhmI6KpP89f1k3PNqoHLQpKKUsMaNecfUcK2XukYXchpe3O5XhJGROu7cmyx4YT3lQHuqsPLQpKKUv0TmgGwMQF2yhvgAl6Vmbk8OoPW3joyzSaBtgZ1rklTfztp/2+vkaHzlZKWeLMuDDOS4rmk6UZfLI0g0WPnEt8s6an9F4Zh45x5dtLAQhr4sfTl3QlVIfCPiW6paCUsoSI8N6YPsSGNwHgkalr6vX6/UcKKS1zDJf2oXMk1h6tI/j+gXO4Otm351k+HVoUlFKWERH+M3YAZyVGsjj9EEu3HQL43cFnYwxPz1jP8H/O54uU3RzKL6L/C/N4ac4mVuzI4ZOlGfSID+ebewbRyllk1KnRi9eUUpYrLClj+D8XABAW5E/GoWM8eH5Hbju7LQCX/XsxabtzARCBq/rE80VK5m/eY9Z9g+ka61nzQzcmvXhNKeUxmvjbefXqHthsjlnRCorLeH72Rmat2UfOseKKgrDsseEE+tl+VxBuHpioBaGB6JaCUspt5BwrZsqKXZzTMYonpq9je1Y+53eNZlrqHmbfdzZdYsOYvzmLmz/6lffGJBMcaMcmQr82kXotwknUdUtBi4JSyi2tyczlkjcXAzCiSzQTx/zv+yyvsETPLqqnuhYFPSVVKeWWusdHcPvZbRARHrnwt1OtaEFwHS0KSim39cSoLlZH8Dl6oFkppVQFLQpKKaUqaFFQSilVQYuCUkqpCloUlFJKVdCioJRSqoIWBaWUUhW0KCillKrgccNciMgR4MRs3+HAkVrutwCyT2N1ld/zVNpU91zVZSfrQ+X72p+6Zz1Zm/r2pepjb+1P5WWn05+69KW2dp7Yn4b8v1P1cUP05wxjTNRJWxljPOoGTKzrfSClodZ1Km2qe67qMu2PNf2pb198pT9Vlp1yf+rSF2/rT0P+33F1f2q7eeLuo2/reb+h1nUqbap7ruoy7c+pO53+1LcvVR97a38asy+1tfPE/jTk/52qjxu6PzXyuN1H9SEiKaYOowJ6Cu2Pe9P+uDftT9144pZCfUy0OkAD0/64N+2Pe9P+1IFXbykopZSqH2/fUlBKKVUPWhSUUkpV0KKglFKqgs8WBREZKiILReQdERlqdZ6GICLBIrJSRC62OsvpEpEk52fzlYjcZXWe0yUil4nIeyLyjYiMsDrP6RKRtiLygYh8ZXWWU+H8v/KJ8zO5weo8p6shPw+PLAoi8qGIZInIuirLLxSRzSKSLiKPnuRtDJAPNAEyXZW1LhqoPwCPAF+4JmXdNUR/jDEbjTF3AlcDlp5G2ED9mW6MuR24GbjGhXFPqoH6s90Yc6trk9ZPPft1BfCV8zO5pNHD1kF9+tOgn4crrohz9Q0YAvQG1lVaZge2AW2BAGA10AXoBsyscmsJ2JyviwYmeUF/zgOuxfGlc7Gn98f5mkuAJcD13tAf5+v+CfT2ov58ZWVfTqNfjwE9nW0mW539dPvTkJ+HHx7IGPOLiCRWWXwWkG6M2Q4gIv8BLjXGvADUtjvlMBDoipx11RD9EZFzgWAcv/DHRWS2MabcpcFr0FCfjzFmBjBDRGYBk12XuHYN9PkI8CLwnTEm1bWJa9fA/3/cRn36hWPvQDyQhpvuMalnfzY01Hrd8odxiuKA3ZUeZzqXVUtErhCRd4HPgDddnO1U1Ks/xpgnjDH34/jyfM+qglCL+n4+Q0XkdednNNvV4U5BvfoDjMOxNTdaRO50ZbBTVN/Pp7mIvAP0EpHHXB3uNNTUr2nAlSLyNo0wdEQDqrY/Dfl5eOSWQg2kmmU1XplnjJmG4xfDXdWrPxUNjPm44aM0iPp+PvOB+a4K0wDq25/XgdddF+e01bc/hwB3LG5VVdsvY8wx4JbGDtMAaupPg30e3rSlkAm0rvQ4HthrUZaGoP1xb9ofz+Bt/XJ5f7ypKPwKdBCRNiISgOOg6wyLM50O7Y970/54Bm/rl+v7Y/UR9lM8Kj8F2AeU4KictzqXXwRswXF0/gmrc2p/tD/uePO2/nhrv6zqjw6Ip5RSqoI37T5SSil1mrQoKKWUqqBFQSmlVAUtCkoppSpoUVBKKVVBi4JSSqkKWhSUy4lIfiOs45I6Di/ekOscKiIDT+F1vUTkfef9m0XELcbeEpHEqsM0V9MmSkTmNFYm1fi0KCiPISL2mp4zxswwxrzognXWNj7YUKDeRQF4HHjjlAJZzBhzENgnIoOszqJcQ4uCalQi8rCI/Coia0TkmUrLp4tj1rj1IjK20vJ8EfmbiCwHBojIThF5RkRSRWStiHR2tqv4i1tEPnaOsLpERLaLyGjncpuIvOVcx0wRmX3iuSoZ54vI8yKyABgvIn8QkeUiskpEfhSRaOeQxncCD4hImoic7fwreqqzf79W98UpIqFAd2PM6mqeO0NE5jl/NvNEJMG5vJ2ILHO+59+q2/ISx0xis0RktYisE5FrnMv7On8Oq0VkhYiEOrcIFjp/hqnVbe2IiF1EXq70Wd1R6enpgMfPVqZqYPWl3Hrz/huQ7/x3BDARx0iPNhwTtgxxPhfp/DcIWAc0dz42wNWV3msnMM55/27gfef9m4E3nfc/Br50rqMLjvHnAUbjGIbbBrTCMZfG6GryzgfeqvS4GVRc/X8b8E/n/aeBP1dqNxkY7LyfAGys5r3PBaZWelw597fATc77fwKmO+/PBK5z3r/zxM+zyvteiWPI9BOPw3FMwrId6OtcFoZjZOSmQBPnsg5AivN+Is4JXYCxwJPO+4FACtDG+TgOWGv175XeXHPzpqGzlfsb4bytcj4OwfGl9Atwn4hc7lze2rn8EFAGTK3yPieGPF+JY1rF6kw3jjklNohItHPZYOBL5/L9IvJzLVn/W+l+PPBfEYnB8UW7o4bXnAd0EakY3ThMREKNMXmV2sQAB2t4/YBK/fkM+Eel5Zc5708GXqnmtWuBV0TkJWCmMWahiHQD9hljfgUwxhwFx1YF8KaI9MTx8+1YzfuNALpX2pIKx/GZ7ACygNga+qA8nBYF1ZgEeMEY8+5vFooMxfGFOsAYUyAi83HMnQ1QaIwpq/I+Rc5/y6j5d7io0n2p8m9dHKt0/w3gVWPMDGfWp2t4jQ1HH47X8r7H+V/fTqbOA5MZY7aISB8cg6W9ICLf49jNU917PAAcAHo4MxdW00ZwbJHNrea5Jjj6obyQHlNQjWku8CcRCQEQkTgRaYnjr9DDzoLQGejvovUvwjHbls259TC0jq8LB/Y4799UaXkeEFrp8ffAvSceOP8Sr2oj0L6G9SzBMRQyOPbZL3LeX4Zj9xCVnv8NEYkFCowxn+PYkugNbAJiRaSvs02o88B5OI4tiHLgRhzz/lY1F7hLRPydr+3o3MIAx5ZFrWcpKc+lRUE1GmPM9zh2fywVkbXAVzi+VOcAfiKyBngWx5egK0zFMQTxOuBdYDlwpA6vexr4UkQWAtmVln8LXH7iQDNwH5DsPDC7gWpmwjLGbALCnQecq7oPuMX5c7gRGO9cfj/woIiswLH7qbrM3YAVIpIGPAE8Z4wpBq4B3hCR1cAPOP7Kfwu4SUSW4fiCP1bN+72PY97fVOdpqu/yv62yc4FZ1bxGeQEdOlv5FBEJMcbki0hzYAUwyBizv5EzPADkGWPer2P7psBxY4wRkWtxHHS+1KUha8/zC3CpMeawVRmU6+gxBeVrZopIBI4Dxs82dkFwehu4qh7t++A4MCxALo4zkywhIlE4jq9oQfBSuqWglFKqgh5TUEopVUGLglJKqQpaFJRSSlXQoqCUUqqCFgWllFIVtCgopZSq8P8MR9GoWgqvZgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's get something like a 1cycle to train our network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def one_cycle(lr, div, lengths, max_mom, min_mom, wds):\n",
    "    return [TrainingPhase(lengths[0], optim.SGD, lr = (lr/div, lr), lr_decay=DecayType.LINEAR,\n",
    "                          momentum = (max_mom,min_mom), momentum_decay=DecayType.LINEAR, wds=wds),\n",
    "            TrainingPhase(lengths[1], optim.SGD, lr = (lr, lr/div), lr_decay=DecayType.LINEAR,\n",
    "                          momentum = (min_mom,max_mom), momentum_decay=DecayType.LINEAR, wds=wds),\n",
    "            TrainingPhase(lengths[2], optim.SGD, lr = lr/div, lr_decay=DecayType.COSINE,\n",
    "                          momentum = max_mom, wds=wds)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use our callback in the training, we just have to add it in the arguments of fit/fit_opt_sched:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8a7d460ad150476585cddc75b13fb5ee",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=25), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--Return--                                                                                                             \n",
      "> <ipython-input-13-9aafc4cf57fe>(6)on_epoch_end()->None\n",
      "-> pdb.set_trace()\n",
      "(Pdb) metrics\n",
      "[1.4692460243225098, 0.4933]\n",
      "(Pdb) q\n"
     ]
    },
    {
     "ename": "BdbQuit",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mBdbQuit\u001b[0m                                   Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-17-d3302fa5ee18>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mlearn\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfit_opt_sched\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mone_cycle\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;36m0.1\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m10\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m[\u001b[0m\u001b[1;36m10\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m10\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;36m5\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m0.95\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m0.85\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m1e-3\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;33m[\u001b[0m\u001b[0mSaveValidationLoss\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32mD:\\Work\\Deeplearning\\Notebooks\\fastai\\learner.py\u001b[0m in \u001b[0;36mfit_opt_sched\u001b[1;34m(self, phases, cycle_save_name, best_save_name, stop_div, data_list, callbacks, cut, use_swa, swa_start, swa_eval_freq, **kwargs)\u001b[0m\n\u001b[0;32m    440\u001b[0m             \u001b[0mmetrics\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mmetrics\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mcallbacks\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mreg_fn\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mreg_fn\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mclip\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mclip\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfp16\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mfp16\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    441\u001b[0m             \u001b[0mswa_model\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mswa_model\u001b[0m \u001b[1;32mif\u001b[0m \u001b[0muse_swa\u001b[0m \u001b[1;32melse\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mswa_start\u001b[0m\u001b[1;33m=\u001b[0m\u001b[0mswa_start\u001b[0m\u001b[1;33m,\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 442\u001b[1;33m             swa_eval_freq=swa_eval_freq, **kwargs)\n\u001b[0m\u001b[0;32m    443\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    444\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0m_get_crit\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdata\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;32mreturn\u001b[0m \u001b[0mF\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmse_loss\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mD:\\Work\\Deeplearning\\Notebooks\\fastai\\model.py\u001b[0m in \u001b[0;36mfit\u001b[1;34m(model, data, n_epochs, opt, crit, metrics, callbacks, stepper, swa_model, swa_start, swa_eval_freq, **kwargs)\u001b[0m\n\u001b[0;32m    150\u001b[0m             \u001b[0mvals\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mvalidate\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mmodel_stepper\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mcur_data\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mval_dl\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmetrics\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    151\u001b[0m             \u001b[0mstop\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mFalse\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 152\u001b[1;33m             \u001b[1;32mfor\u001b[0m \u001b[0mcb\u001b[0m \u001b[1;32min\u001b[0m \u001b[0mcallbacks\u001b[0m\u001b[1;33m:\u001b[0m \u001b[0mstop\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mstop\u001b[0m \u001b[1;32mor\u001b[0m \u001b[0mcb\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mon_epoch_end\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mvals\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    153\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mswa_model\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    154\u001b[0m                 \u001b[1;32mif\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mepoch\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m>=\u001b[0m \u001b[0mswa_start\u001b[0m \u001b[1;32mand\u001b[0m \u001b[1;33m(\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mepoch\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;36m1\u001b[0m \u001b[1;33m-\u001b[0m \u001b[0mswa_start\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m%\u001b[0m \u001b[0mswa_eval_freq\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;36m0\u001b[0m \u001b[1;32mor\u001b[0m \u001b[0mepoch\u001b[0m \u001b[1;33m==\u001b[0m \u001b[0mtot_epochs\u001b[0m \u001b[1;33m-\u001b[0m \u001b[1;36m1\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m<ipython-input-13-9aafc4cf57fe>\u001b[0m in \u001b[0;36mon_epoch_end\u001b[1;34m(self, metrics)\u001b[0m\n\u001b[0;32m      4\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      5\u001b[0m     \u001b[1;32mdef\u001b[0m \u001b[0mon_epoch_end\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mmetrics\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 6\u001b[1;33m         \u001b[0mpdb\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mset_trace\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\bdb.py\u001b[0m in \u001b[0;36mtrace_dispatch\u001b[1;34m(self, frame, event, arg)\u001b[0m\n\u001b[0;32m     50\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdispatch_call\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0marg\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     51\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mevent\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;34m'return'\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 52\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdispatch_return\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0marg\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     53\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mevent\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;34m'exception'\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     54\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mdispatch_exception\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mframe\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0marg\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\Anaconda3\\envs\\fastai\\lib\\bdb.py\u001b[0m in \u001b[0;36mdispatch_return\u001b[1;34m(self, frame, arg)\u001b[0m\n\u001b[0;32m     94\u001b[0m             \u001b[1;32mfinally\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     95\u001b[0m                 \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mframe_returning\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 96\u001b[1;33m             \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mquitting\u001b[0m\u001b[1;33m:\u001b[0m \u001b[1;32mraise\u001b[0m \u001b[0mBdbQuit\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     97\u001b[0m             \u001b[1;31m# The user issued a 'next' or 'until' command.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     98\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstopframe\u001b[0m \u001b[1;32mis\u001b[0m \u001b[0mframe\u001b[0m \u001b[1;32mand\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mstoplineno\u001b[0m \u001b[1;33m!=\u001b[0m \u001b[1;33m-\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mBdbQuit\u001b[0m: "
     ]
    }
   ],
   "source": [
    "learn.fit_opt_sched(one_cycle(0.1, 10, [10,10,5], 0.95, 0.85, 1e-3), callbacks=[SaveValidationLoss()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As seen in the debugger, metrics contains two elements, the first one is our validation loss, the second one is the accuracy. We can then save it and had a function plot we will be able to call later to have the graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "class SaveValidationLoss(Callback):\n",
    "    def on_train_begin(self):\n",
    "        self.val_losses = []\n",
    "    \n",
    "    def on_epoch_end(self, metrics):\n",
    "        self.val_losses.append(metrics[0])\n",
    "    \n",
    "    def plot(self):\n",
    "        plt.plot(list(range(len(self.val_losses))), self.val_losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "save_val = SaveValidationLoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cc3066f34be6481780d2cfb688b6d178",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=25), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  7%|████▋                                                                 | 52/782 [00:01<00:26, 27.80it/s, loss=2.13]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread Thread-10:\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\Sylvain\\Anaconda3\\envs\\fastai\\lib\\threading.py\", line 916, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"C:\\Users\\Sylvain\\Anaconda3\\envs\\fastai\\lib\\site-packages\\tqdm\\_tqdm.py\", line 144, in run\n",
      "    for instance in self.tqdm_cls._instances:\n",
      "  File \"C:\\Users\\Sylvain\\Anaconda3\\envs\\fastai\\lib\\_weakrefset.py\", line 60, in __iter__\n",
      "    for itemref in self.data:\n",
      "RuntimeError: Set changed size during iteration\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                                                                              \n",
      "    0      1.321514   1.333771   0.5328    \n",
      "    1      1.105817   1.150064   0.6001                                                                                \n",
      "    2      1.044877   1.01592    0.6504                                                                                \n",
      "    3      0.980082   1.006362   0.6439                                                                                \n",
      "    4      0.958601   0.956591   0.6698                                                                                \n",
      "    5      0.937474   0.927707   0.6884                                                                                \n",
      "    6      0.92569    0.902543   0.6854                                                                                \n",
      "    7      0.937417   0.891721   0.688                                                                                 \n",
      "    8      0.920953   0.902342   0.6854                                                                                \n",
      "    9      0.941321   1.037726   0.6434                                                                                \n",
      "    10     0.915001   1.021755   0.6488                                                                                \n",
      "    11     0.90355    1.088233   0.6244                                                                                \n",
      "    12     0.897824   0.865036   0.702                                                                                 \n",
      "    13     0.8678     1.051379   0.644                                                                                 \n",
      "    14     0.846668   0.835397   0.7215                                                                                \n",
      "    15     0.85498    0.818815   0.7105                                                                                \n",
      "    16     0.818486   0.830917   0.717                                                                                 \n",
      "    17     0.788532   0.776732   0.7338                                                                                \n",
      "    18     0.756384   0.73861    0.7493                                                                                \n",
      "    19     0.696657   0.656809   0.7835                                                                                \n",
      "    20     0.676386   0.617412   0.7868                                                                                \n",
      "    21     0.628199   0.561833   0.8096                                                                                \n",
      "    22     0.545142   0.520807   0.8246                                                                                \n",
      "    23     0.502239   0.462369   0.8496                                                                                \n",
      "    24     0.512865   0.449487   0.8513                                                                                \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.44948736786842347, 0.8513]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.load('init')\n",
    "learn.fit_opt_sched(one_cycle(0.1, 10, [10,10,5], 0.95, 0.85, 1e-3), callbacks=[save_val])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAD8CAYAAAB9y7/cAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4VdW9//H3OicJISEDGYEwBEIYwhQgDAFlUKhDvaLWAZxqrXWo3lY7e6+//jrea7XWobZWpCo4ltah3tYqIKgIYUgQBMKYQCAMSQhDEiDzun8kcCNTppPsM3xez8OT5Jyds7/72eFz1ll77bWMtRYREfE/LqcLEBGRjqGAFxHxUwp4ERE/pYAXEfFTCngRET+lgBcR8VMKeBERP6WAFxHxUwp4ERE/FeTUjuPi4mxycrJTuxcR8Uk5OTmHrLXxLdnWsYBPTk4mOzvbqd2LiPgkY0xBS7dVF42IiJ9SwIuI+CkFvIiIn1LAi4j4KQW8iIifUsCLiPgpBbyIiJ/yuYDfUVTOL/+RS1VtndOliIh4NZ8L+L1HTvDnz3aRlVfqdCkiIl7N5wJ+UkocYSFuFuUWOV2KiIhX87mADw12MyU1niW5RdTXW6fLERHxWj4X8AAz0xIpLq/ii33HnC5FRMRr+WTAXzIkAbfLsDj3oNOliIh4LZ8M+O7hIYxL7s5i9cOLiJyXTwY8wMy0HmwvqqCg9LjTpYiIeCWfDfivpCUCqBUvInIePhvwfWLCGNIjQsMlRUTOw2cDHhpG02TvPszh49VOlyIi4nV8PuDrLSzdWux0KSIiXsenA35EUhQ9IkM1XFJE5Bx8OuCNMcxIS+DT7YeorNHkYyIiTfl0wEPDcMmTNXWs2HnI6VJERLyKzwf8xAExdOsSpOGSIiJn8PmA7xLkZurgeJZsKdbkYyIiTfh8wEPDTU+HKqr4fO9Rp0sREfEafhHw0wYnEOQy6qYREWnCLwI+qmswEwbEaLikiEgTfhHwADOHJpJXcpz8kgqnSxER8Qp+E/AzNPmYiMiX+E3A9+4eRlrPSAW8iEgjvwl4gK8MSyRnzxFKyqucLkVExHF+FfAz0xKxFpZuVSteRMSvAj6tZyRJ0V3VTSMigp8FvDGGmWmJLN9xiBPVtU6XIyLiqGYD3hjzojGm2Biz6TzP32KM+aLx30pjzCjPl9lyM9MSqaqtZ/kOTT4mIoGtJS34l4HLL/D8LmCqtXYk8EtgrgfqarPx/WOIDNXkYyIizQa8tfZT4PAFnl9prT3S+OMqoLeHamuTYLeL6UMSWLq1mDpNPiYiAczTffDfBP7l4ddstZlpiRw+Xk1OwZHmNxYR8VMeC3hjzHQaAv7HF9jmbmNMtjEmu6SkxFO7PsvUQfEEu43mphGRgOaRgDfGjATmAbOstaXn285aO9dam2GtzYiPj/fErs8pIjSYzJQ4FucWYa26aUQkMLU74I0xfYG3gdustdvbX5JnzExLZHfpCXYWa/IxEQlMLRkm+QaQBQw2xhQaY75pjLnXGHNv4yY/BWKBPxpj1htjsjuw3habObRh8rFFGk0jIgEqqLkNrLVzmnn+LuAuj1XkIT2iQhnZO4rFuUXcP32g0+WIiHQ6v7qT9Uwzhyayfu9RissqnS5FRKTT+XfAD2voplmypdjhSkREOp9fB/zgxAj6xHTVcEkRCUh+HfDGGGYO7cGKvFKOV2nyMREJLH4d8NAwXLK6tp5Pt3fcjVUiIt7I7wN+XHJ3osOCNfmYiAQcvw/4ILeLmUMTWZRbRIW6aUQkgPh9wAPcMrEfFVW1vL2u0OlSREQ6TUAEfHqfaEb1jmL+yt2am0ZEAkZABDzA7ZnJ5JUcZ2XeeedCExHxKwET8F8d2ZOY8BDmr9ztdCkiIp0iYAI+NNjNnPF9WLKliMIjJ5wuR0SkwwVMwAPcMqEfAK+u2uNwJSIiHS+gAr5XdFe+ktaDv6zdQ2VNndPliIh0qIAKeIDbJ/XjyIka/mfDfqdLERHpUAEX8JkDYhmU2I35WRoyKSL+LeAC3hjD7ZnJbNpXxro9R50uR0SkwwRcwANcOzqJiC5BLMja7XQpIiIdJiADPrxLENdn9Ob9jQcoLtdqTyLinwIy4KHhztaaOssbq/c6XYp0oDfX7OGzHYecLkPEEQEb8P3jwpk6KJ7XVhdQU1fvdDnSAd5cs4efvL2Rxxdtc7oUEUcEbMADfH1SP4rLq/hws5b08zcrdh7ikXc30TXYzaZ9xyivrHG6JJFOF9ABP3VQAn1jwjQ/jZ/ZWVzOva/mkBLfjadmp1NXb1m7+7DTZYl0uoAOeLfLcHtmP9buPsLm/cecLkc8oLSiim+8vJYuQW7+fEcGUwfFE+J2kaVZRCUABXTAA9wwtg+hwS5eySpwuhRpp8qaOu5+JYfisirmfT2D3t3DCA12M7pvNFn5CngJPAEf8FFhwVw7Ool31+/j6Ilqp8uRNrLW8sO/fUFOwRGevCmd9D7Rp5/LTIll8/4yjp1QP7wEloAPeIDbJiZTWVPPwmwNmfRVTy7Zwf9s2M+PLh/MlSN6fum5zAGxWAurd6kVL4FFAQ+k9YpkfHIMr6wqoK5e89P4mrfXFfLMRzu4MaM3901NOev59L7RdAlyddhqXuWVNWTrIq54IQV8o69PSmbv4ZN8vK3Y6VKkFVbnl/Ljt74gc0Asv7pmBMaYs7bpEuQmI7k7qzqoH/7ZpTu58fksist0V7R4FwV8o68MSyQxsgvzdbHVZ+w6dJx7Xs2hT0wYf7p1LCFB5/9zzhwQy9aD5ZRWVHm0BmstH24+SL1F6/2K11HANwp2u7hlQj8+3V5CXkmF0+X4PWstq/JL29zqPXqimjtfXosBXrpjHFFhwRfcPjMlDoDVuzzblZJXcpzdpQ1LQK7YqSkRxLso4JuYPb4PwW6jIZOd4ONtJcyeu4rx//URVzy9nEf/tZWsvFKqa5ufNqK6tp57Xslh35GTzL09g36x4c3+zsjeUYSFuD0+Hv6jLUUAjOoTzcq8Uq0xIF5FAd9EQkQoXx3Rk7dyCqmoqnW6HL/28srdJER04ceXDyEyNIh5y/OZ88IqxvxyMXcvyOa11QXsPXz24ujWWh5+eyOrdx3m8RtGMi45pkX7C3a7GJcc4/Hx8Eu2FJHWM5Lrx/Zm39GTFJRqQXfxHkFOF+Btbp+UzLvr9/POukJuy0x2uhy/tPvQcT7ZXsKDM1K5b1oK901LobyyhpV5pXyyvYRPtpWwKLehZZwSH87UQQlMHRzPhP4xzFuez1vrCnlwRiqz0pNatd/MlFge/ddWissrSYgIbfdxlFZUkVNwhAcuSWVySiwAn+08RHJc858oRDqDAv4Mo/tEMyIpivlZBdw6sd85R2VI+7y6qoAgl+Hm8X1PPxYRGsxlw3pw2bAeWGvJK2l4E/hkewmvri7gxRW76BLkoqq2nmvSe/HdS1Nbvd/MAQ0hvCr/MFeP6tXu41i2rYR6CzOHJtI/LpyeUaGszDvErRP7tfu1RTxBAX+GhiX9+vHDv33BvOW7uCg1jgHx4XQJcjtdml84WV3Hwuy9XD68BwmR525FG2MYmNCNgQnd+OZF/TlZXcfqXQ2t+5PVdfx81rA2vfEO6xVJRJcgsvJKPRLwS3KLSIzswvCkSIwxTB4Yx0dbiqivt7hcahiI85oNeGPMi8BVQLG1dvg5njfA08CVwAngDmvtOk8X2pn+bVQvnvs4j1+/vwVomJSsX2wYgxMjSE2MYFBiNwYlRpAcG37BoXlytr+v30dZZS23t6L7q2uIm2mDE5g2OKFd+w5yuxjfP8Yj4+Era+r4dEcJ145OOv1mM3lgLH/LKST3QBnDk6LavQ+R9mpJC/5l4FlgwXmevwJIbfw3AXiu8avPCg1288GDU8g/VMH2ogq2Hyxne1E5Ww+Wnx7zDBDkMvSPC2dQYgSpid0Y3iuKS4YkqPV2HtZaFmQVMKRHBOOSuztSQ2ZKLB9tLebgsUp6RLW9Hz4rv5QT1XXMSEs8/dikxqGYK3YeUsCLV2g24K21nxpjki+wySxggW0YH7bKGBNtjOlprT3goRodERLkYkiPSIb0iIRR//d4ZU0deSUV7CiqYFtROTuKytm47xjvbzqAtQ2t/9/eMFJdOuewbs8Rcg+U8V/XnvuO084wsbEfPiv/ENeO7t3m11mSW0RYiPt0vz5AYmQoAxO6sSKvlHvOMWWCSGfzRB98EtB0lq7Cxsd8OuDPJzTYzbBeUQzr9eUW2snqOl5auYvHPthGUVklL9yW0ezNN4Fm/soCIkKDuGZ0+/u/2yqtZyRRXYPJyittc8Bba1mypYgpqfGEBn/5jXxySiwLswuprq1X9504zhN/gedqip3zbg9jzN3GmGxjTHZJSYkHdu09uoa4+fa0gTw9O531e47ytT+tPOc47kBVUl7FvzYd4PqxvQkLce7avstlmNC/fePhN+0ro6is6kvdM6dMHhjHyZo6Pt9zpD1liniEJwK+EOjT5OfewP5zbWitnWutzbDWZsTHx3tg195nVnoSC745nuKySq57biWb9mmlKGhYALumznKbFwwhzEyJZe/hk21+A168pQiXgemDz/4bnjAgFpfRtAXiHTwR8O8Bt5sGE4Fjvt7/3l4TB8Ty1n2TCHG7uPH5LJYF+AyVtXX1vLZ6DxenxjEgvpvT5Zy+GNrWVvyS3CLG9utObLcuZz0X1TWYEb2jWaGJx8QLNBvwxpg3gCxgsDGm0BjzTWPMvcaYexs3eR/IB3YCLwDf7rBqfUhqYgTvfHsS/ePCuWt+Nm+s2eN0SY5ZnFvEwbLKVg2N7EiDErsRGx7CqjaE8L6jJ8k9UMaMoWd3z5wyOSWWDXuParoLcVxLRtHMaeZ5C9zvsYr8SEJkKAvvyeT+19fx8Nsb2XfkJN//yqCAuzt2QVYBSdFduWRI+8axe4oxhokDYsnKb5gcrDXn49TkYufqfz/looFx/PHjPNbsKuWSIeffTqSj6TJ/BwvvEsS82zOYPa4Pzy7byfcWbmjRjIn+YkdROVn5pdw6sR9uL7o/YGJKLAeOVbZ6crDFuUUMiAsn5QJdTWP6dadLkIvPdqibRpylgO8EQW4X/33dCH542WDe+Xwfd7y0hrLKwFgAekFWASFBLm4a16f5jTtR5unx8C0P4fLKGlbll3Lp0At/EgkNblhBamWeLrSKsxTwncQYw/3TB/K7G0exZtdhbngui/1HTzpdVocqr6zh7XWFXDWyJzHhIU6X8yUp8eHER3Rp1fzwy3ccoqbOXrD//ZRJKXFsPVjOIQ+vICXSGgr4TnbdmN7Mv3M8+4+e5No/rmDzfv8dRvnO5/s4Xl3nNRdXmzLGkNmkH74lluQWER0WzNh+zU+zcNHAhpE6WsZPnKSAd8DkgXH89b5MXMYw69kV/Mc7G/2uNX9q3plRvaNI7xPtdDnnlJkSS0l5FXklx5vdtraunqXbirlkcAJB7ub/2wxPiiIyNIgVO9RNI85RwDtkSI9I3nvgIuaM78tfs/cy7fGP+enfN1HUxjVKvU1WXik7iyu8etGU1vTD5xQc4eiJmguOnmnK7WoYqbNC/fDiIAW8g+IjuvDLa4bz8Q+n87WxvXl99R4ufmwZv/ifXIrLfTvoF2QV0D0smKtG9nS6lPPqFxtGz6jQFo2HX7KliBC3iymDWn4H9uSBcRQeOckeLeMnDlHAe4Gk6K7893UjWPr9acwa1Yv5WbuZ8tgy/uv9LZT64EW6/UdPsnhLETeN63vWZFze5FQ//Kpm+uGttSzOLWJiSizdurR8Hp3Jjf3wasWLUxTwXqRvbBiP3zCKJd+byhXDezJveT4XP7aMxz7YypHj1U6X12Kvr95DvbXcMqFv8xs7bGJKLKXHq9leVHHebfJKjrO79AQzmxkeeaaU+HASI7toXhpxjALeC/WPC+fJm9JZ9NAULh2ayHOf5HHxY8v43eLtHDvp3ePnq2rreHPtHi4dkkCfmDCny2nWpMbFsrMu0Mpe0nj36qUtGB7ZlDGGySlxrMwrpb6+ZSN1RDxJAe/FBiZE8Ps5o/ngu1O4ODWOZz7awUW/WcoTi7Z5bR/9B5sOcqii2qsvrjbVu3sYfWK6XvBC65LcIob1iqRXdNdWv/6kgXEcPl7N1oPl7SlTpE0U8D5gcI8Inrt1LP/8zkVkDojl2WU7uejRZXx/4QZy95c5Xd6XzF+5m/5x4Vzc2P/sCxr64Q+fs5VdWlFFzp4jLbq56VwmD2z4hKC7WsUJCngfMqxXFHNvz2Dp96cxZ3wf3t94gCufWc4t81axdGuR490Am/YdY92eo9w6sZ9PrUubmRLLsZM15B44+81y6dZirIWZLRweeaaeUV0ZEB+ufnhxhALeB/WPC+fns4az6uFL+fHlQ8grPs6dL2cz48lPeHVVASer6xyp65WsAroGu7l+bNvXOnVC5oCGTxurztFNs2RLET0iQxnWK7LNrz85JY7Vuw4H1CRz4h0U8D4sKiyY+6alsPzH03l6djrhIUE88u4mMh/9iMc/3NqpN00dPVHN3zfs45rRvYjq6ltr0faICqV/XPhZ89JU1tTx6fZDzEhLaNcUz5MHxnKiuo4NhUfbW6pIqyjg/UCw28Ws9CTee2AyC+/JZHxyDH/8OI+LfrOU7/1lPRsLj7V4vpW2+mt2IZU19dw2MblD99NRJg6IZc2uw9TW/V8rOyuvlJM1dW3uf2/62kbL+IkDnFv9WDzOGMP4/jGM7x9DQelxXlqxm4XZe3n7831EhgaR1iuStJ5RDOsVybCkSAbGd2vRvCpnOnK8mtwDZeTuLzv9dWdJBeOSu5PWjq4MJ2WmxPLGmj1s3l/GqMa5cxZvKSI8xE1m41DKtooOC2FEUhQrd5by4AxPVCvSMgp4P9UvNpyfXT2Mh2YO4v2NB9i47xib95fx2uoCqhr7gkOCXAzpEcGwXpGk9WoI/iE9IggLafizqK+37D1y4ktBnnugjAPH/q/rp0dkKGm9IpmZlsgcH7ix6XwmDogBGualGdUnGmstH20pYsqgeLoEtf9u3Ekpccxbns/xqlrCW3E3rEh76C/Nz0V1DWbO+L6cWnextq6e/EPHyd1fxub9DaH//saDvLFmLwAu03ARNzoshG0Hy0+vK+p2GVLiw5nQP+b0J4GhPSPOufC0L0qICGVgQjey8kq5d2oKm/aVUVRW1e7umVMmD4zlT5/ksWb3YaYP9o6lC8X/KeADTJDbxaDECAYlRnDN6CSgYa6VfUdPsnl/WWPwl1FWWcPXxiSdDvPUxG5ePa+MJ2QOiOWtdYXU1NWzeEsRLgPTPbSObEa/GELcLlbuPKSAl06jgBeMMfTuHkbv7mFcNqyH0+U4JjMllldWFfBF4TGW5BaR0S/GYytRdQ1xM7Zfd1bs1AIg0nk0ikak0cTG+eHfWldI7oEyZqR5tqU9eWAsuQfKfHKGUPFNCniRRjHhIQzpEcGba/YAeKz//ZRJjdM3tGahb5H2UMCLNJGZEku9hQHx4QyI7+bR1x6ZFEVElyB100inUcCLNHFqGb+ZHm69Q8MF7gkDYjXxmHQaBbxIExelxnHVyJ7MHt8xY/onD4yloPQEew9rGT/peAp4kSbCQoJ49uYx9I8L75DXP7WMn1rx0hkU8CKdKDWhG/ERXdQPL51CAS/SiRqW8Wvoh+/oCeBEFPAinWzSwDgOVVSzrUjL+EnHUsCLdLJT/fDqppGOpqkKRDpZUnTDMn5PLNrG6vxSLhmSwLTBCfSICvXofurrLbX1lpAgteMClQJexAF/unUs81fuZtnWYhblFgGQ1jOSS4YkMH1IAul9onG3cl3bkvIq1u89yoa9Rxu+Fh6lrt7yh5vHeGzSNPEtxqkLPRkZGTY7O9uRfYt4C2st24rKWba1hGVbi8nZc4S6ekv3sGCmDopn+pAEpg6KJzrsy5OenayuY+O+Y6fDfP3eo+w7ehJomNp5cGIE6X2j2bD3KNuLynlm9miuGNHTiUMUDzPG5FhrM1q0rQJexHscO1HDJzsawv7jbcUcOVGDy8CYvt2ZPDCO4sZW+vaicurqG/7vJkV3Jb1vNKP7RDOqTzTDe0XRNaRhaueyyhq+8dJaPt9zhN/eMIrrxvjWguhyNgW8iB+oq7es33uUj7cVs3RrMZv3lxERGkR6n2jS+0QzqndDoMdHXHjRlRPVtXxrQTYr80r51TXDuWVCv046AukIHg94Y8zlwNOAG5hnrX30jOf7AvOB6MZtfmKtff9Cr6mAF2mdssoauoUE4Wpl3zxAZU0d335tHUu3FvPIV4dy18UDOqBC6QytCfhmL68bY9zAH4ArgDRgjjEm7YzNHgEWWmtHA7OBP7auZBFpTmRocJvCHSA02M2fbh3LV0f05Ff/3MLTS3boRqsA0JJRNOOBndbafABjzJvALCC3yTYWiGz8PgrY78kiRaT9QoJcPD07ndBgN08u2c6Jmlp+cvkQjGnbm4Z4v5YEfBKwt8nPhcCEM7b5GbDIGPPvQDgwwyPViYhHBbldPH79SLqGuHj+k3xOVtfxs38b1uZPBuLdWhLw5zrzZ362mwO8bK19whiTCbxijBlura3/0gsZczdwN0Dfvh0zHauIXJjLZfjlrOGEhQQx99N8TlTX8ZuvjWz1uHvxfi0J+EKgT5Ofe3N2F8w3gcsBrLVZxphQIA4obrqRtXYuMBcaLrK2sWYRaSdjDA9fMYTwkCCeXLKdkzV1PHVTOsFu3fXqT1pyNtcCqcaY/saYEBouor53xjZ7gEsBjDFDgVCgxJOFiohnGWP47oxU/vPKofzziwPc92oOlTV1TpclHtRswFtra4EHgA+BLTSMltlsjPmFMebqxs2+D3zLGLMBeAO4w+oSvYhP+NaUAfzqmuEs2VLMXfOzOVFd63RJ4iG60UlEAHgrp5Af/m0DQ3tG8uRN6QxKjHC6JDkHj46DF5HA8LWxvZl7WwYHjlVy1TOf8dzHedTW1Tf/i+K1FPAictqMtEQWPTSFS4cm8JsPtnL9n7LYWVzhdFnSRgp4EfmSuG5d+OMtY3hmzmh2lx7nymeW88Kn+acnNxPfoYAXkbMYY7h6VC8WPTSFqYPi+fX7W7jx+SzyS9Sa9yUKeBE5r4SIUObeNpanbkpnZ3EFVz6znBc/20W9WvM+QQEvIhdkjOGa0UksemgKk1Pi+MU/cpk9dxUFpcedLk2aoYAXkRZJjAxl3tcz+O0No9hysIzLn1rOgqzdas17MQW8iLSYMYbrx/Zm0UNTGN8/hp/+fTO3zFtNcVml06XJOSjgRaTVekZ15eVvjOM3XxvB+r1H+c6bn6sl74UU8CLSJsYYbhrXl5/PGsaq/MO8uGKX0yXJGRTwItIuN4ztzcy0RB77YBvbDpY7XY40oYAXkXYxxvDf140gsmsQD/5lPVW1mpHSWyjgRaTd4rp14dHrRrLlQBlPLdnhdDnSSAEvIh4xIy2R2eP68PwneazdfdjpcgQFvIh40CNXpZHUvSvfW7ieiirNK+80BbyIeEy3LkE8eWM6+46c5Ff/yHW6nICngBcRj8pIjuGeqSm8uXYvi3OLnC4noCngRcTjHpoxiKE9I3n47S84VFHldDkBSwEvIh4XEuTiqZvSKTtZy8Nvb0RLNDtDAS8iHWJwjwh+dPlgFucW8decQqfLCUgKeBHpMHdO7s/EATH8/L3N7D18wulyAo4CXkQ6jMtl+O0No3AZw/cXbtCyf51MAS8iHap39zB+dvUw1uw+zAvL850uJ6Ao4EWkw103JokrhvfgiUXbyN1f5nQ5AUMBLyIdzhjDr68dQVTXEL63UBOSdRYFvIh0ipjwEB6/fiRbD5bzu0XbnS4nIAQ5XYCIBI7pQxK4ZUJfnv80n+q6en58+RBCg91Ol+W3FPAi0qn+31VpBLtdvLRiN8t3HOKpm9IZnhTldFl+SV00ItKpQoPd/OzqYSy4czzllTVc84cV/GHZTmrr6p0uze8o4EXEEVMGxfPhg1O4bHgPHv9wGzfNXUVB6XGny/IrCngRcUx0WAjPzhnN07PT2V5UzhVPL+eNNXs0d42HKOBFxFHGGGalJ/Hhg1MY3Teah9/eyF3zsykp1yyU7aWAFxGv0Cu6K6/cOYGfXpXG8p2HuOypT/lw80Gny/JpCngR8Roul+HOi/rzz3+/iJ5RodzzSg4/+tsGyitrnC7NJyngRcTrpCZG8M63J/PA9IH8LaeQK55eTrYW8m41BbyIeKWQIBc/uGwwf703E7fLcPO81Xy8rdjpsnxKiwLeGHO5MWabMWanMeYn59nmRmNMrjFmszHmdc+WKSKBamy/GN799mRSE7px94Iclmid1xZrNuCNMW7gD8AVQBowxxiTdsY2qcDDwGRr7TDgwQ6oVUQCVPfwEF6/ayJDe0Zw32s5fLBJF19boiUt+PHATmttvrW2GngTmHXGNt8C/mCtPQJgrdXnKBHxqKiwYF65awLDk6K4//V1/POLA06X5PVaEvBJwN4mPxc2PtbUIGCQMWaFMWaVMebyc72QMeZuY0y2MSa7pKSkbRWLSMCKDA1mwZ3jGdM3mu+8+Tl/X7/P6ZK8WksC3pzjsTNvMwsCUoFpwBxgnjEm+qxfsnautTbDWpsRHx/f2lpFRIgIDeblb4xnXHJ3HvrLet7Sgt7n1ZKALwT6NPm5N7D/HNv83VpbY63dBWyjIfBFRDwuvEsQL90xnsyUWH7wtw0sXLu3+V8KQC0J+LVAqjGmvzEmBJgNvHfGNu8C0wGMMXE0dNlo8UUR6TBdQ9z8+evjuDg1nh+99QWvrS5wuiSv02zAW2trgQeAD4EtwEJr7WZjzC+MMVc3bvYhUGqMyQWWAT+01pZ2VNEiItAw9fDc28ZyyZAE/vOdTcxfudvpkryKcWrWtoyMDJudne3IvkXEv1TX1nP/6+tYnFvEI18dyl0XD3C6pA5jjMmx1ma0ZFvdySoiPi8kyMUfbxnDFcN78Kt/buFPn+Q5XZJXUMCLiF8Idrt4Zs5orhrZk0f/tZXff7TD6ZIcpzVZRcRvBLtdPHVTOsFuF08s3k5xeRX/+dWhAbuwtwJeRPxKkNvFb28YRWx4CPM+28Xa3Yf5/ZzRpCZGOF3okEFOAAAGjUlEQVRap1MXjYj4HbfL8MhVabx0xzhKyqu46vef8eqqgoBbClABLyJ+a/qQBP714MWM7x/DI+9u4t5Xczh6otrpsjqNAl5E/FpCRCjzvzGe/7hyCEu3FnPF08tZlR8Yt+ko4EXE77lchrunpPD2fZMJDXYz54VVPLFoG7V19U6X1qEU8CISMEb0juIf/34RXxvTm98v3cmNz2ex9/AJp8vqMAp4EQko4V2C+O0No3hmzmh2FFVw5dPLeW/DmfMn+gcFvIgEpKtH9eL9717MwMRufOeNz/nBXzdwvKrW6bI8SgEvIgGrT0wYC+/J5IHpA3lrXSH/9vvP/KrLRgEvIgEt2O3iB5cN5vW7JnKoooqb563iwLGTTpflEQp4EREgMyWWV745gaPHa7j5hdUUl1U6XVK7KeBFRBqN6hPNy3eOo6iskpvnreZQRZXTJbWLAl5EpImx/WJ48Y5xFB45wa3zVnPkuO/e+aqAFxE5w8QBscy7fRz5h45z+4trOHayxumS2kQBLyJyDhelxvH8rWPZerCMO15aQ4UPDqFUwIuInMf0IQk8e/MYvig8xp0vreVEtW+FvAJeROQCLhvWg6dnp5NdcJi75mdTWVPndEktpoAXEWnGVSN78cSNo8jKL+WeV3KoqvWNkFfAi4i0wLWje/PodSP4ZHsJ97/2OdW13j8TpQJeRKSFbhrXl1/OGsaSLUV8983PvX66YQW8iEgr3JaZzCNfHcq/Nh3k+3/dQF299y4DqEW3RURa6a6LB1BdV89jH2zj8PFqvpKWyJh+3RmcGEGQ23vazQp4EZE2+Pa0gQS5DPOW72L5jkMAhIe4Se8bzdi+3RnTrzuj+3YnqmuwYzUap1YZz8jIsNnZ2Y7sW0TEU6y1FB45ybo9R8gpaPi35UAZ9RaMgdSEboztF8PYft0Z2687ybFhGGPavD9jTI61NqNF2yrgRUQ863hVLRv2Hm0I/D1HWFdwhLLKhpukYsJDuG9qCt+aMqBNr92agFcXjYiIh4V3CWLSwDgmDYwDoL7ekldScbqFnxgV2il1KOBFRDqYy2VITYwgNTGC2eP7dt5+O21PIiLSqRTwIiJ+SgEvIuKnFPAiIn5KAS8i4qcU8CIifkoBLyLipxTwIiJ+yrGpCowxJUBBG389DjjkwXJ8TSAffyAfOwT28evYG/Sz1sa35JccC/j2MMZkt3QuBn8UyMcfyMcOgX38OvbWH7u6aERE/JQCXkTET/lqwM91ugCHBfLxB/KxQ2Afv469lXyyD15ERJrnqy14ERFphs8FvDHmcmPMNmPMTmPMT5yupzMZY3YbYzYaY9YbY/x+OSxjzIvGmGJjzKYmj8UYYxYbY3Y0fu3uZI0d5TzH/jNjzL7G87/eGHOlkzV2FGNMH2PMMmPMFmPMZmPMdxsfD5Rzf77jb/X596kuGmOMG9gOzAQKgbXAHGttrqOFdRJjzG4gw1obEGOBjTFTgApggbV2eONjjwGHrbWPNr7Bd7fW/tjJOjvCeY79Z0CFtfa3TtbW0YwxPYGe1tp1xpgIIAe4BriDwDj35zv+G2nl+fe1Fvx4YKe1Nt9aWw28CcxyuCbpINbaT4HDZzw8C5jf+P18Gv7w/c55jj0gWGsPWGvXNX5fDmwBkgicc3++4281Xwv4JGBvk58LaeOB+ygLLDLG5Bhj7na6GIckWmsPQMN/BCDB4Xo62wPGmC8au3D8souiKWNMMjAaWE0Anvszjh9aef59LeDNOR7znT6m9ptsrR0DXAHc3/gxXgLHc0AKkA4cAJ5wtpyOZYzpBrwFPGitLXO6ns52juNv9fn3tYAvBPo0+bk3sN+hWjqdtXZ/49di4B0auqwCTVFjH+Wpvspih+vpNNbaImttnbW2HngBPz7/xphgGsLtNWvt240PB8y5P9fxt+X8+1rArwVSjTH9jTEhwGzgPYdr6hTGmPDGCy4YY8KBrwCbLvxbfuk94OuN338d+LuDtXSqU+HW6Fr89PwbYwzwZ2CLtfZ3TZ4KiHN/vuNvy/n3qVE0AI1Dg54C3MCL1tpfO1xSpzDGDKCh1Q4QBLzu78dujHkDmEbDTHpFwP8H3gUWAn2BPcAN1lq/uxh5nmOfRsPHcwvsBu451SftT4wxFwHLgY1AfePD/0FDP3QgnPvzHf8cWnn+fS7gRUSkZXyti0ZERFpIAS8i4qcU8CIifkoBLyLipxTwIiJ+SgEvIuKnFPAiIn5KAS8i4qf+Fzgy4NRcvvJiAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "save_val.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "People often like to train a neural net with constant learning rates, dividing it automatically if the validation loss didn't progres. This is easily done with callbacks again, we just have to pass a few arguments:\n",
    "\n",
    "- the learner object, to be able to modify its learning rate (with sched.layer_opt.set_lrs)\n",
    "- the first learning rate\n",
    "- the number to divide it\n",
    "\n",
    "We can even tell the learner to stop the training through the callback by using return True. Here we specify a minimum learning rate, and once we are below it, we stop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LRScheduler(Callback):\n",
    "    def __init__(self, learn, init_lr, div_lr, min_lr):\n",
    "        self.learn, self.init_lr, self.div, self.min_lr = learn, init_lr, div_lr, min_lr\n",
    "    \n",
    "    def on_train_begin(self):\n",
    "        self.first_epoch = True\n",
    "        self.best_loss = 0.\n",
    "        self.current_lr = self.init_lr\n",
    "        self.learn.sched.layer_opt.set_lrs(self.current_lr)\n",
    "    \n",
    "    def on_epoch_end(self, metrics):\n",
    "        val_loss = metrics[0]\n",
    "        if self.first_epoch:\n",
    "            self.best_loss = val_loss\n",
    "            self.first_epoch = False\n",
    "        elif val_loss > self.best_loss:\n",
    "            self.current_lr /= self.div\n",
    "            if self.current_lr < self.min_lr: return True\n",
    "            else: self.learn.sched.layer_opt.set_lrs(self.current_lr)\n",
    "        else: self.best_loss = val_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = ConvLearner.from_model_data(model, data)\n",
    "learn.crit = F.cross_entropy\n",
    "learn.opt_fn = partial(optim.SGD, momentum=0.9)\n",
    "learn.metrics = [accuracy]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_sched = LRScheduler(learn, 0.1, 10,1e-4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fe3a8faae4e44f80b9a932209e076e00",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1000), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                                                                              \n",
      "    0      1.265199   1.222944   0.5689    \n",
      "    1      1.035041   1.032537   0.6321                                                                                \n",
      "    2      0.936886   0.87412    0.6865                                                                                \n",
      "    3      0.866005   0.834753   0.7139                                                                                \n",
      "    4      0.81119    0.730867   0.7478                                                                                \n",
      "    5      0.764424   0.708939   0.7559                                                                                \n",
      "    6      0.696581   0.657594   0.7718                                                                                \n",
      "    7      0.69288    0.65185    0.7756                                                                                \n",
      "    8      0.6514     0.599916   0.793                                                                                 \n",
      "    9      0.644742   0.596359   0.7967                                                                                \n",
      "    10     0.631083   0.579257   0.8036                                                                                \n",
      "    11     0.595398   0.580039   0.7993                                                                                \n",
      "    12     0.494971   0.495061   0.8317                                                                                \n",
      "    13     0.501904   0.494321   0.8309                                                                                \n",
      "    14     0.507604   0.481217   0.835                                                                                 \n",
      "    15     0.474797   0.479913   0.8363                                                                                \n",
      "    16     0.469313   0.472549   0.8408                                                                                \n",
      "    17     0.476938   0.473378   0.839                                                                                 \n",
      "    18     0.454314   0.469444   0.8414                                                                                \n",
      "    19     0.459996   0.471778   0.8411                                                                                \n",
      "    20     0.458264   0.469791   0.8398                                                                                \n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.4697907167434692, 0.8398]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.load('init')\n",
    "learn.fit(0.1, 1000, callbacks=[lr_sched]) #Big number in fit to be sure we reach the point the learner stops by itself."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this wasn't as efficient as a 1cycle in this situation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's see how we can have the dropout probability change through training. We could begin with 0 and slowly increase to a maximum value. First let's add some dropout in the model (note that is just for the sake of showing how to do it since we weren't overfitting before, so dropout probably won't help much)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "layers = [ConvBlock(3,64), \n",
    "          ConvBlock(64,64,stride=2), #size 16x16\n",
    "          ConvBlock(64,128,stride=2), #size 8x8\n",
    "          ConvBlock(128,256,stride=2), #size 4x4\n",
    "          nn.Dropout(0.2),\n",
    "          ConvBlock(256,10, stride=2), #size 2x2 \n",
    "          nn.AdaptiveAvgPool2d(1), \n",
    "          Flatten()]\n",
    "model = nn.Sequential(*layers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we identify where it is in the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "  (0): Sequential(\n",
       "    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
       "    (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (2): ReLU(inplace)\n",
       "  )\n",
       "  (1): Sequential(\n",
       "    (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
       "    (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (2): ReLU(inplace)\n",
       "  )\n",
       "  (2): Sequential(\n",
       "    (0): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
       "    (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (2): ReLU(inplace)\n",
       "  )\n",
       "  (3): Sequential(\n",
       "    (0): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
       "    (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (2): ReLU(inplace)\n",
       "  )\n",
       "  (4): Dropout(p=0.2)\n",
       "  (5): Sequential(\n",
       "    (0): Conv2d(256, 10, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)\n",
       "    (1): BatchNorm2d(10, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (2): ReLU(inplace)\n",
       "  )\n",
       "  (6): AdaptiveAvgPool2d(output_size=1)\n",
       "  (7): Flatten()\n",
       ")"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's the layer of index 4. We can access the p of the dropout by model[4].p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model[4].p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's all we need to create our dropout scheduler! I'll use the class DecayScheduler to have something giving me the values going from 0 to the max. It's the one that's used to create the lr_decay or momentum_decay in the new training API.\n",
    "\n",
    "It's fairly simple: you specify a decay type (linear, exponential...), a number of iterations, a min value and a max value, then you just have to call next_val each time you want to update your parameter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "nb_batches = len(data.trn_dl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need the number of batches to give our DecayScheduler the number of iterations, then we just update the p value accordingly. We have a list of schedulers, one for each phase, so we also update the index of the current phase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DropoutScheduler(Callback):\n",
    "    def __init__(self, dp, scheds):\n",
    "        self.dp = dp\n",
    "        self.phase = 0\n",
    "        self.scheds = scheds\n",
    "        \n",
    "    def on_train_begin(self):\n",
    "        self.phase = 0\n",
    "    \n",
    "    def on_phase_begin(self):\n",
    "        self.sched = self.scheds[self.phase]\n",
    "    \n",
    "    def on_phase_end(self):\n",
    "        self.phase += 1\n",
    "    \n",
    "    def on_batch_begin(self):\n",
    "        self.dp.p = self.sched.next_val()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp_phases = [DecayScheduler(DecayType.LINEAR, nb_batches * 10, 0, 0.1), #first phase: go from 0 to 0.1\n",
    "             DecayScheduler(DecayType.LINEAR, nb_batches * 10, 0, 0.2), #second phase: from 0.1 to 0.2\n",
    "             DecayScheduler(DecayType.NO, nb_batches * 5, 0.2),] #then stay at 0.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "dp_sched = DropoutScheduler(learn.model[4], dp_phases)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "655ee15a7d434f51a5ae73279cce1114",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n",
       "  that the widgets JavaScript is still loading. If this message persists, it\n",
       "  likely means that the widgets JavaScript library is either not installed or\n",
       "  not enabled. See the <a href=\"https://ipywidgets.readthedocs.io/en/stable/user_install.html\">Jupyter\n",
       "  Widgets Documentation</a> for setup instructions.\n",
       "</p>\n",
       "<p>\n",
       "  If you're reading this message in another frontend (for example, a static\n",
       "  rendering on GitHub or <a href=\"https://nbviewer.jupyter.org/\">NBViewer</a>),\n",
       "  it may mean that your frontend doesn't currently support widgets.\n",
       "</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=25), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 22%|███████████████                                                      | 170/782 [00:05<00:21, 28.63it/s, loss=1.86]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception in thread Thread-100:\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\Sylvain\\Anaconda3\\envs\\fastai\\lib\\threading.py\", line 916, in _bootstrap_inner\n",
      "    self.run()\n",
      "  File \"C:\\Users\\Sylvain\\Anaconda3\\envs\\fastai\\lib\\site-packages\\tqdm\\_tqdm.py\", line 144, in run\n",
      "    for instance in self.tqdm_cls._instances:\n",
      "  File \"C:\\Users\\Sylvain\\Anaconda3\\envs\\fastai\\lib\\_weakrefset.py\", line 60, in __iter__\n",
      "    for itemref in self.data:\n",
      "RuntimeError: Set changed size during iteration\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch      trn_loss   val_loss   accuracy                                                                              \n",
      "    0      1.335023   1.206959   0.573     \n",
      "    1      1.099644   1.171765   0.5953                                                                                \n",
      "    2      1.063678   1.023522   0.6493                                                                                \n",
      "    3      1.022308   0.922295   0.679                                                                                 \n",
      "    4      0.979578   1.005916   0.6481                                                                                \n",
      "    5      0.948324   0.983416   0.6652                                                                                \n",
      "    6      0.924334   0.926138   0.6809                                                                                \n",
      "    7      0.920498   0.915692   0.6822                                                                                \n",
      "    8      0.907746   1.216691   0.5947                                                                                \n",
      "    9      0.920743   1.165621   0.6061                                                                                \n",
      "    10     0.892298   1.075023   0.63                                                                                  \n",
      "    11     0.928535   0.956639   0.6784                                                                                \n",
      "    12     0.855941   1.143143   0.6338                                                                                \n",
      "    13     0.905828   0.963923   0.6736                                                                                \n",
      "    14     0.84746    0.83386    0.7176                                                                                \n",
      "    15     0.816802   0.773124   0.7346                                                                                \n",
      "    16     0.837587   0.864828   0.7118                                                                                \n",
      "    17     0.798023   0.949911   0.6705                                                                                \n",
      "    18     0.760953   0.731981   0.7502                                                                                \n",
      "    19     0.694424   0.611342   0.7982                                                                                \n",
      "    20     0.677468   0.607355   0.7948                                                                                \n",
      "    21     0.616413   0.605901   0.7919                                                                                \n",
      "    22     0.560451   0.502508   0.8352                                                                                \n",
      "    23     0.529024   0.461323   0.8466                                                                                \n",
      "    24     0.481621   0.447865   0.8508                                                                                \n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[0.4478648525238037, 0.8508]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.load('init')\n",
    "learn.fit_opt_sched(one_cycle(0.1, 10, [10,10,5], 0.95, 0.85, 1e-3), callbacks=[save_val])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}