Tying_out_training.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Tying_out_training.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "FU0TYajRTt23"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/davidcpage/Imagenette-experiments/blob/master/Tying_out_training.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5QHvmGOBpYuT",
        "colab_type": "text"
      },
      "source": [
        "The aim of today's notebook is to setup training runs on fastai and fastai2 codebases which achieve identical results after fixing random seeds. We continue to use Nvidia DALI for dataloading."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FU0TYajRTt23",
        "colab_type": "text"
      },
      "source": [
        "### Setup"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tZSzbQ9Ag47l",
        "colab_type": "text"
      },
      "source": [
        "Install fastai2 and DALI. You may need to restart afterwards."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vLGuUwFpLzc1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python -m pip install typeguard\n",
        "!python -m pip install --upgrade pillow fastprogress\n",
        "!python -m pip install git+https://github.com/fastai/fastai2\n",
        "!python -m pip install --extra-index-url https://developer.download.nvidia.com/compute/redist/cuda/10.0 nvidia-dali\n",
        "\n",
        "RANGER = 'https://raw.githubusercontent.com/lessw2020/Ranger-Mish-ImageWoof-5/master/ranger.py'\n",
        "UTILS = 'https://raw.githubusercontent.com/davidcpage/Imagenette-experiments/master/utils.py'\n",
        "!wget $RANGER -O ranger.py\n",
        "!wget $UTILS -O utils.py"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i7c9ywHZhNiT",
        "colab_type": "text"
      },
      "source": [
        "### Main"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ch4lAVe4quvT",
        "colab_type": "text"
      },
      "source": [
        "Imports, device setup and dataset download:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FbkdYs8IM37W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from functools import partial\n",
        "import numpy as np\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import fastai, fastai.vision\n",
        "import fastai2, fastai2.callback.all\n",
        "\n",
        "import ranger\n",
        "from utils import *\n",
        "\n",
        "\n",
        "data_dir = fastai.datasets.untar_data(fastai.datasets.URLs.IMAGENETTE_160)\n",
        "device = torch.device(torch.cuda.current_device())\n",
        "\n",
        "torch.backends.cudnn.benchmark = True"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UuJ5Rcx2HZm-",
        "colab_type": "text"
      },
      "source": [
        "Today is about being deterministic:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5XAxQJUoj5RE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "torch.backends.cudnn.deterministic = True"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fP-8yRCPHj2S",
        "colab_type": "text"
      },
      "source": [
        "DALI DataLoaders"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_wO66U5HPRvI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "size = 128\n",
        "bs = 64\n",
        "random_aspect_ratio = (3/4, 4/3)\n",
        "random_area = (0.35, 1.)\n",
        "val_xtra_size = 32\n",
        "\n",
        "to = lambda dtype: (lambda b: (b[0].to(dtype), b[1]))\n",
        "\n",
        "train_dl = lambda folder, bs, dtype=torch.float16, seed=-1: (\n",
        "        Map(to(dtype), DALIDataLoader(imagenet_train_graph(folder, size, random_aspect_ratio, random_area), bs, drop_last=True, device=device, seed=seed)))\n",
        "valid_dl = lambda folder, bs, dtype=torch.float16: Chain(\n",
        "        Map(to(dtype), DALIDataLoader(imagenet_valid_graph(folder, size, val_xtra_size), bs, drop_last=False, device=device)),\n",
        "        Map(to(dtype), DALIDataLoader(imagenet_valid_graph(folder, size, val_xtra_size, mirror=1), bs, drop_last=False, device=device)),\n",
        "    )\n",
        "\n",
        "data_v1 = lambda data_dir=data_dir, bs=bs, dtype=torch.float16, seed=-1: MockV1DataBunch(train_dl(data_dir/'train', bs, dtype, seed), valid_dl(data_dir/'val', bs, dtype))\n",
        "data_v2 = lambda data_dir=data_dir, bs=bs, dtype=torch.float16, seed=-1: fastai2.basics.DataBunch(train_dl(data_dir/'train', bs, dtype, seed), valid_dl(data_dir/'val', bs, dtype))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C4r2i0K7ITC5",
        "colab_type": "text"
      },
      "source": [
        "It turns out that there's a small discrepancy in cosine annealing schedules between the v1 and v2 codebases. The v2 API for param schedules (as functions on the unit interval) is nice and simple so let's add an adaptor for v1 to use this:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "h705uhjykqto",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class FuncScheduler(fastai.train.LearnerCallback):\n",
        "    def __init__(self, func, learn, n_epoch):\n",
        "        super().__init__(learn)\n",
        "        self.learn, self.func, self.n_epoch = learn, func, n_epoch\n",
        "        \n",
        "    def on_train_begin(self, **kwargs):\n",
        "        self.step()\n",
        "\n",
        "    def on_batch_end(self, train, **kwargs):\n",
        "        if train: self.step()\n",
        "\n",
        "    def step(self):\n",
        "        if not hasattr(self, 'iter_vals'):\n",
        "            n_batch = len(self.learn.data.train_dl)*self.n_epoch \n",
        "            self.iter_vals = iter(self.func(x/n_batch) for x in range(0, n_batch+1))\n",
        "        self.learn.opt.set_stat('lr', next(self.iter_vals))\n",
        "\n",
        "def flat_then_cosine_sched(learn, n_epoch, lr, pct_start, div_final=1e5):\n",
        "    func = fastai2.callback.schedule.combined_cos(pct_start, lr, lr, lr/div_final)\n",
        "    return FuncScheduler(func, learn, n_epoch)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GPzOXiieJJnR",
        "colab_type": "text"
      },
      "source": [
        "Ok, ready to go. Here's our model from last time:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SeMsOlhbJIHL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model_fn = partial(xresnet18, c_out=10, sa=1, sym=0, act_cls=MishJit)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eWjmBztWJQtC",
        "colab_type": "text"
      },
      "source": [
        "a v1 learner:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "geLWPNtrSNA_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "learner_v1 = partial(\n",
        "    fastai.basic_train.Learner, \n",
        "    wd=1e-2, bn_wd=False, true_wd=True,\n",
        "    opt_func=partial(ranger.Ranger, betas=(0.95, 0.99), eps=1e-6),\n",
        "    metrics=(fastai.metrics.accuracy,),\n",
        "    loss_func=fastai.layers.LabelSmoothingCrossEntropy())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U8HGqIrvJVug",
        "colab_type": "text"
      },
      "source": [
        "and training code:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EypadbirnOSI",
        "colab_type": "code",
        "outputId": "1999c518-bc3d-40c1-ed58-3cc840025fbc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "torch.manual_seed(123)\n",
        "learn = learner_v1(data_v1(seed=123, dtype=torch.float), model_fn())\n",
        "n_epoch = 5\n",
        "learn.fit(n_epoch, callbacks=[flat_then_cosine_sched(learn, n_epoch, lr=4e-3, pct_start=0.72)])"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>1.477235</td>\n",
              "      <td>1.506388</td>\n",
              "      <td>0.601000</td>\n",
              "      <td>00:16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>1.224201</td>\n",
              "      <td>1.079142</td>\n",
              "      <td>0.765000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>1.111008</td>\n",
              "      <td>1.079000</td>\n",
              "      <td>0.772000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>1.032655</td>\n",
              "      <td>0.973072</td>\n",
              "      <td>0.812000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.928292</td>\n",
              "      <td>0.865962</td>\n",
              "      <td>0.868000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l1aRxFymJqq2",
        "colab_type": "text"
      },
      "source": [
        "Now for v2, here's the learner from last time:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "omJkSU1SwIPX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "RangerWrapper = lambda *args, **kwargs: fastai2.basics.OptimWrapper(ranger.Ranger(*args, **kwargs))\n",
        "\n",
        "learner_v2 = partial(\n",
        "    fastai2.basics.Learner, lr=4e-3,\n",
        "    opt_func=partial(RangerWrapper, betas=(0.95, 0.99), eps=1e-6),\n",
        "    metrics=(fastai2.metrics.accuracy,),\n",
        "    loss_func=fastai2.basics.LabelSmoothingCrossEntropy())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lSKwrfs_JwYD",
        "colab_type": "text"
      },
      "source": [
        "In order to handle weight decay in the same way as v1 (weight decay is not applied to BatchNorm params or biases) we create the optimiser manually outside of the call to learn.fit:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xCjOnwW5KjMR",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "c44ea489-e9ec-496d-8243-cba949ce1651"
      },
      "source": [
        "torch.manual_seed(123)\n",
        "\n",
        "learn = learner_v2(data_v2(dtype=torch.float, seed=123), model_fn())\n",
        "skip_wd = split_params((lambda mod, name: isinstance(mod, nn.BatchNorm2d) or name=='bias'), learn.model)\n",
        "learn.opt = learn.opt_func([{'params': skip_wd[False]}, {'params': skip_wd[True]}], lr=4e-3)\n",
        "\n",
        "learn.fit_flat_cos(5, lr=4e-3, wd=[1e-2, 0.0], pct_start=0.72)"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>1.477165</td>\n",
              "      <td>1.506689</td>\n",
              "      <td>0.599000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>1.224083</td>\n",
              "      <td>1.079232</td>\n",
              "      <td>0.768000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>1.111179</td>\n",
              "      <td>1.076746</td>\n",
              "      <td>0.774000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>1.032706</td>\n",
              "      <td>0.971359</td>\n",
              "      <td>0.813000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.928698</td>\n",
              "      <td>0.865830</td>\n",
              "      <td>0.869000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iueeZmrULtxl",
        "colab_type": "text"
      },
      "source": [
        "Not bad! We're getting agreement in the first 2-3 dps of train_loss with the v1 code above. The main remaining difference is the code used to apply weight decay inside the Ranger optimiser. We can get (hack) around this as by applying weight decay ourselves outside the optimiser:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gcDfQaiv-5cu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class WeightDecay(fastai2.learner.Callback):\n",
        "    def __init__(self, params, wd):\n",
        "        stepper = lambda p, wd: fastai2.basics.weight_decay(p, self.opt.hypers[0]['lr'], wd)\n",
        "        self.step = fastai2.basics.Optimizer(params, [stepper], wd=wd).step\n",
        "        \n",
        "    def after_backward(self):\n",
        "        self.step()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-iMJM4U7-nfo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "6f0c8bc2-5947-49c1-e5f5-087c818aa641"
      },
      "source": [
        "torch.manual_seed(123)\n",
        "\n",
        "learn = learner_v2(data_v2(dtype=torch.float, seed=123), model_fn())\n",
        "skip_wd = split_params((lambda mod, name: isinstance(mod, nn.BatchNorm2d) or name=='bias'), learn.model)\n",
        "learn.fit_flat_cos(5, lr=4e-3, wd=0.0, pct_start=0.72, cbs=[WeightDecay(skip_wd[False], 1e-2)])"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>1.477233</td>\n",
              "      <td>1.506388</td>\n",
              "      <td>0.601000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>1.224201</td>\n",
              "      <td>1.079142</td>\n",
              "      <td>0.765000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>1.111007</td>\n",
              "      <td>1.079000</td>\n",
              "      <td>0.772000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>1.032653</td>\n",
              "      <td>0.973072</td>\n",
              "      <td>0.812000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.928291</td>\n",
              "      <td>0.865962</td>\n",
              "      <td>0.868000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SBNfSkVZNZpp",
        "colab_type": "text"
      },
      "source": [
        "These agree with the v1 results to 5 dps."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FWTxl-z9Ny62",
        "colab_type": "text"
      },
      "source": [
        "### Manual training loop"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tyE17_v4N5TB",
        "colab_type": "text"
      },
      "source": [
        "In case it's useful, here's a manual training loop based somewhat loosely on the v1 codebase which also ties out to 5dp. This may be helpful for future comparison/debugging:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1JhqL1n3gKjh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "update = lambda metric, output, target: metric.on_batch_end(output, target)\n",
        "\n",
        "format_str = lambda name, stat: (\n",
        "    fastai.basic_train.format_time(stat) if 'time' in name \n",
        "    else '#na#' if stat is None \n",
        "    else str(stat) if isinstance(stat, int) \n",
        "    else f'{stat:.6f}'\n",
        ")\n",
        "       \n",
        "def fit(learn, n_epoch, lr_sched):\n",
        "    learn.create_opt(0, learn.wd)\n",
        "    pbar = fastai.basics.master_bar(range(n_epoch))\n",
        "    smoothener = fastai.basics.SmoothenValue(0.98)\n",
        "    def valid_loss(output, target): return learn.loss_func(output, target)\n",
        "    metrics = [fastai.basics.AverageMetric(m) for m in ([valid_loss] + learn.metrics)]\n",
        "    names = ['epoch', 'train_loss'] + [m.name for m in metrics] + ['time']\n",
        "    pbar.write(names, table=True)\n",
        "\n",
        "    timer, logs = Timer(), []\n",
        "    for epoch in pbar:\n",
        "        learn.model.train()\n",
        "        num_batch = 0 \n",
        "        for met in metrics: met.val, met.count = 0., 0 \n",
        "        for xb,yb in fastai.basics.progress_bar(learn.data.train_dl, parent=pbar):\n",
        "            lr_sched.step()\n",
        "            output = learn.model(xb)\n",
        "            loss = learn.loss_func(output, yb)\n",
        "            smoothener.add_value(loss.item())\n",
        "            pbar.child.comment = f'{smoothener.smooth:.4f}'\n",
        "            logs.append({'lr': learn.opt.lr, 'mom': learn.opt.mom, 'loss': smoothener.smooth})\n",
        "            loss.backward()\n",
        "            learn.opt.step()\n",
        "            learn.opt.zero_grad()\n",
        "            num_batch += 1\n",
        "\n",
        "        learn.model.eval()\n",
        "        with torch.no_grad():\n",
        "            for xb, yb in fastai.basics.progress_bar(learn.data.valid_dl, parent=pbar):\n",
        "                output = learn.model(xb)\n",
        "                for met in metrics: update(met, output, yb)\n",
        "\n",
        "        summary = union(\n",
        "            {'epoch': epoch, 'num_batch': num_batch, 'train_loss': smoothener.smooth,  'time': timer()},\n",
        "            {m.name: (m.val/m.count).item() for m in metrics}\n",
        "        )\n",
        "        logs.append(summary)\n",
        "        pbar.write([format_str(k, summary[k]) for k in names], table=True)\n",
        "\n",
        "    return group_by_key(((k,v) for entry in logs for (k,v) in entry.items()), np.array)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9HfQqkTbD6XY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "fae39432-c4bf-4a33-9644-67335caa3201"
      },
      "source": [
        "torch.manual_seed(123)\n",
        "learn = learner_v1(data_v1(seed=123, dtype=torch.float), model_fn())\n",
        "n_epoch = 5\n",
        "logs = fit(learn, n_epoch, flat_then_cosine_sched(learn, n_epoch, lr=4e-3, pct_start=0.72))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: left;\">\n",
              "      <th>epoch</th>\n",
              "      <th>train_loss</th>\n",
              "      <th>valid_loss</th>\n",
              "      <th>accuracy</th>\n",
              "      <th>time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>0</td>\n",
              "      <td>1.477233</td>\n",
              "      <td>1.506388</td>\n",
              "      <td>0.601000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>1</td>\n",
              "      <td>1.224201</td>\n",
              "      <td>1.079142</td>\n",
              "      <td>0.765000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>2</td>\n",
              "      <td>1.111007</td>\n",
              "      <td>1.079000</td>\n",
              "      <td>0.772000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>3</td>\n",
              "      <td>1.032654</td>\n",
              "      <td>0.973072</td>\n",
              "      <td>0.812000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>4</td>\n",
              "      <td>0.928292</td>\n",
              "      <td>0.865962</td>\n",
              "      <td>0.868000</td>\n",
              "      <td>00:15</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oNTzJ5Z7okfU",
        "colab_type": "code",
        "outputId": "fd482cd2-c10d-4d48-b479-ca4f1527e22e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 173
        }
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "fig, axs = plt.subplots(1, 2, figsize=(8, 2))\n",
        "for ax, k in zip(axs, ['lr', 'loss']):\n",
        "    ax.plot(logs[k])\n",
        "    ax.set_title(k)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAfIAAACcCAYAAACEGClOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXwV9bn48c+TPSQkIQshGwQIW1jC\nJosrFVRQkF+tVLRa2ur1UrXaam8v3Fbbcuut3t6qteKudS2IOyqKIqi4sCtIWBMWCfuaEMjO8/vj\nDDTGhBzg5Mw5Oc/79Tov5nznOzPPDJk8mZnvfL+iqhhjjDEmOIW5HYAxxhhjTp8lcmOMMSaIWSI3\nxhhjgpglcmOMMSaIWSI3xhhjgpglcmOMMSaIWSI3p0xEtojIKLfjMMY0z87X1s8SuTHGGBPELJEb\nnxGRCLdjMMaYUGOJ3Jw2EfmDiLwiIi+ISBnwE7djMsY0TkSiReQBEdnhfB4QkWhnXqqIvC0ih0Tk\ngIgsFJEwZ95/ish2ETksIutFZKS7e2Iasisoc6bGAxOAHwPRLsdijGnab4FhQH9AgTeB3wF3AncA\nJUCaU3cYoCLSA7gFOEtVd4hILhDu37BNc+yK3JypL1T1DVU9pqoVbgdjjGnSj4BpqrpHVfcCfwSu\nc+bVABlAJ1WtUdWF6hmIow7PH+j5IhKpqltUtdiV6E2TLJGbM7XN7QCMMV7JBLbW+77VKQP4C1AE\nvC8im0RkCoCqFgG/BP4A7BGRmSKSiQkolsjNmbLh84wJDjuATvW+d3TKUNXDqnqHqnYBLgduP/4s\nXFX/qarnOssqcK9/wzbNsURujDGhYQbwOxFJE5FU4C7gBQARGSsieSIiQCmeW+rHRKSHiFzoNIqr\nBCqAYy7Fb5pgidwYY0LDn4BlwCrga2CFUwbQDZgHlANfAA+r6gI8z8fvAfYBu4D2wFT/hm2aI572\nDMYYY4wJRnZFbowxxgQxS+TGGGNMELNEbowxxgQxS+TGGGNMELNEbowxxgSxoOprPTU1VXNzc90O\nw5iAtnz58n2qmtZ8TffYuWyMd7w5n71K5CIyGvgbns7yn1TVexrMjwaeAwYB+4GrVHWLM28qcD2e\nDgZuVdW59ZYLx/Ne43ZVHdtcHLm5uSxbtsybkI0JWSKytfla7rJz2RjveHM+N3tr3Um204ExQD5w\ntYjkN6h2PXBQVfOA+3G68HPqTQR6A6OBh531HXcbsLb5XTHGuEFEckRkgYisEZFCEbmtkTojRKRU\nRL5yPne5EasxocqbZ+RDgCJV3aSq1cBMPENX1jceeNaZfgUY6XT1Nx6YqapVqroZT6f8QwBEJBu4\nDHjyzHfDGNNCaoE7VDUfz9CWNzfyhzzAQlXt73ym+TdEY0KbN7fWs/j2CFclwNCm6qhqrYiUAilO\n+aIGy2Y50w8AvwHannrYjbv+maVU1X63G2AR79chTVRuahWNVW+67nfnnMp6m1qiqbqnFnMT+91Y\n3Sa3530gLXmMGhaLCD07tGVsQSZZSbFNrM00RlV3Ajud6cMishbPObymJbf71sodvLBoKzP+bRhh\nYadwAhsTglxp7CYiY4E9qrpcREY0U/dG4EaAjh07nnS9lbV1VNZ8O5E31QVtY6VN9VbbZCe2jSzQ\nVN3G1q1N1G4yjkbX0VRd77vebXq/G9m/UzhGp3Lsm5pxqvvXWGlN7TFe/3I7f5m7nmuHdeI/R/ck\nNiq8kZrmZEQkFxgALG5k9nARWYlnNK1fq2rhmWxLgcWbD/DXD9bzH5f0PJNVGdPqeZPItwM59b5n\nO2WN1SkRkQggEU+jt6aWvRy4XEQuBWKABBF5QVWvbbhxVX0ceBxg8ODBJ81OL94wzIvdMaFo24Gj\nPPJxMc98voXPivbxzM+G2NX5KRCReOBV4JeqWtZg9gqgk6qWO+f0G3gG4Wi4Dq//KB/Zsz0A0xcU\nWyI3phnePCNfCnQTkc4iEoWn8drsBnVmA5Oc6SuB+eq5ZJoNTBSRaBHpjOfkXqKqU1U1W1VznfXN\nbyyJG+MrOclt+J/v9+WF64eyq6ySKx/5nK37j7gdVlAQkUg8SfxFVX2t4XxVLVPVcmd6DhDpDJPZ\nsN7jqjpYVQenpZ387bi46AiGd0kBoLbORs005mSaTeSqWgvcAszF08J8lqoWisg0EbncqfYUkCIi\nRcDtwBRn2UJgFp7nae8BN6tqne93wxjvnNstlZduHE5FTR0//cdSDh2tdjukgOY0Wn0KWKuq9zVR\np4NTDxEZguf3yv4z3fb3B3ia05QcrDjTVRnTqnn1jNz5K3tOg7K76k1XAhOaWPZu4O6TrPsj4CNv\n4jDGF/IzE3j8usFc++Ribv7nCp7/2VBrUNW0c4DrgK9F5Cun7L+AjgCq+iieu3A/F5FaoAKYqD4Y\nH7lr+zgANu0rJzc17kxXZ0yrFVQ9uxnjK0M6JzNtfG+mvPY1TyzcxL9f0NXtkAKSqn5K0y8OHK/z\nEPCQr7fdJTUegOI9R7jQHpMb0yTra92ErKvOymF07w783/vrWberYfst47Z2cVG0jY5g+yG7tW7M\nyVgiNyFLRPjzFX2Jj47gd6+v5tixM74bbHysfUI0u8sq3Q7DmIBmidyEtHZxUUwd04tlWw/y6ooS\nt8MxDaQnxFgiN6YZlshNyLtyUDaDOrXj3vfWcaSq1u1wTD2eRF7ldhjGBDRL5CbkhYUJv72sF/vK\nq/nHZ5vdDsfUk54Qw57DlafUU6ExocYSuTHAwI7tGNUrncc+2WTvlgeQTiltqKlTNu4pdzsUYwKW\nJXJjHL++pDvlVbU8+vEmt0MxjrNykwFYvb3U5UiMCVyWyI1x9OyQwGV9M3hh0VZKK2rcDsfguSKP\nCBOK7IrcmCZZIjemnskXdKW8qpYXF291OxQDRIaHkZsax4bdlsiNaYolcmPq6ZOVyHndUnn60y1U\n1tiwAIGgR4e2rN9tHfYY0xRL5MY0MPmCruwrr+L1LxuO1mvc0DO9LdsOVFgjRGOaYIncmAbO7ppC\n36xEnvp0s732FADO6uxp8Pbe6l0uR2JMYLJEbkwDIsKPh3eiaE85X2w649E4zRka2jmZ1Pgolmw+\n4HYoxgQkS+TGNGJcQSZJbSJ5YZE1enObiNA3K5HPi/dbf/jGNMISuTGNiIkM54eDc5hbuNv6+g4A\n5+SlsquskjU7rdGbMQ15lchFZLSIrBeRIhGZ0sj8aBF5yZm/WERy682b6pSvF5FLnLIYEVkiIitF\npFBE/uirHTLGV340tCPHVPnn4m/cDiXkjSvIBOCzon0uR2JM4Gk2kYtIODAdGAPkA1eLSH6DatcD\nB1U1D7gfuNdZNh+YCPQGRgMPO+urAi5U1QKgPzBaRIb5ZpeM8Y1OKXFc0D2NGUu+obbumNvhhLT0\nhBh6ZybwwuKtVFTba4HG1OfNFfkQoEhVN6lqNTATGN+gznjgWWf6FWCkiIhTPlNVq1R1M1AEDFGP\n4z08RDofe/hlAs7Eszqy53AVCzfalaDbfndZPtsOVPDEQutC15j6vEnkWcC2et9LnLJG66hqLVAK\npJxsWREJF5GvgD3AB6q6+HR2wJiWdGHP9iTHRfHy8m3NVzYtanjXFAZ0TGL+uj1uh2JMQHGtsZuq\n1qlqfyAbGCIifRqrJyI3isgyEVm2d+9e/wZpQl5URBjj+2cyb80eDh6xDkncdm5eKqtKDlF61PrC\nN+Y4bxL5diCn3vdsp6zROiISASQC+71ZVlUPAQvwPEP/DlV9XFUHq+rgtLQ0L8I1xrcmDMqhuu4Y\nb35lPb257ZLeHTim8OwXW9wOxZiA4U0iXwp0E5HOIhKFp/Ha7AZ1ZgOTnOkrgfnq6RJrNjDRadXe\nGegGLBGRNBFJAhCRWOAiYN2Z744xvpefmUDvzAReXl7idighr3dmAsO7pDBjyTf2TrkxjmYTufPM\n+xZgLrAWmKWqhSIyTUQud6o9BaSISBFwOzDFWbYQmAWsAd4DblbVOiADWCAiq/D8ofCBqr7t210z\nxncmDMqmcEcZa3aE1nvMIpIjIgtEZI3zquhtjdQREXnQec10lYgMbMF4uOqsHHaWVlqve8Y4Iryp\npKpzgDkNyu6qN10JTGhi2buBuxuUrQIGnGqwxrhlfP8s7p6zlldXlJCf2fDty1atFrhDVVeISFtg\nuYh8oKpr6tUZg+duWzdgKPCI82+LuCg/naykWH4x40vm33EBSW2iWmpTxgQF69nNGC+0i4tiRI/2\nvL1qB3UhdEtXVXeq6gpn+jCeu3IN31oZDzznvFa6CEgSkYyWiikuOoKHfzSQA0eqeefrnS21GWOC\nhiVyY7w0vn8mu8uqWLw5NG/pOj02DgAavirqzSuqPtUvO5EuaXE2IpoxWCI3xmsje6YTFxXO7K92\nuB2K34lIPPAq8EtVPa2GAr58lVREOL9bGku3HKCyxnp6M6HNErkxXoqNCufi3h2Y8/VOqmpDJ3mI\nSCSeJP6iqr7WSBVvXlH1+aukF/dOp7LmGHPs9roJcZbIjTkFl/fPpKyylk82hEaXrU5Xy08Ba1X1\nviaqzQZ+7LReHwaUqmqLZ9fhXVLITWnDrGXW654JbZbIjTkF5+alkhwXFUqdw5wDXAdcKCJfOZ9L\nRWSyiEx26swBNuEZS+EJ4CZ/BCYiTBicw6JNB9i6/4g/NmlMQPLq9TNjjEdkeBiX9c3g5eXbKK+q\nJT66dZ9CqvopIM3UUeBm/0T0bVcOyuaBeRt46tPNTBvfaC/PxrR6dkVuzCka3z+TyppjfLDGWky7\nLT0hhh8MzGbm0m3sOFThdjjGuMISuTGnaGDHdmQlxfLGl6HXej0Q3XBeF8IEbnh2mY0bb0KSJXJj\nTlFYmDCuIJPPivZxwEZEc11e+3j+OqE/a3aW8cKirW6HY4zfWSI35jSMK8ig9phahyQB4tK+HTiv\nWyp/eGsN7xfa/4kJLZbIjTkN+RkJdEmL462Vdns9EIgI9/6gHwC/n11ot9hNSLFEbsxpEBHG9ctk\n0eb97CmrdDscA2QmxTL9moHsLK3krVX2B5YJHZbIjTlN4woyUMUG7gggY/p0oEd6W+56o9DulpiQ\nYYncmNOU174tvTISLGEEkLAwYfqPBnK4qpZfzPiS/3r9azyvuRvTelkiN+YMjCvIYMU3h9h24Kjb\noRhHXvt4Xr/pbAD+ufgbfv3yKpcjMqZleZXIRWS0iKwXkSIRmdLI/GgRecmZv9gZ7vD4vKlO+XoR\nucQpyxGRBSKyRkQKReQ2X+2QMf40rl8mYLfXA82Aju1YO200Azom8eqKEr4oDs2hZ01oaDaRi0g4\nMB0YA+QDV4tIfoNq1wMHVTUPuB+411k2H5gI9AZGAw8766sF7lDVfGAYcHMj6zQm4OUkt6F/TpLd\nXg9AsVHhPH/9ULKSYrnzzdXUWEt200p5c0U+BChS1U2qWg3MBMY3qDMeeNaZfgUY6YyaNB6YqapV\nqroZz6AKQ1R1p6quAFDVw8BaIOvMd8cY/xvbL4PCHWUU7y13OxTTQHx0BH+8vDdFe8r527yNbodj\nTIvwJpFnAfXHCSzhu0n3RB1VrQVKgRRvlnVuww8AFje2cRG5UUSWiciyvXv3ehGuMf41tl8mIvD2\nSru9HohG9mrPmD4deGhBEc9/scXtcIzxOVcbu4lIPPAq8EtVLWusjqo+rqqDVXVwWlqafwM0xgsd\nEmM4KzeZ2Su3WwvpACQi3PODfiS1ieSed9exv7zK7ZCM8SlvEvl2IKfe92ynrNE6IhIBJAL7T7as\niETiSeIvquprpxO8MYFiXEEmxXuPsG7XYbdDMY1IjI3klcnDqaip46EFRW6HY4xPeZPIlwLdRKSz\niEThabw2u0Gd2cAkZ/pKYL4zRvFsYKLTqr0z0A1Y4jw/fwpYq6r3+WJHjHHTmD4dCA8Ta/QWwPLa\nt2XCoBz+8dkWnly4ye1wjPGZZhO588z7FmAunkZps1S1UESmicjlTrWngBQRKQJuB6Y4yxYCs4A1\nwHvAzapaB5wDXAdcKCJfOZ9LfbxvxvhNanw0Z3dN4a1VO+z2egC7a1w+F3RP40/vrOXyhz61/yvT\nKnj1jFxV56hqd1Xtqqp3O2V3qepsZ7pSVSeoap6qDlHVTfWWvdtZroeqvuuUfaqqoqr9VLW/85nT\nEjtojL+MK8hk24EKVpaUuh2KaUJcdASPXDsQgFUlpTbsqWkVrGc3Y3zkkt4diAy32+uBrk1UBGun\njaYgO5G7Zluf7Cb4WSI3xkcSYyO5oHt73l61g2PH7JZtIIuNCuc5p7OYX8z4khufW0ZFdZ3bYRlz\nWiyRG+ND4woy2F1WxdItB9wOxTQjMTaS9391Phd0T+P9Nbu54+Wv7A8wE5QskRvjQ6N6pRMTGWbj\nYQeJNlERPDVpMOd1S2XO17sY/bdP7MrcBB1L5Mb4UFx0BCN7pTPn613UtoK+vUXkaRHZIyKrm5g/\nQkRK6719cpe/YzxTEeFhPPezIfz7+V3YsLuc22Z+aVfmJqhYIjfGx8b1y+TAkWo+bx0jbj2DZ8Cj\nk1lY7+2TaX6IyedEhKmX9uLOsfm8v2Y3P3lmKau329sHJjhYIjfGx0b0SCM+OqJVtIZW1U+AkHng\n/7NzcvnZOZ1ZvGk/k55ewo5DFW6HZEyzLJEb42MxkeFc3Dud9wp3UVUbEs9bh4vIShF5V0R6ux3M\nmRAR7hqXzzu3nkdV7TF++NgXNqqdCXiWyI1pAeMKMjlcWcsnG/a5HUpLWwF0UtUC4O/AG01VDKaR\nDPPax/P3awawv7yakX/9mB8++gUPfriRPYcr3Q7NmO+wRG5MCzg3L5WkNpGt4vb6yahqmaqWO9Nz\ngEgRSW2iblCNZPi9Hu15+9ZzOSu3HUu2HOC+DzYw5O4PufON1ewuq6TOGsSZABHhdgDGtEaR4WGM\n6ZPBm19tp6K6jtiocLdDahEi0gHYraoqIkPwXBy0ilZ+AF3T4nl58tkU7Snn9S9LmL6gmOcXbeV5\np2vXP1/RlwmDsokIt2si4x776TOmhYwryOBodR3z1+1xO5TTJiIzgC+AHiJSIiLXi8hkEZnsVLkS\nWC0iK4EHgYnaCkciyWsfz39c0pPNf76U+68qID0hGoCpr31Nnz/M5donF1N6tMblKE2osityY1rI\n0M4ppLWN5q2VO7isX4bb4ZwWVb26mfkPAQ/5KRzXiQjfH5DN9wdkU1Vbx7w1e7j3vXV8WrSPgmnv\n85Ozc/n9uHw8IzUb4x92RW5MCwkPEy7rm8H89Xs4XGlXa61NdEQ4l/XLYN7tF3DbyG5ERYTxzOdb\nGHXfx7z51XZ7hm78xhK5MS1oXEEm1bXH+GDNbrdDMS0kKiKMX13UnfX/PZrfXdaL4r1HuG3mV4z9\n+6f2B5zxC68SuYiMFpH1IlIkIlMamR8tIi858xeLSG69eVOd8vUickm98pN2/WhMazCwYxJZSbGt\nvvW68dx2v+G8Lrxz67lcnJ/O2p1l9J/2AQ/M29Aquus1gavZRC4i4cB0YAyQD1wtIvkNql0PHFTV\nPOB+4F5n2XxgItAbTzePDzvrA++6fjQmqIkIYwsyWLhxHwePVLsdjvGD3pmJPP7jwdxzRV86JMTw\nwLyN5P32XXKnvMOkp5dwpKrW7RBNK+PNFfkQoEhVN6lqNTATGN+gznjgWWf6FWCkeFp7jAdmqmqV\nqm4Gipz1hVzXjyZ0jeuXSe0x5d3Vu9wOxfjRxCEd+WzKhfz96gEnyj7esJfev5/Lr19eyfQFRfYc\n3fiEN63Ws4Bt9b6XAEObqqOqtSJSCqQ45YsaLJt12tEaE4R6ZyaQ1z6el5dv45qhHd0Ox/jZuIJM\nxhVkAvDmV9v50ztrmfP1To5W1/GXueuZfEFXbjy/CxHhQnxUBGFh1uLdnJqAf/1MRG4EbgTo2NF+\nCZrgIyJMPCuHP72zlnW7yujZIcHtkIxLxvfPYnz/LFSVv324kQc/3MijHxfz6MfFAKTERTGqVzq3\njupGVlKsy9GaYOHNrfXtQE6979lOWaN1RCQCSMTTu5M3y55UsHXraExjrhiYTVR4GDOXbGu+smn1\nRIRfjurO2v8ezWs3nc3YfhkMyU1GRHhp2TbOuWc+1z21mH3lVW6HaoKAN4l8KdBNRDqLSBSexmuz\nG9SZDUxypq8E5ju9O80GJjqt2jsD3YAlvgndmOCRHBfF6D4deG1FCRXVITEimvFCdEQ4Azu246Fr\nBjJr8nCW/W4UT/54MJmJMSzcuI/Bf5rHnW+sprrWWr2bpjWbyFW1FrgFmAusBWapaqGITBORy51q\nTwEpIlIE3A5McZYtBGYBa4D3gJtVtQ4a7/rRt7tmTGC5ekhHyiprmfP1TrdDMQFsVH46n08dycuT\nhzO4UzueX7SVcX//1N56ME2SYOoWefDgwbps2TK3wzDmtKgqF/71Y1Ljo3h58tktth0RWa6qg1ts\nAz5g57L3Zi3bxm9eWQVAn6wEJgzK4dphnQi3RnEhwZvzOeAbuxnTWogIVw/J4X/mrGPtzjJ6ZVij\nN9O8Hw7OIToijP99bz2rt5exensh095eQ1ZSLMO6JHPdsFx2llbw/prdFGQnkhAbyZg+GURFWMed\nocKuyI3xo0NHqxn+5/mM7ZfBXyYUtMg27Iq89dpxqILXv9zO8q0Hqak7xsKN+5qsO6ZPB5LaRHHo\naDVHquvol5XIed1SGdolxY8RmzNlV+TGBJikNlFcOSibl5Zu4zeje5LWNtrtkEwQyUyK5ebv5Z34\n/uU3B3l+0VY6p8Txw7NyOHS0hme/2MK8NbtPdEDUNjqCw1W1fLJhLw8tKGLS8E7cNqo7yXFRLu2F\n8TW7IjfGzzbtLWfkfR/ziwu7cftF3X2+frsiN3XHlNKKGuKjI4iKCENVKd5bzt3vrGXB+r0ATBnT\nkxvO7cwnG/eSGBtFTd0xhnZOtiFYA4xdkRsTgLqkxTOyZzovLNrKTSO6EhMZ3vxCxpyC8DD51hW3\niJDXvi3/+OkQvvzmIP/23HLueXcd9763jvrXcm1jIuie3paC7CQyk2LYe7iKUfnpDO7UzhJ8ALNE\nbowLbjivMxMf383Ly7Zx3fBct8MxIWRAx3Ys/q+RPPZJMUs2H6C8spaeGW3JSIxl9lc72H6wgsId\npVTWeN5df+yTTQAkxERwdtdUhnVJpnt6W7p3aMsnG/ZybrdU2reNcXOXQp4lcmNcMLRzMoM7tWP6\ngmImDM6xq3LjV+Fhwk0j8rhpxLfLjz9/L6usYePucqLCw/i8eB+rSkpRlPcLd/Ne4XcH/7luWCcm\nj+jaZLey5VW1vLVyB2fltuNwZS25KXG0s2f0PmOJ3BgXiAi/uqg7P3pyMS8t3caks3PdDsmYExJi\nIhnUqR0AfbMTT5RvP1TB9oMVLNt6gN2llVTXKYeOVvP8oq08v2gr6QnR/L8BWajCM59tIa99PCJQ\nuKPsW+sPEzi3WxqTz+/C2XmpVNXWUVOnHCivJic5lu2HKkiNj7Y/cL1kidwYl5zdNYUhuck8/FER\nV51lV+Um8GUlxZKVFMuQzsnfKi/eW878tXt48tNNPPbxphPla3b+K4H/YGA2yXGRJLWJYl95Fa9/\nuZ1rnlzc5LbCBP5weW8uL8gkqY1dvZ+MtVo3xkVfFO/n6icWMWVMTyZf0NUn67RW68Yt1bXHWFly\niOI95Yzu04GEmEiq644RHiZEhn+7g5q9h6t4/JNi1u48TFKbSMJE2LL/CDER4dSpsnzrQQCiIsI4\nLy+VlPgoctq14YpB2cxft4dNe8vplZHAZX0ziItuvdek3pzPlsiNcdkNzy7li+L9zP/1CNITzrzR\nkCVy0xpsO3CUXWWVPP/FVlbvKGXT3iNN1s1KimXOreeR2CbSjxH6h71+ZkwQuHNsPhfd9wn3vLuO\n+6/q73Y43yIiTwNjgT2q2qeR+QL8DbgUOAr8RFVX+DdK0xrlJLchJ7kNZ+V6buOXV9WyZd8RFm8+\nQHREGClxUaS1jeb3swsp3FFGwbT3+eHgbG44rwvd2sef9HW5ypo6yiprWk1re0vkxrisU0oc/3Z+\nZ6YvKObKQdmck5fqdkj1PQM8BDzXxPwxeIYn7gYMBR5x/jXGp+KjI+iTlUifrMRvlb9z63l8XryP\na55YzKxlJcxaVnKifkxkGJlJsfTJSqTkYAUlB48SHx1BycEKSitqOCcvlXO6pjCiR3sOHKmmpu4Y\n53VLDbp35u3WujEBoKK6jsv+vpCjVXXM/eX5Z3SL0Ne31kUkF3i7iSvyx4CPVHWG8309MEJVTzpW\nq53LxtdUlR2llSxYt4cZS74hLiqChNhIPi/ex9HqOkQgMiyMLmlxZCTGcOBoDSu3HfrOemIiw7hu\nWCeGdk4hN7UNHZPj2LSvnMqaYxRkJ/o9ydutdWOCRGxUOA9c1Z8rHv6cqa+vYvo1A4PlqiAL2Fbv\ne4lTZoOuG78SEbKSYrl2WCeuHdbpRLmqsu1ABRlJMd9pcFdde4yt+4/w/KKtlFXUkNc+nrdX7eSJ\nhZt5YuHm72yjd2YCI3qkERMRzpi+HeiSGk9YAAwna4ncmADRLzuJX1/Sg3veXceDHxZx26hubofk\nUyJyI3AjQMeOHV2OxoQKEaFjSptG50VFhNEtvS3Txv/rZtPN38tj6/6jFO8tZ+W2Q8xYuo1RvdrT\nvm0Mj3xUfOKd+L9+sIG89vH8dUIBBTlJftmXpniVyEVkNJ4GLeHAk6p6T4P50XieoQ0C9gNXqeoW\nZ95U4HqgDrhVVed6s05jQtG/n9+FjbvLuX/eBpLjIoOh+9btQE6979lO2Xeo6uPA4+C5td7yoRlz\n6kSE3NQ4clPjGNkrndsv7nFi3qSzc9my/whrd5ZxtKqOe99bx/cf/ozBnZIZ0CmJ0qM1dE6NIz8z\ngaTYKPpkJfjlzlqziVxEwoHpwEV4bpstFZHZqrqmXrXrgYOqmiciE4F7gatEJB+YCPQGMoF5InJ8\nuKfm1mlMyBER/nxFX0orqrnzzUJ2llbyq4u6f+eWYACZDdwiIjPxNHIrbe75uDHBKjkuiuS4KAZ2\n9PR6Nyi3Hc98toV5a3ezZMuB79RPjI1kWJdkpo3v45NXS5vizRX5EKBIVTcBOCfseKB+0h0P/MGZ\nfgV4yHktZTwwU1WrgM0iUsYLOekAAAbUSURBVOSsDy/WaUxIiooI45FrB3HnG6t5+KNiPlq/l19c\nmMf3erb3e+9vIjIDGAGkikgJ8HsgEkBVHwXm4Hn1rAjP62c/9WuAxrhoYMd2DOzYjsqaOnaXVZIQ\nE8mR6lo+WLObHYcqWLfrMHMLdzO3cDc/H9GV8f0zSY6L8vlrb94k8sYaszR8veREHVWtFZFSIMUp\nX9Rg2Sxnurl1GhOyIsPDuOcH/Ti/exr3vLuOn7+4gogwIT0hhg/vuMBvCV1Vr25mvgI3+yUYYwJU\nTGQ4nVLiAGgXF8VPz+l8Yt6DH27kvg828MhHxTzyUTEAcVHhhIcJPx+Rx89HnHmPjgHf2M0ayJhQ\ndmnfDC7OT2dh0T6Wbj7A3sNV1ie7MUHk1pHd+MWFeazdeZglm/dTVXuMnaWVqCo9OsT7ZBveJHJv\nGrMcr1MiIhFAIp5Gbydb1hrIGOOFiPAwvtejPd/r0d7tUIwxp0FEyM9MID8zoUXW700LmqVANxHp\nLCJReBqvzW5QZzYwyZm+Epjv3HKbDUwUkWgR6Yyn96clXq7TGGOMMc1o9orceeZ9CzAXz6tiT6tq\noYhMA5ap6mzgKeB5pzHbATyJGafeLDyN2GqBm1W1DqCxdfp+94wxxpjWzatn5Ko6B0/r1Ppld9Wb\nrgQmNLHs3cDd3qzTGGOMMacmqPpaF5G9wNZmqqUC+/wQzqkItJgCLR6wmLzlTUydVDXNH8GcLjuX\nfcpial6gxQPex9Ts+RxUidwbIrIs0MZiDrSYAi0esJi8FYgxtZRA3FeLyTuBFlOgxQO+jSlgu4sy\nxhhjTPMskRtjjDFBrDUm8sfdDqARgRZToMUDFpO3AjGmlhKI+2oxeSfQYgq0eMCHMbW6Z+TGGGNM\nKGmNV+TGGGNMyGg1iVxERovIehEpEpEpftxujogsEJE1IlIoIrc55cki8oGIbHT+beeUi4g86MS5\nSkQGtlBc4SLypYi87XzvLCKLne2+5PSoh9Pr3ktO+WIRyW2heJJE5BURWScia0VkeAAco185/2er\nRWSGiMT4+ziJyNMiskdEVtcrO+XjIiKTnPobRWRSY9sKJm6cz4F6LjvbsvO5+ZhC93xW1aD/4Okd\nrhjoAkQBK4F8P207AxjoTLcFNgD5wP8CU5zyKcC9zvSlwLuAAMOAxS0U1+3AP4G3ne+zgInO9KPA\nz53pm4BHnemJwEstFM+zwA3OdBSQ5OYxwjMK32Ygtt7x+Ym/jxNwPjAQWF2v7JSOC5AMbHL+bedM\nt/PHz38L/ay4cj4H6rnsbMvO55PHE9Lnc4ueGP76AMOBufW+TwWmuhTLm8BFwHogwynLANY7048B\nV9erf6KeD2PIBj4ELgTedn5Q9gERDY8Xnm5yhzvTEU498XE8ic5JJg3K3TxGx4feTXb2+23gEjeO\nE5Db4MQ/peMCXA08Vq/8W/WC7RMo53MgnMvOeu18bj6mkD6fW8ut9cbGTM9qom6LcW7PDAAWA+mq\nutOZtQtId6b9EesDwG+AY873FOCQqtY2ss1vjSUPHB9L3pc6A3uBfzi3B58UkThcPEaquh34P+Ab\nYCee/V6Ou8fpuFM9LgHx8+9Dru9PAJ3LYOdzs0L9fG4tidx1IhIPvAr8UlXL6s9Tz59Vfnk9QETG\nAntUdbk/tuelCDy3mx5R1QHAETy3mE7w5zECcJ5TjcfzSykTiANG+2v73vL3cTGBcy47sdj57IVQ\nP59bSyL3Zsz0FiMikXhO/BdV9TWneLeIZDjzM4A9for1HOByEdkCzMRzO+5vQJJ4xopvuM0T8ci3\nx5L3pRKgRFUXO99fwfOLwK1jBDAK2Kyqe1W1BngNz7Fz8zgdd6rHxdWf/xbg2v4E2LkMdj57K6TP\n59aSyF0b31xEBM8wrmtV9b56s+qP0T4Jz/O24+U/dlosDgNK6912OWOqOlVVs1U1F89xmK+qPwIW\n4BkrvrF4GhtL3mdUdRewTUR6OEUj8Qxt68oxcnwDDBORNs7/4fGYXDtO9ZzqcZkLXCwi7Zwrk4ud\nsmDlyvkcaOcy2Pl8CkL7fPZlgwM3P3haAG7A09r1t37c7rl4bpWsAr5yPpfied7yIbARmAckO/UF\nmO7E+TUwuAVjG8G/Wrl2AZYARcDLQLRTHuN8L3Lmd2mhWPoDy5zj9Aae1piuHiPgj8A6YDXwPBDt\n7+MEzMDzTK8Gz5XO9adzXICfObEVAT/1189/C/7s+v18DuRz2dmenc8njylkz2fr2c0YY4wJYq3l\n1roxxhgTkiyRG2OMMUHMErkxxhgTxCyRG2OMMUHMErkxxhgTxCyRG2OMMUHMErkxxhgTxCyRG2OM\nMUHs/wP2oIoqdYixlQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 576x144 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X3cDWmLtSvGB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}