7.CustomModelHPO.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tuning your own models \n",
    "\n",
    "Here we demonstrate how AutoGluon's HPO methods can help you tune various pieces of your own non-AutoGluon/non-MXNet models. As an example, we consider a custom PyTorch model for classifying images from the [MNIST handwritten digit dataset](https://en.wikipedia.org/wiki/MNIST_database). The PyTorch code used in this tutorial is adapted from this [git repo](https://github.com/kuangliu/pytorch-cifar). In your applications, this code can be replaced with analogous code for your own model.\n",
    "\n",
    "In order to run the second half of this tutorial, please first [install PyTorch](https://pytorch.org/).\n",
    "\n",
    "\n",
    "\n",
    "## Search Space Overview\n",
    "\n",
    "We first explain some of the search spaces supported by AutoGluon, including simple search spaces (`Int`, `Real`, and `Categorical`) and nested search spaces (`Categorical`, `List`, `Dict`). Each search space describes the set of possible values for a hyperparameter, from which the searcher will try training models with particular values during hyperparameter optimization. AutoGluon also enables search spaces inside user-defined Python objects using the decorator `@ag.obj` and user-defined functions using the decorator `@ag.func`.\n",
    "\n",
    "Let's start by importing AutoGluon:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import autogluon as ag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Integer Space**: The searcher will choose an integer between the provided lower and upper bounds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Int: lower=0, upper=10\n"
     ]
    }
   ],
   "source": [
    "a = ag.space.Int(lower=0, upper=10)\n",
    "print(a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sample a random value from the search space:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "a.rand"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Real Space**: The searcher will choose an integer between the provided lower and upper bounds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Real: lower=0.0001, upper=0.01\n"
     ]
    }
   ],
   "source": [
    "b = ag.space.Real(lower=1e-4, upper=1e-2)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also specify that values should be searched on a logarithmic rather than linear scale between these bounds:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Real: lower=0.0001, upper=0.01\n",
      "Example configuration: 0.0008096174072908544\n"
     ]
    }
   ],
   "source": [
    "c = ag.space.Real(lower=1e-4, upper=1e-2, log=True)\n",
    "print(c)\n",
    "\n",
    "print(\"Example configuration:\", c.rand)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Categorical Space**: The searcher will chooses one value from a discrete set of values. The first value listed corresponds to the *default* that will be tried first during the search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Categorical['Monday', 'Tuesday', 'Wednesday']\n",
      "Example configuration: Wednesday\n"
     ]
    }
   ],
   "source": [
    "d = ag.space.Categorical('Monday', 'Tuesday', 'Wednesday')\n",
    "print(d)\n",
    "\n",
    "print(\"Example configuration:\", d.rand)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Nested Search Space\n",
    "\n",
    "A categorical space can also be used as a nested search space, where its options may themselves be search spaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Categorical[Int: lower=0, upper=10, Real: lower=0.0001, upper=0.01, Real: lower=0.0001, upper=0.01]\n",
      "Example configuration is also a search space:\n",
      " Int: lower=0, upper=10\n",
      "Example configuration from the previously-selected search space:\n",
      " 8\n"
     ]
    }
   ],
   "source": [
    "a_b_or_c = ag.space.Categorical(a, b, c)\n",
    "print(a_b_or_c)\n",
    "\n",
    "which_space = a_b_or_c.rand\n",
    "print(\"Example configuration is also a search space:\\n\", which_space)\n",
    "\n",
    "which_value = which_space.rand\n",
    "print(\"Example configuration from the previously-selected search space:\\n\", which_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**List Space**: The Searcher will choose one option from each element of the list (where each element is a search space), and then return a list of these choices. This can be useful for specifying things like network architectures which might require choosing the width of each layer say.\n",
    "\n",
    "In the example below, the first element of the list is an `Int` space sampled\n",
    "from 0 to 3, and the second element is a `Categorical` space sampled\n",
    "from the choices `'alpha'` or `'beta'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "List[Int: lower=0, upper=3, Categorical['alpha', 'beta']]\n",
      "Example configuration: [2, 'alpha']\n"
     ]
    }
   ],
   "source": [
    "f = ag.space.List(\n",
    "        ag.space.Int(0, 3),\n",
    "        ag.space.Categorical('alpha', 'beta'),\n",
    "    )\n",
    "print(f)\n",
    "\n",
    "print(\"Example configuration:\", f.rand)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Dict Space**: Similar to a `List` space, a configuration returned by the Searcher will be a dict where the values of each key will be chosen from among the options provided in this `Dict` space. \n",
    "\n",
    "In the example below, the value of `'key1'` is chosen from\n",
    "a `Categorical` space with the choices `'alpha'` or `'beta'`,\n",
    "and the value of `'key2'` is chosen from an `Int` space that spans 0-9."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dict{'key1': Categorical['alpha', 'beta'], 'key2': Int: lower=0, upper=9, 'key3': 'constant'}\n",
      "Example configuration: {'key1': 'alpha', 'key2': 3, 'key3': 'constant'}\n"
     ]
    }
   ],
   "source": [
    "g = ag.space.Dict(\n",
    "        key1=ag.space.Categorical('alpha', 'beta'),\n",
    "        key2=ag.space.Int(0, 9),\n",
    "        key3='constant'\n",
    "    )\n",
    "print(g)\n",
    "\n",
    "print(\"Example configuration:\", g.rand)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decorators for Searchable Objects and Training Functions\n",
    "\n",
    "We now demonstrate how you can insert search spaces into arbitrary Python objects and functions, without really needing to modify their existing code.\n",
    "\n",
    "**Searchable Object:** With the addition of one AutoGluon decorator, any object returned by a user-defined class can have its constructor-arguments turned into searchable hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AutoGluonObject -- MyObj\n",
      "Example configuration: MyObj -- name: auto, rank: 3, static_value: 10\n"
     ]
    }
   ],
   "source": [
    "@ag.obj(\n",
    "    name=ag.space.Categorical('auto', 'gluon'),\n",
    "    static_value=10,\n",
    "    rank=ag.space.Int(2, 5),\n",
    ")\n",
    "class MyObj:\n",
    "    def __init__(self, name, rank, static_value):\n",
    "        self.name = name\n",
    "        self.rank = rank\n",
    "        self.static_value = static_value\n",
    "    def __repr__(self):\n",
    "        repr = 'MyObj -- name: {}, rank: {}, static_value: {}'.format(\n",
    "                self.name, self.rank, self.static_value)\n",
    "        return repr\n",
    "\n",
    "h = MyObj()\n",
    "print(h)\n",
    "print(\"Example configuration:\", h.rand)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that none of the of the existing `MyObj` code would need to be modified in order to make this object searchable. We just added the `@ag.obj` decorator and provided a search space to consider. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Custom Training Function:** `train_func` is where to put your model training script, which takes in various keyword `args` as its hyperparameters. Usually, your training-script will evaluate your model (either at the end of training or after each epoch) and log this somewhere. The only extra code required to hyperparameter-tune your training-function with AutoGluon is to add the evaluation-score into the provided `reporter` object as well. Here, we show a dummy `train_func` that doesn't do much."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "@ag.args(\n",
    "    a=ag.space.Int(1, 10),\n",
    "    b=ag.space.Real(1e-3, 1e-2),\n",
    "    c=ag.space.Real(1e-3, 1e-2, log=True),\n",
    "    d=ag.space.Categorical('a', 'b', 'c', 'd'),\n",
    "    e=ag.space.Bool(),\n",
    "    f=ag.space.List(\n",
    "            ag.space.Int(1, 2),\n",
    "            ag.space.Categorical(4, 5),\n",
    "        ),\n",
    "    g=ag.space.Dict(\n",
    "            a=ag.Real(0, 10),\n",
    "            obj=MyObj(),\n",
    "        ),\n",
    "    h=ag.space.Categorical('test', MyObj()),\n",
    ")\n",
    "def train_fn(args, reporter=None):\n",
    "    performance = args.a + args.b + args.c # you should do stuff here\n",
    "    epoch = 0 # can (optionally) record each epoch's performance \n",
    "    print('args: {}'.format(args))\n",
    "    print('performance: ', performance)\n",
    "    if reporter is not None:\n",
    "        reporter(epoch=epoch, performance=performance)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Begin Hyperparameter-tuning your Custom Code\n",
    "\n",
    "With all the pieces in place, we're ready to create a Searcher object, which will follow a particular hyperparameter-tuning strategy (basic random search in this case)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'a': 6, 'b': 0.0055, 'c': 0.0031622777, 'd▁choice': 0, 'e': 0, 'f▁0': 2, 'f▁1▁choice': 0, 'g▁a': 5.0, 'g▁obj▁name▁choice': 0, 'g▁obj▁rank': 4, 'h▁1▁name▁choice': 0, 'h▁1▁rank': 4, 'h▁choice': 0}\n"
     ]
    }
   ],
   "source": [
    "searcher = ag.searcher.RandomSearcher(train_fn.cs)\n",
    "config = searcher.get_config()\n",
    "print(config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `config` contains indices corresponding to choices of `Categorical` hyperparameters and AutoGluon's internal representations of certain nested hyperparameters. \n",
    "Here's how to run one training job with this hyperparameter-configuration specified by the searcher:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "args: {'a': 6, 'b': 0.0055, 'c': 0.0031622777, 'd': 'a', 'e': 0, 'f': [2, 4], 'g': {'a': 5.0, 'obj': MyObj -- name: auto, rank: 4, static_value: 10}, 'h': 'test', '_default_config': {}}\n",
      "performance:  6.0086622777\n"
     ]
    }
   ],
   "source": [
    "train_fn(train_fn.args, config)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can tune the argument-values of this `train_fn` by creating a scheduler that serially orchestrates training jobs under hyperparameter-configurations specified by this searcher."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "scheduler_options: Key 'resume': Imputing default value False\n",
      "scheduler_options: Key 'visualizer': Imputing default value none\n",
      "scheduler_options: Key 'training_history_callback_delta_secs': Imputing default value 60\n",
      "scheduler_options: Key 'delay_get_config': Imputing default value True\n",
      "\n",
      "Starting Experiments\n",
      "Num of Finished Tasks is 0\n",
      "Num of Pending Tasks is 7\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "76f5b40605b64381aba505e4266d0f9c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=7.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "args: {'a': 4, 'b': 0.00834265613313634, 'c': 0.005740342482431713, 'd': 'b', 'e': 1, 'f': [2, 4], 'g': {'a': 5.6177650049668495, 'obj': MyObj -- name: auto, rank: 5, static_value: 10}, 'h': 'test', 'task_id': 0}\n",
      "performance:  4.014082998615568\n",
      "args: {'a': 7, 'b': 0.0025860934950148106, 'c': 0.007035691031609521, 'd': 'b', 'e': 1, 'f': [2, 5], 'g': {'a': 4.1863394009431145, 'obj': MyObj -- name: gluon, rank: 3, static_value: 10}, 'h': 'test', 'task_id': 1}\n",
      "performance:  7.009621784526624\n",
      "args: {'a': 3, 'b': 0.003760566924092516, 'c': 0.007004655038721295, 'd': 'd', 'e': 0, 'f': [1, 4], 'g': {'a': 8.419395583612442, 'obj': MyObj -- name: auto, rank: 2, static_value: 10}, 'h': 'test', 'task_id': 2}\n",
      "performance:  3.0107652219628136\n",
      "args: {'a': 2, 'b': 0.009383737772154229, 'c': 0.004485339037776226, 'd': 'b', 'e': 1, 'f': [1, 5], 'g': {'a': 1.7120445565470621, 'obj': MyObj -- name: auto, rank: 5, static_value: 10}, 'h': 'test', 'task_id': 3}\n",
      "performance:  2.0138690768099305\n",
      "args: {'a': 9, 'b': 0.0055259439879873995, 'c': 0.0010395732339477993, 'd': 'b', 'e': 0, 'f': [1, 4], 'g': {'a': 6.332616532640345, 'obj': MyObj -- name: auto, rank: 5, static_value: 10}, 'h': MyObj -- name: gluon, rank: 5, static_value: 10, 'task_id': 4}\n",
      "performance:  9.006565517221935\n",
      "args: {'a': 2, 'b': 0.0029683405694828995, 'c': 0.0014361298528051594, 'd': 'd', 'e': 1, 'f': [2, 5], 'g': {'a': 8.606162403971279, 'obj': MyObj -- name: auto, rank: 3, static_value: 10}, 'h': MyObj -- name: auto, rank: 2, static_value: 10, 'task_id': 5}\n",
      "performance:  2.004404470422288\n",
      "\n",
      "args: {'a': 3, 'b': 0.009931624797943975, 'c': 0.0010646287036405657, 'd': 'd', 'e': 1, 'f': [2, 5], 'g': {'a': 6.3266743753484835, 'obj': MyObj -- name: auto, rank: 4, static_value: 10}, 'h': 'test', 'task_id': 6}\n",
      "performance:  3.0109962535015846\n",
      "\n",
      "Best configuration:\n",
      " {'a': 9, 'b': 0.0055259439879873995, 'c': 0.0010395732339477993, 'd▁choice': 1, 'e': 0, 'f▁0': 1, 'f▁1▁choice': 0, 'g▁a': 6.332616532640345, 'g▁obj▁name▁choice': 0, 'g▁obj▁rank': 5, 'h▁1▁name▁choice': 1, 'h▁1▁rank': 5, 'h▁choice': 1}\n",
      "\n",
      "Best performance value:\n",
      " 9.006565517221935\n"
     ]
    }
   ],
   "source": [
    "scheduler = ag.scheduler.FIFOScheduler(train_fn, searcher=searcher,\n",
    "                                       resource={'num_cpus': 2, 'num_gpus': 0},\n",
    "                                       num_trials=7,\n",
    "                                       reward_attr='performance', # name of performance metric in reporter \n",
    "                                       time_attr='epoch') # name of time-attribute in reporter\n",
    "scheduler.run()\n",
    "scheduler.join_jobs()\n",
    "best_config = scheduler.get_best_config()\n",
    "best_perf = scheduler.get_best_reward() # assumes higher values = better\n",
    "print(\"\\nBest configuration:\\n\", best_config)\n",
    "print(\"\\nBest performance value:\\n\", best_perf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tuning PyTorch Model for MNIST\n",
    "\n",
    "We'll now demonstrate these concepts for a practical application of training a Pytorch neural network on the MNIST image dataset. Let's first install & import the necessary packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install torch torchvision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "import torchvision\n",
    "import torchvision.transforms as transforms\n",
    "from tqdm.auto import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some random images from each class in the MNIST dataset:\n",
    "\n",
    "<img src=\"images/MnistExamples.png\" width=\"400\" height=\"400\">\n",
    "\n",
    "We'll apply standard image transforms to our training and validation images:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "transform = transforms.Compose([\n",
    "   transforms.ToTensor(),\n",
    "   transforms.Normalize((0.1307,), (0.3081,))\n",
    "])\n",
    "\n",
    "# load the datasets:\n",
    "trainset = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=transform)\n",
    "testset = torchvision.datasets.MNIST(root='./data', train=False, download=True, transform=transform)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Main Training Script\n",
    "\n",
    "The following `train_mnist` function represents normal training code you'd write for\n",
    "fitting neural networks to the MNIST data. In Python, one can use an argparser to conveniently\n",
    "change default argument values. If your training function does this, then the only additional argument you need to add to your existing code is a `reporter` object that logs performance achieved after each training epoch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_mnist(args, reporter):\n",
    "    # get variables from args\n",
    "    lr = args.lr\n",
    "    wd = args.wd\n",
    "    epochs = args.epochs\n",
    "    net = args.net\n",
    "    print('lr: {}, wd: {}'.format(lr, wd))\n",
    "\n",
    "    device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
    "    # Model\n",
    "    net = net.to(device)\n",
    "\n",
    "    if device == 'cuda':\n",
    "        net = nn.DataParallel(net)\n",
    "    criterion = nn.CrossEntropyLoss()\n",
    "    optimizer = torch.optim.SGD(net.parameters(), lr=args.lr, momentum=0.9, weight_decay=wd)\n",
    "\n",
    "    # datasets and dataloaders\n",
    "    trainset = torchvision.datasets.MNIST(root='./data', train=True, download=False, transform=transform)\n",
    "    trainloader = torch.utils.data.DataLoader(trainset, batch_size=128, shuffle=True, num_workers=2)\n",
    "\n",
    "    testset = torchvision.datasets.MNIST(root='./data', train=False, download=False, transform=transform)\n",
    "    testloader = torch.utils.data.DataLoader(testset, batch_size=128, shuffle=False, num_workers=2)\n",
    "\n",
    "    # Training\n",
    "    def train(epoch):\n",
    "        net.train()\n",
    "        train_loss, correct, total = 0, 0, 0\n",
    "        for batch_idx, (inputs, targets) in enumerate(trainloader):\n",
    "            inputs, targets = inputs.to(device), targets.to(device)\n",
    "            optimizer.zero_grad()\n",
    "            outputs = net(inputs)\n",
    "            loss = criterion(outputs, targets)\n",
    "            loss.backward()\n",
    "            optimizer.step()\n",
    "\n",
    "    def test(epoch):\n",
    "        net.eval()\n",
    "        test_loss, correct, total = 0, 0, 0\n",
    "        with torch.no_grad():\n",
    "            for batch_idx, (inputs, targets) in enumerate(testloader):\n",
    "                inputs, targets = inputs.to(device), targets.to(device)\n",
    "                outputs = net(inputs)\n",
    "                loss = criterion(outputs, targets)\n",
    "\n",
    "                test_loss += loss.item()\n",
    "                _, predicted = outputs.max(1)\n",
    "                total += targets.size(0)\n",
    "                correct += predicted.eq(targets).sum().item()\n",
    "\n",
    "        acc = 100.*correct/total\n",
    "        # This is the only extra line to add to your code:\n",
    "        reporter(epoch=epoch+1, accuracy=acc) # 'epoch' reports the number of epochs completed\n",
    "\n",
    "    for epoch in tqdm(range(0, epochs)):\n",
    "        train(epoch)\n",
    "        test(epoch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using AutoGluon for HPO\n",
    "\n",
    "Now we demonstrate how to define a searchable network architecture, convert the training function to be searchable, create the scheduler, and then launch the HPO experiment.\n",
    "\n",
    "### Define a Searchable Network Achitecture\n",
    "\n",
    "Let's define a 'dynamic' network with searchable configurations by simply adding our `@ag.obj` decorator. In this example, we search two arguments `hidden_conv` and\n",
    "`hidden_fc`, which represent the number of channels/hidden-units in convolutional/fully-connected layers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "@ag.obj(\n",
    "    hidden_conv=ag.space.Int(6, 12),\n",
    "    hidden_fc=ag.space.Categorical(80, 120, 160),\n",
    ")\n",
    "class Net(nn.Module): # put your Pytorch network here\n",
    "    def __init__(self, hidden_conv, hidden_fc):\n",
    "        super().__init__()\n",
    "        self.conv1 = nn.Conv2d(1, hidden_conv, 5)\n",
    "        self.pool = nn.MaxPool2d(2, 2)\n",
    "        self.conv2 = nn.Conv2d(hidden_conv, 16, 5)\n",
    "        self.fc1 = nn.Linear(16 * 4 * 4, hidden_fc)\n",
    "        self.fc2 = nn.Linear(hidden_fc, 84)\n",
    "        self.fc3 = nn.Linear(84, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = self.pool(F.relu(self.conv1(x)))\n",
    "        x = self.pool(F.relu(self.conv2(x)))\n",
    "        x = x.view(-1, 16 * 4 * 4)\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make the Training Function Searchable\n",
    "\n",
    "We can simply add the `@ag.args` decorator and the `train_mnist` function argument values will become tunable by AutoGluon. In the example below, we specify that `lr` is a real-value that should be searched on a log-scale in the range 0.01 - 0.2. Before passing `lr` to your train function, AutoGluon always selects an actual floating point value to assign to `lr`, so your existing code doesn't need to change for accommodating the hyperparameter search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "@ag.args(\n",
    "    lr = ag.space.Real(0.01, 0.2, log=True),\n",
    "    wd = ag.space.Real(1e-4, 5e-4, log=True),\n",
    "    net = Net(),\n",
    "    epochs=5, # small value used here for quick demo, you should increase this to see more realistic performance.\n",
    ")\n",
    "def ag_train_mnist(args, reporter):\n",
    "    return train_mnist(args, reporter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Scheduler and Launch the Experiment\n",
    "\n",
    "For hyperparameter tuning, AutoGluon provides different schedulers you can choose from:\n",
    "\n",
    "- [**FIFOScheduler**](https://autogluon.mxnet.io/api/autogluon.scheduler.html#autogluon.scheduler.FIFOScheduler): Each training jobs runs for the full number of epochs\n",
    "\n",
    "- [**HyperbandScheduler**](https://autogluon.mxnet.io/api/autogluon.scheduler.html#hyperbandscheduler): Uses Hyperband with successive halving in order to terminate unpromising jobs in early epochs, to free compute resources for new training jobs.\n",
    "\n",
    "Each scheduler is internally configured by a searcher, which determines what hyperparameter configurations should be tried. The default searcher simply draws configurations uniformly at random from the search space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "scheduler_options: Key 'searcher': Imputing default value random\n",
      "scheduler_options: Key 'resume': Imputing default value False\n",
      "scheduler_options: Key 'visualizer': Imputing default value none\n",
      "scheduler_options: Key 'training_history_callback_delta_secs': Imputing default value 60\n",
      "scheduler_options: Key 'delay_get_config': Imputing default value True\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FIFOScheduler(\n",
      "DistributedResourceManager{\n",
      "(Remote: Remote REMOTE_ID: 0, \n",
      "\t<Remote: 'inproc://172.31.68.247/10038/1' processes=1 threads=4, memory=64.28 GB>, Resource: NodeResourceManager(4 CPUs, 1 GPUs))\n",
      "})\n",
      "\n"
     ]
    }
   ],
   "source": [
    "myscheduler = ag.scheduler.FIFOScheduler(ag_train_mnist,\n",
    "                                         resource={'num_cpus': 4, 'num_gpus': 1},\n",
    "                                         num_trials=2, # small value used here for quick demo, you should increase this to see more realistic performance.\n",
    "                                         time_attr='epoch',\n",
    "                                         reward_attr=\"accuracy\")\n",
    "print(myscheduler)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Starting Experiments\n",
      "Num of Finished Tasks is 0\n",
      "Num of Pending Tasks is 2\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "211e5bb6efc3419c8b075efa1f5a6494",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=2.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lr: 0.0447213595, wd: 0.0002236068\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "94daad6d703647ac9d7fb2842fcf1180",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=5.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "lr: 0.028245913732173278, wd: 0.00017160776862349322\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "51af311711d44460abd66678a0816a75",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=5.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "myscheduler.run()\n",
    "myscheduler.join_jobs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We plot the test accuracy achieved over the course of training under each hyperparameter configuration that AutoGluon tried out (represented as different colors)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nOzdd3hVVdbA4d9KJ5ACJPQSeu8IiFRBKXbHsY/jOOpnB0Ydy9gr9jKOOo7OOGPvFQERlSIi0g0ECL1DQkhISE/W98c5gWu8gQRyc1LW+zx5knvuKeuWnHX23mfvLaqKMcYYU1qQ1wEYY4ypnixBGGOM8csShDHGGL8sQRhjjPHLEoQxxhi/LEEYY4zxyxKER0TkIRFJFZHdXsdSV4lIsIhkiUibyly3sonI6yLy1yo4zlgRWRHo45SXiESIiIpIK69jqUwiMldEflfOdVNFZEigYyqLJYhyEpHNIpLjniT2iMh/RKTBMe6rNXAz0F1Vm1VupLWX+96X/BT7fB5ZInJJRfenqkWq2kBVt1bmuhUhIp1LvS4VkYM+j09Q1ctV9fHKPK4/qvqNqvYJ9HECRUSmisirpZb9Ksm46xS47226iMwXkRN81m8rIu+KSJq7zkIRGVfG8cb6fE4H3eP4fpZN/G2nqiNU9aPKfO2BYgmiYs5Q1QZAf+AE4K6K7kBEQoC2wD5V3XuM29dJ7gm6gfsZbMX9PNyft0qvXxPeK1Vd5/Oa4tzFXXxe189exldL/dd9v5sAS4APANwT+g9AOtDVff4l4EMROaP0TtyEWvLZDQCKfL+jpf+/RSRIRCSgr6ySWYI4Bqq6A5gO9AQQkRgReU1EdonIDrf6KNh97nIR+UFEnhGRNOB7YBbQwr3KeN1d70wRWeVe1XwvIt1KjueWXm4TkZXAQREJcZfdKiIr3auX10SkqYhMF5FMEflGRBr67OMDEdktIhluEbeHz3Ovi8g/RGSau+1PItLB5/keIjLLvaraIyJ3usuDROR2EdkgIvtE5H0RaeTvPRORJBE53edxiFt87u9e5b3p7iNdRH4WkaYV/Vzc9/09EXlHRDKBS0XkRPcqMN39fJ4XkVCfGFREEtzHb7rPl7yHP4pIu4qu6z4/QUTWue/3393vwOUVfU3uvt4Vkbvcv8eLyHoRuct9/3aIyEQROcvnc7jZZ9tgEblbRDa6678lIrFlHGe8iKz3ebxbRKaISKL7Ot4SkbAjxPl/IrLW/Z5ME5GWPs+9JCLbReSAiCwSn2oT9729143xgPv5+5asJ7ivbb+IPHMs76E/qpoP/A9oK05twK3AHlW9RlX3qmq2qv4XeBJ46liOISKLReQ+EVkEZANN3WUXus93d/8f00RkrxxHzUQgWII4BuJUEU0ElrmL/gsUAh2BfsCpwJU+mwwGNuJckZwCTAB2ulcZl4tIZ+AdYDIQD3wFfFHqn/Ei4DQgVlUL3WW/c/fXGTgDJ2ndiXMlGgTc5LP9dKCTG8NSoPQV90XA/UBDYD3wsPtao4BvgBlAC/c1zna3uQk4GxjpPrcf+EcZb9s77jFKjANSVXUp8EcgBmgNNAauAXLK2M/RnAO87e7vPZzPZRLOe3ISMB74vyNsfzFwN9AIp5TyYEXXFedK9H2cE04csAkYdIyvx5+2QAHQDJgK/Bs4D+gNjAUe9jk534rzfRwGtHK3q8hJ9jxgDM7nPhjnNf+Ge8KbjPM9bIrzv/Gmzyo/Ar1wPt/PgA9KEjVwB8736FQgFrgayPXZdgLO/1V/4E8iMqoC8ZdJRCJwvnvrVTUL53/pQz+rvg90EpG2x3ioS4FLgGggxc/z9+C8Z32AHsDtx3icyqeq9lOOH2AzkIVT/NwCvAjUw/lg84B6PuteBHzn/n05sLXUvkYB230e3w287/M4CNgBjPI59hV+4rnE5/FHwEs+j28EPi3jtcQCCsS4j18HXvV5fiKwxue1LCtjP0nAGJ/HzXFOQCF+1u0IZAKR7uO3gHvcv68AFgC9K/h5jC217CHg26Nsdwvwgft3iPs+JLiP3wRe9ln3TCDxGNa9Apjn85wAu4DLjxJbhHuMVqWWvwvc5f49HsgAgtzH8e42fXzWXwWMd//eBJzk81w7nCtZ8XP88Tgny5LHu4HzfB4/DzxbRuzflfo+hrrfhaZ+1hU3hi7u4y3AuCO8HwN9ln0OTC4jhqk4/4vppX4Ovael1tmLU5rv7T633d9nxOH/lwFH+Oy6AoV+li8G/upn2YVl7OfSUt+dVGBIef8vKvun2tfRVjNnq+o3vgtEpBfOP8MuOVy9GARs81nN929/WuD8kwCgqsUisg1o6bOOv33s8fk7x8/jBm6MwTglgt/jnFCK3XXicE424JwMSmSXbItzVb+hjLjbAp+ISLHPsiKcpLnDd0VVXS8iScAZIvIFzgm1n/v0G+5x3nWrP94E/qaqBWUc90h+9T6JSFec6oEBQCTOif6nI2xf1vtQkXVb+Mahqioi231iWsvhz/YUVf3xCMfwJ0VVS97zkpLWbz57cb6QrYGvRMR3VM4gnCv51HIcq/RrjCtjvbbAyyLiW4IsxCm17BGRO3AulprhnGwjgDgRWYfzXpT1HfMXw5E+kzdU9VDp3S0llC6N/modH6k4FzmlNfd5/liU+f8vTuP5s8BQnNdV+tzhKatiOn7bcK5I4lQ11v2JVtUePuscbcjcnTj/YAD4/GP7nmSPZ9jdi4GzcKofYoCEkkOVY9ttQIcjPDfB53XHqmqEOm00/pRUM50FrFbV9QCqWqCq96tqd5x/lNOBy8oRmz+l36d/AolAR1WNxinOB7qhcBfOiRE49HkeSvaq6tsIXdHkUG7qXILuAE728xkd68muLNtwrr59j1NPVZeIyCk4JdpzcK7GG+GctMUnxrK+Y1XpG5wqtdLOBzao6hY/z5XHkf53n8K5SOvufj+vIfDfz3KzBHGcVHUX8DXwlIhEi9Nw20FERlZgN+8Dp4nIGLde9macpLOgksKMcve3D+cq+pEKbPsl0ExEJotIuIhEichg97mXceq72wKISLyInHWEfb2LU898LU47Ae52o0Wkl1vSOYBTNVFUgRiPJArnH/CgOA3/R2p/qCxfAv1F5Axx7qSahFNy88LLwFS33QwRaSJ+7sippOPcJSJd3OM0lMP3+kfhfKYpQBjwAE4JosSrwCMi0l4c/cpqSA+wJ4DmIvKy+z7VE5HLcNpxbgnQMaNwql4PiHMDxJQAHeeYWIKoHJfhfPFX4zTUfoj/oqpfqroWp+7x7zjF2DNwbuHMr6T4/odThbXDjXFhBWLLxGm8OwOnqJ8MjHaffg6nTvhrce4aWojTkFnWvnbhNFYOxWlALtEM5z07gNOuMYdfN3Aej5txGiIzcUoT7x159eOnqnuAC4CncZJyB5xG27xAH9uPx3GujL91P6MFOI29lUpV3wFeAD4WkQPAcpzvDcAXwFycaqSNON9x38baqcA04Fuc78DLQHhlx3g07ud2Ek4JZy1OnNcD56vqpwE67F04/08HcP4HPgjQcY6JuA0hxpgAcUtGO3EafOd5HY8x5WUlCGMCwO1TECMi4Th3qRUCizwOy5gKsQRhTGAM43B1ynicO+C8qGIy5phZFZMxxhi/rARhjDHGr1rVUS4uLk4TEhK8DsMYY2qMJUuWpKqq39uwa1WCSEhIYPHixV6HYYwxNYaIlNkB0KqYjDHG+GUJwhhjjF+WIIwxxvhlCcIYY4xfliCMMcb4ZQnCGGOMX5YgjDHG+GUJwhhjaqqsFFj5PsyvyDTj5VerOsoZY0ytVpgP2xfB+tmwYTbsWuEsj24JJ94IwZV7SrcEYYwx1VnaRjchfAub5kJ+FkgwtB4MJ98FHcZA874QVPkVQpYgjDGmOsnLgs3zDpcS0jY6y2PbQK/fQ8ex0G4EREQHPBRLEMYY46XiYtjzy+FSwtaFUFwAoZGQMBwGX+OUEhp3AJEqDc0ShDHGVLWsFCcZbHCTwkF3iu6mveDE65yE0GYIhFT51Ny/YgnCGGMCrTAftv3kJIT1s2H3Smd5ZGPocLKTEDqMhqhm3sZZiiUIY4wJhJLG5fWznTaF/CwICoFWgwLeuFxZLEEYY0xlyMuETfMOlxL2b3KWx7aF3uc7CaGKGpcrS0AThIhMAq4CBPiXqj4rIn2Bl4EIoBC4TlUX+dn2ceA0nM58s4BJahNoG2Oqi+Jip6pow2xY/61ThVRcAKH1od1wGHIddBwDjdpXeeNyZQlYghCRnjjJYRCQD8wQkWnA48D9qjpdRCa6j0eV2nYocBLQ2100HxgJfB+oeI0x5qiy9jqNyutnw8bvftu43HGs0z/B48blyhLIEkQ3YKGqZgOIyBzgHECBkjJWDLDTz7aKU8IIwyl9hAJ7AhirMcb8Vrkal0+GqKbexhkggUwQicDDItIYyAEmAouBycBMEXkSp/poaOkNVfVHEfkO2IWTIF5Q1SR/BxGRq4GrAdq0aROI12GMqUv2bThcSvBtXG49GE6+26k2atanWjcuV5aAJQhVTRKRx3DaD7KAFThtDtcCU1T1IxE5H3gNGOu7rYh0xCmBtHIXzRKREao6189xXgFeARg4cKC1URhjKiYv0xnCoqTn8v7NzvKGCdD7AichJAyvUY3LlSWgjdSq+hpOAkBEHgG2A48Ck9xVPgBe9bPpOTjVU1nuttOBIcBvEoQxxlTIrxqXZ7uNy4Vu4/IIOPEGp9qocQevI/VcoO9iaqKqe0WkDXAucCJwI4cbnE8Gkv1suhW4SkQexaliGgk8G8hYjTG1mG/j8oZvITvVWd6sl5MQOo6B1kMgJMzbOKuZQPeD+MhtgygArlfV/SJyFfCciIQAubjtByIyELhGVa8EPsRJHr/gNFjPUNUvAhyrMaa2KMyHbQsPVxvt/sVZHhnnlA46joH2o2tt43JlkdrUtWDgwIG6ePFir8Mwxnhh34bDCWHTPCg46DYuD4GO7h1HzXrXicblihCRJao60N9z1pPaGFMzHalxuc+FdbpxubJYgjDG1AzFxbB7xeF2BGtcDjhLEMaY6itzj8+w2N/5NC73hqE3OtVGrQdb43KAWIIwxlQfRQVOySD5a2d8oz1u43L9eKfKqGRY7AZNvI2zjrAEYYzxVnYarP8G1s1wfudmHG5cHnOvkxia9rLGZQ9YgjDGVC1VSFkL66bDuplOiUGLoX4T6HYGn+f05r19HRjQqg0j28TRp0ksIZYcPGEJwhgTeIV5sHm+kxDWzYD0Lc7yZr1h+C3QeTy06MfKnQeY9I8faBYt/PhtMs/PTiY6IoSTOsYxonM8IzrH0zK2nrevpQ6xBGGMCYysvU5bwtrpTgNzwUEIiYD2o2DYFOh0KsS0PLR6UbFy16eJxDUIZ+aUERQXKz+s38fcdSnMTU5heuJuADrE1z+ULIa0a0y9sGBvXl8dYAnCGFM5VJ0xjkpKCTuWOMujW0KfC5xSQsJwCIv0u/k7i7aycnsGz13Yl+iIUABO692c03o3R1VZvzeLOetSmLMuhbd/2sp/fthMWEgQg9s1YkQnJ2F0btoAqaGT81RH1pPaGHPs8rNh0xwnIaz7GjJ3AgItB0CX8U5SaNrzqDOqpWblcfKT39OjRQxvXzX4qCf53IIiftqU5pQu1qWQvDcLgGbREQzv5FRHDesYR8P6dvvr0VhPamNM5cnY7pYSZjrJoTAXwho4ndQ6j4dOp1T4NtSp09eQnV/Eg2f3KFcJICI0mJGd4xnZOR6Anek5zEtOYe66VL5evYcPlmxHBHq3imWkmzD6to4lJNgauyvCShDGmCMrLnaqi9bNcJJCSd+E2LbQZYKTFNoOPeZpNn/enMbvX/6Ra0Z24PYJXY873KJiZcX29EOli+Xb0ilWiIoI4aQOJY3dcbRq6L+qq645UgnCEoQx5rdyDzhzLpeUFLJTQYKhzRDoPM5JCnGdj1p1dDQFRcWc/vx8MnML+ObmkUSGVX6lRkZ2AfPXpx5q7N6VkQtA+/j6jOgUz8gudbux26qYjDFHl7bxcAPz5h+guAAiYp0qo87jnSqkyEaVesj/LtjM2j2Z/PMPAwKSHABiIkP9NnbPTU7lnUVbeX2B09g9KKERIzo7JYwuTaOssRsrQRhTdxUVOp3USqqOUtc6y+O6HC4ltB4MwYE5ce/OyGXMU98zqF0j/n35CZ6ckHMLilhU0tidnMK6PU5jd9PocIa7d0YNr+WN3VaCMMY4stOc0VDXzYD1s9xhLUIh4SQYeAV0PhUata+SUB6ctprCYuX+M3t6drUeERp8qE8FwK6MHOatS2VOcgqzVu/hw5LG7pYxh9brV4cau60EYUxtpgqp6w6XErYuBC1yZlbrPM75aT+6yudMmJecwh9eW8SUsZ2ZNLZTlR67vIqKlZXb053qKN/G7vAQhnZs7CSMTvG0blSzG7utkdqYuqQwD7b8cLg9oWQinaa9nITQZQK06O/Z4Hd5hUWMf3YeqsqMySOICK0ZjcMZ2QX8sCH10N1RO0s3dneOZ3D7RgFrSwkUq2IyprbLSnGGtVg3w5k/IT/LGdai3UgYepOTGGJaeR0lAK/M2cim1IP894pBNSY5gNPYPbFXcyb2chq7N6RkMWedkzAONXYHB3FCu4aHenZ3bVazG7utBGFMTaQKu38pNayFQlRzt+pogjPLWhnDWnhlW1o2Y5+ew5huTXjxkgFeh1NpcguK+HlzSc/uVNbuyQSgSVRJY3ccwzvF06gaNnZbFZMxtUFBjjMHc0l7woEdzvKWA5w7jjqPc0ZHrcZXrH9+/Wd+3LiP2TePpHlM7R2VdXdGLnOTnaqoecmpZOQUIAK9WsYw0m3s7ts6ltBq0NjtWYIQkUnAVYAA/1LVZ0WkL/AyEAEUAtep6iI/27YBXgVaAwpMVNXNRzqeJQhT62TsgGS3s9rGOVCY4w5rMdpJCh1PgaimXkdZLrNW7+Gq/y3mzolduXpE3Zk3uqSxe+66VOYmp7Bs6/5q1djtSYIQkZ7Au8AgIB+YAVwL/AN4RlWni8hE4K+qOsrP9t8DD6vqLBFpABSravaRjmkJwtR4xcWwc5lbSpjuVCMBxLZxqo06j4OEYcc8rIVXcvKLGPv0HOqHBzPtpuHV4srZKxk5BSxYn+qWMFLZkZ4DQPu4+oeGARnSvnGVNXZ71UjdDVhYclIXkTnAOTilgZJ76mKAnaU3FJHuQIiqzgJQ1awAxmmMt/IynfkS1s10SgsHU0CCnE5qY+93SgrxXap11dHRvPBdMjvSc3jv6iF1OjkAxNQLZUKv5kw41Nh98FBHvXd/PtzYPTCh4aHSRbfm3jR2B7IE0Q34DDgRyAFmA4uBF4GZONVOQcBQVd1SatuzgStxSh7tgG+A21W1yM9xrgauBmjTps2ALVu2lF7FmOonbdPhu442z4eifAiPgU5j3aqjsZU+rIVX1u/NYsJzczmjdwuevqCv1+FUa2U1dsdHhTO8Uxwj3WHMGzeovBKkl20QfwauB7KA1TiJIhiYo6oficj5wNWqOrbUducBrwH9gK3Ae8BXqvrakY5nVUym2ioqhO2LDjcwp6xxljfudLhvQuvBEBzqbZyVTFW59LWfWLk9g29vHkV8VM2qGvOab2P3/PWppGcfbuwuuZW2X5vja+yuFncxicgjwHbgUSBWVVWcMlOGqkaXWncIMLWkbUJE/gAMUdXrj3QMSxCmShQVQt4B5yc3wxn5NO+Az+8M56dkWW4G7FoOOfshKATannT4rqPGtbux9vMVO7npnWU8cFYPLjsxwetwarSiYuWXHRmHOuot25ZOUbESFe7M2f3Cxf2OaQgQzzrKiUgTVd3r3pF0Lk51043ASOB74GQg2c+mPwMNRSReVVPc9ezMb45fcTHkZx4+cR/pxO4vAeRmOHMrH01oJIRHO0NYhEcfTggdToaImMC/zmogM7eAh75cTc+W0VwyuK3X4dR4wUFC39ax9G0dy01jOpGRU8CPG1KZsy6VfVl5ARkfKtDN5B+JSGOgALheVfeLyFXAcyISAuTith+IyEDgGlW9UlWLROQWYLZbylgC/CvAsZrqThXyD/o5cWeUfTIvfbLPy8S5T+IIgsOck7jvCT6qmft3jPO79PMRMb9+vpZVFR2LZ2Ylk5KVxyuXDSQ4qOY2sFdXMfVCGd+zOeN7Ng/YMQKaIFR1uJ9l84HfdKFU1cU4DdMlj2cBvQMZn6lCqs7UlLmlTuhHOpnnHoC8DJ/nM52B5o5Egn1O1u6Ju1G7Mk7m0T4ndZ9loRFV857UYqt3HuC/P27mokFt6Ns61utwzDGysZhMhWXs201kWhKhBZllXLmn+z/ZFxccZc/y66vw8GiIbgVNoo98te57NR8aWaNvB60NiouVuz9LJKZeKH8d18XrcMxxsARhKiQzM4Pcvw8lhn2/fTIs6tdX5vXjoXHHXy+LiPl1AvC9cg9r4NkIo6byfLhkO0u27Ofx83oTG1n9xh4y5WcJwlTIkg+fZBT7+Ev+NXTqfSLXjut3+AQfVHNG5jSBsf9gPo9OT2Jg24ac1796jB5rjp1drply252SQu/N/yEp8gRiT/wjjy0P5cttoVCvoSUHA8DjM9dyILeQB8/uSZA1TNd4liBMuS394HEaSSaNTr+P2yd0pX+bWG77cCUbUmwkFAPLtu7n3Z+3cvnQBLo1r9oZ6kxgWIIw5ZK0eRtD97xFcsxJNO0+jLCQIF64uD9hIUFc9+ZScvKPcneRqdWK3Ibp+AbhTK6mU4iairMEYY5KVVn18WPEykGanXX/oeUtYuvx7IX9WLc3k7s+TaQ2zS1iKuatn7aQuOMAd5/enagI6wNSW1iCMEc1P3E9p2Z8xJb4UUS1P+FXz43sHM+NJ3fio6XbeX/xNo8iNF5KyczjiZlrGdYxjtN7B67Tlql6liDMERUWFbPli8eJlmxanP2A33UmjenEsI5x3P3ZKlbtzKjiCI3XHv0qidyCIu4/q0eNnn/Z/JYlCHNEn/2YyNl5n7Or5ThCW/bxu05wkPDchX1pFBnGdW8t5UDu0TrEmdpi4cZ9fLxsB1ePaE+H+AZeh2MqmSUIU6aDeYVkfvs0kZJHszPvO+K6jRuE88LF/di+P4dbP1hh7RF1QEFRMXd/mkjL2HrcMNoapmsjSxCmTG98s5jzi74ivf0ZSNPuR11/YEIj7pjQlZmr9vDa/E1VEKHx0r/nbyJ5bxb3ndmDemHWD6Y2sgRh/NpzIJewn/5OhBTQaOI95d7uz8PaMa5HU6ZOX8PizWkBjNB4aWd6Ds/NTmZstyac0r2p1+GYALEEYfz617QFXCwzye76O4grf/WBiPD4eX1o2bAeN7y9jNSsvABGabzy4JerKVbl3jN6eB2KCSBLEOY3knYdoOWqlwmTYhqcckeFt4+pF8qLl/QnLTufye8up6jY2iNqk+/X7mV64m5uGN2R1o0ivQ7HBJAlCPMbL38xj4tDvqWg14XHPCVmjxYxPHBmD+avT+X52f4mDTQ1UW5BEfd+vor2cfW5akR7r8MxAWYJwvzK3HUpDNz6b0JFCT/5tuPa1wUntOZ3/Vvx/LfJzF2XUkkRGi+9PGcDW/Zl88BZPQkPsYbp2s4ShDmkqFh59cs5XBjyPdrvUmh4fPMIiwgPnd2Tzk2imPTuMnam51RSpMYLW/Yd5MXvN3B67+YM6xTndTimCliCMId8tHQ7E9LeJChICB55a6Xss15YMC9e2p/8wmJueHsp+YXFlbJfU7VUlXs/X0VokHDXaUe/5dnUDpYgDADZ+YW8O3MO54fMIWjA5RBTeZO9dIhvwGPn9Wbp1nSmTl9Tafs1VWfmqj18vzaFKad0plmMzdldV1iCMAC8Om8TF+W8hwSHIsNvrvT9n967BZcPTeDfP2xi+i+7Kn3/JnCy8wt54ItVdG0WxeVDE7wOx1ShgCYIEZkkIokiskpEJrvL+orIQhFZLiKLRWTQEbaPFpEdIvJCIOOs6/Zm5jJ9zjx+FzKfoBOuhOjAjMh558Ru9G0dy60frmRT6sGAHMNUvudnr2dnRi4Pnt2TkGC7pqxLAvZpi0hP4CpgENAHOF1EOgGPA/eral/gHvdxWR4E5gQqRuN4ZlYy1+iHEBIOwyYH7DhhIUH845L+hAQL1765hNwCm2Soukvek8mr8zZy3oBWnJDQyOtwTBUL5OVAN2ChqmaraiHOif4cQIGS+QhjgJ3+NhaRAUBT4OsAxljnrduTyZLFCzgzeAFBg6+GBk0CeryWsfV45oK+rNmdyT2fJQb0WOb4qDqzxNUPD+GOCV29Dsd4IJAJIhEYISKNRSQSmAi0BiYDT4jINuBJ4DdddUUkCHgKOOqtNCJytVtVtTglxe61r6ip09fwl7BPIDQShk6qkmOO7tKEG0Z35P3FNslQdfbZ8p0s3JjGreO60LhBuNfhGA8ELEGoahLwGDALmAGsAAqBa4EpqtoamAK85mfz64CvVPWoZw9VfUVVB6rqwPj4+EqLvy74YX0qO9YuZjw/IkOuhfqNq+zYU07pzNAOjbn700RW7zxQZcc15XMgt4CHpiXRp1UMFw1q43U4xiMBbXFS1ddUtb+qjgDSgGTgj8DH7iof4LRRlHYicIOIbMYpZVwmIlMDGWtdU1ysPDwtiTvrfYKGR8GJ11fp8Z1JhvoRUy+U699eSqZNMlStPP31OvYdzOPBs3sSHGSzxNVVgb6LqYn7uw1wLvAOTpvDSHeVk3GSxq+o6iWq2kZVE4BbgP+p6u2BjLWu+WTZDmT3CkYW/4QMuR4iq74BMj4qnBcu7s/WtGxu+2ilTTJUTSTuyOB/P27m0sFt6d0q1utwjIcCfc/aRyKyGvgCuF5V9+Pc2fSUiKwAHgGuBhCRgSLyaoDjMUBOfhFPfr2Wext8hkbEwInXeRbLoHaN+Ou4Lnz1y27+88Nmz+IwjuJi5a5PE2lUP4xbTu3idTjGYyGB3LmqDvezbD4wwM/yxcCVfpa/DrwegPDqrH//sIkmBxIZFL4Iht8FETGexnP1iPb8vHk/j3yVRJ/WsQxo29DTeOqy9xZvY/m2dJ76fR9iIkO9Dsd4zHq91DGpWXm89P0GHor9Auo1gsHXeB0SIsJTv+9D87JDNtsAACAASURBVNgIbnh7KWkH870OqU5KO5jPYzPWMCihEef2b+l1OKYasARRxzz3TTLdCpPolfMznDQJwqO8DgmAmMhQXrpkAPsO5jP5veUU2yRDVe6x6WvIzC3kwbN7ImIN06acCUJEPhKR09z+CaaGWr83i7cXbWVqwy+gfjwMusrrkH6lZ8sY7j2jO3PXpfDCd+u9DqdOWbJlP+8t3safh7WjS7PqcdFgvFfeE/5LwMVAsohMFRHrVlkDTZ2+huGha+mQtRiGTYGw+l6H9BsXD2rDOf1a8sw365ifnOp1OHVCYVExd32aSLPoCCaNKf/846b2K1eCUNVvVPUSoD+wGZglIgtE5E8iYi1ZNcCPG/bxTdJuHo79HBo0g4FXeB2SXyLCw+f0pGN8Aya9u4zdGbleh1TrvbFwC0m7DnDPGd2pHx7Q+1ZMDVPuKiMRaQxcjnOn0TLgOZyEMSsgkZlKU1ysPPJVEmdEJdMyYykMvxlC63kdVpkiw0J46dL+5BQUccPbSykoskmGAmXvgVye+nodwzvFMaFnM6/DMdVMedsgPgbmAZHAGap6pqq+p6o3Ag0CGaA5fp+v2MkvO9K5r8GnEN0S+l/mdUhH1bFJFFN/15vFW/bzxMy1XodTaz38VRL5hcU8cJY1TJvfKm958gVV/dbfE6o6sBLjMZUst6CIJ2au5bK49TTevxxOexpCa8aMYGf2acHPm9J4Ze5G+rdpyHi7wq1UC9an8tnyndx0ckfaxVW/9ijjvfJWMXUTkUN97kWkoYh41/3WlNvrCzazIz2bW8M/gpg20O8PXodUIXed3o3erWK49YMVbNlnkwxVlvzCYu7+LJHWjepx3eiOXodjqqnyJoirVDW95IHPkBmmGks7mM8/vl3PX9psJGrfShh5K4SEeR1WhYSHBPOPi/sTFCRc++ZSm2Sokrw6fyMbUg5y/5k9iAgN9jocU02VN0EEiU8FpYgEAzXrTFMHPT87meyCQv6v+F1o2A76XOR1SMekdaNInj6/D6t3HeD+L1Z5HU6Nt31/Nn+fvZ5Tuzfl5K5NvQ7HVGPlTRAzgfdFZIyInIwzKuuMwIVljtfGlCzeXLiFBzptIjx1FYy8DYJr7h3JY7o15dpRHXhn0TY+WrLd63BqtAe+WI2i3HNGd69DMdVceRPEbcC3OJP9XA/MBv4aqKDM8XtsxhoiQuCCg29A447Q6/deh3Tcbj6lM4PbNeJvn/7Cmt02ydCx+HbNHr5evYebxnSiVcNIr8Mx1Vx5O8oVq+pLqnqeqv5OVf+pqlYZXE0t2pTGzFV7eKL7ZkJS18CoOyC45neACgkO4u8X9yMqIpTr3lpKVl6h1yHVKLkFRdz7+So6xNfnymHtvQ7H1ADl7QfRSUQ+FJHVIrKx5CfQwZmKU1Ue/iqJ5lEhjEv5D8R3hR7neB1WpWkSFcHzF/Zjc+pBm2Sogl78bj3b0nJ48OyehIXYsGrm6Mr7LfkPznhMhcBo4H/AG4EKyhy7L1fuYsW2dJ7tsZGgfeuc0kNQ7bpL5cQOjbllXBemrdzF/37c4nU4NcKm1IO8PGcjZ/VtwdAOcV6HY2qI8iaIeqo6GxBV3aKq9+FMF2qqkbzCIh6bsYYezeozaOu/oGlP6Ham12EFxDUjOjCmaxMemraa5dvSj75BHaaq3PNZIuEhQfxtYjevwzE1SHkTRK471HeyiNwgIucATQIYlzkG/1uwhe37c3im6xokbYNbeqidVQlBQcJT5/ehSVQE17+1lP02yVCZpifuZl5yKn85tTNNomtGL3pTPZT37DEZZxymm3CmC70U+GOggjIVt/9gPn//NpkxnRvSec2L0LwPdD3N67ACKjYyjJcu7U9KZh5T3rdJhvzJyivkgS9W0715NH8Y0tbrcEwNc9QE4XaKO19Vs1R1u6r+yb2TaWEVxGfK6e/fricrr5CHE1ZC+hYY/TeoA4Ov9W4Vy92nd+P7tSm8NGeD1+FUO8/PTmb3gVwePLsnIcG1szRpAueo3xj3dtYBYkM9VlubUw/yxsLNXNS/Kc1WvAAtB0KnU70Oq8pcOqQtZ/ZpwVNfr2XBBptkqMTa3Zm8Nn8TFwxszYC2Db0Ox9RA5b2kWAZ8JiJ/EJFzS36OtpGITBKRRBFZJSKT3WV9RWShiCwXkcUiMsjPdn1F5Ed3u5UickHFXlbd8vjMNYQEBXF7058hYxuMvrNOlB5KiAiPntuLdnH1uemdZew5YJMMqSp3f5pIVEQIt02wCSDNsSlvgmgE7MO5c+kM9+f0I20gIj1xBvQbBPQBTheRTsDjwP2q2he4x31cWjZwmar2AMYDz/qOJmsOW7JlP1/9spvrhrck6ufnoPUQ6FD3bjCrHx7CS5cO4GBeETe+s4zCOj7J0MdLd7Bocxq3je9Ko/o2bJo5NuXqXquqfzqGfXcDFqpqNoCIzAHOARSIdteJAXb6Od46n793isheIB6w+xl9qCoPT1tNk6hw/q/+XMjcBee+UqdKD746N43i0XN7Mfm95Tz59Tpur6NXzhnZBTw6PYm+rWO5YGBrr8MxNVi5EoSI/AfnxP4rqnqkiY0TgYfdqUpzgInAYpw7omaKyJM4JZihRzn2IJyRY/22QIrI1cDVAG3atDnqa6lNpifuZunWdJ48uyNh866FhOHQboTXYXnq7H4tWbQ5jZfnbGBA24ac0r3ujVb65NdrSTuYz+t/GkRQUN28WDCVo7xVTF8C09yf2TglgKwjbaCqScBjOHNWzwBW4PTEvhaYoqqtgSnAa2XtQ0Sa4/TY/pOq+q0zUNVXVHWgqg6Mj48v58up+fILi5k6fQ1dmkZxbtFMOLjXaXsw3HN6d3q2jObm95ezLS3b63Cq1Mrt6bz50xYuOzGBni1jvA7H1HDlHazvI5+ft4DzgZ7l2O41Ve2vqiOANCAZp//Ex+4qH+C0UfyGiETjJKS77Jba33pj4Ra2pmVz1ymtCfrhWWg/GtoesTBWZ0SEBvPixQNQ4Nq3ltSZSYaKip2G6cb1w/nLqZ29DsfUAsd6Y3Qn4Kj1OSLSxP3dBjgXZx6JncBId5WTcZJG6e3CgE+A/6nqB8cYY62VkV3A87OTGd4pjmH7P4HsfU6/B3NIm8aRPH1+XxJ3HODBL1d7HU6VeGfRVlZsz+Cu07oRHVFz5/4w1Ud52yAy+XUbxG6cOSKO5iO3DaIAuF5V94vIVcBzIhIC5OK2H4jIQOAaVb0Sp4QyAmgsIpe7+7pcVZeXJ97a7oXvkjmQW8DfxrRC3nve6fPQ+gSvw6p2TunelP8b2Z5/ztnICQmNOLtfS69DCpjUrDyemLmWIe0bcVbfFl6HY2qJ8t7FFHUsO1fV4X6WzccZrqP08sXAle7fbwJvHssxa7ttadn8d8EWzuvfiq5b3oKc/c6YS8avW0/twrIt6dzx8S/0aBFNp6bH9FWu9qZOX8PBvEIePKsn1qfVVJbyzgdxjojE+DyOFZGzAxeWKcvjM9cSFAS3jGgGC16ALqdBy/5eh1VtlUwyVD88mGvfWsrBWjjJ0M+b0/hwyXauHN6+1iZA443ytkHcq6oZJQ9UNR24NzAhmbIs27qfL1bs5Orh7Wm6+jXIy4BRt3sdVrXXNNqZZGhjShZ3fPxLrZpkqLComLs/TaRFTAQ3jenodTimlilvgvC3Xs2fw7IGUVUe+SqJuAbhXH1CQ/jxRWeuh+a9vQ6tRhjaMY6/nNKZz1fs5M2ftnodTqV5fcFm1uzO5J4zehAZZv+SpnKVN0EsFpGnRaSDiLQXkWeAJYEMzPzazFV7+Hnzfqac0okGS16C/Cxre6ig60Z1ZFSXeB78YjUrt9f8Tvm7M3J5ZtY6RnWJZ1yPutch0AReeRPEjUA+8B7wPk7P6OsDFZT5NadTXBIdmzTggm714Kd/Qs9zoWl3r0OrUYKChGfO70t8VDjXvbWUjOwCr0M6Lg9NW01BsXL/mT2sYdoERHk7yh1U1dtLeiyr6p2qejDQwRnH2z9tYfO+bO6c2JWQhc9DYQ6MtLaHY9GwfhgvXNyPPQdy+UsNnmRofnIqX67cxXWjOtC2cX2vwzG1VHnvYprlO5qqiDQUkZmBC8uUOJBbwHOzkxnaoTGjWyosehV6nQ/x1lP2WPVr05C/TezG7DV7+efcjV6HU2F5hUXc81kibRtHcs3IDl6HY2qx8lYxxbl3LgGgqvuxOamrxIvfbSA9p4A7J3ZD5j8LRfkw8q9eh1Xj/XFoAqf1bs4TM9ewcOM+r8OpkH/N3cjG1IPcf2YPIkKDvQ7H1GLlTRDF7nAZAIhIAn5GdzWVa/v+bP79wybO6deSnlEHYfG/oe9F0NiuGo+XiPDY73qT0Lg+N76zjL2ZNWOSoW1p2fz92/VM6NmMUV3sGs0EVnkTxN+A+SLyhoi8AcwB7BaaAHty5loEuOXULjDvKdAiGHGr12HVGg3CQ3jx0v5k5hZwUw2ZZOj+L1YRHCTcfbrdoGACr7yN1DOAgcBanDuZbsa5k8kEyMrt6Xy6fCd/HtaOFqTCkv9Cv0uhYYLXodUqXZtF8/DZvVi4MY1nvll39A08NGv1Hr5J2sukMZ1oEVvP63BMHVDewfquBCYBrYDlwBDgR5zRWE0lc2aKS6Jx/TCuHdUBZt3izBI3/BavQ6uVfjegFT9vTuMf3zmTDJ3ctfr1KcjJL+K+z1fRqUkDrhjWzutwTB1R3iqmScAJwBZVHQ30A1ICFlUd903SXn7alMbksZ2Iyt4Oy96E/n+EWJs+MlDuO7MH3ZtHM+W9FdVykqEXvktmR3oOD57dk9DgYx2l35iKKe83LVdVcwFEJFxV1wBdAhdW3VVQVMyj05NoH1+fCwe1gblPggTD8Ju9Dq1WiwgN5qVL+1NcrNzw9lLyCqvPJEMbUrJ4Ze5Gzu3XkiHtG3sdjqlDypsgtrv9ID4FZonIZzgT/5hK9u7P29iYcpA7JnQjNH0TrHgHTvgzRDf3OrRar23j+jzx+z6s2J7Bw9OSvA4HcKob7/kskYjQYO6Y2M3rcEwdU975IM5x/7xPRL4DYnDmmTaVKDO3gGdnrWNwu0aM7dYEPrkbgsPgpMleh1ZnjO/ZjKuGt+Nf8zYxMKERZ/bxdvKdL1fu4of1+3jgrB7ER4V7Goupeyo8/KOqzglEIAZenrOBfQfz+c9p3ZDUdfDLB3DiDRBV/RpNa7O/ju/Ksq3p3P7RSro3j6JjE2/mWMjMLeDBL1fTs2U0lwxu60kMpm6z1q5qYmd6Dq/O28RZfVvQu1UsfD8VQurBSZO8Dq3OCQ0O4oWL+1MvNJhr31xKdr43kww9+00yKVl5PHhWT4KDbDA+U/UsQVQTT369FsXtFLdnFaz6BIZcA/XjvA6tTmoWE8FzF/ZjfUoWf/sksconGUradYDXF2zmwhPa0K9Nwyo9tjElLEFUA4k7Mvhk2Q7+dFICrRtFwvePQlgDp3rJeGZYpzgmj+nMJ8t28M6ibVV23OJi5a5PE4mpF8pfx9nNgsY7liA8VjJTXGy9UK4b1RF2rYCkL+DE6yCykdfh1Xk3ntyREZ3jue+LVSTuyDj6BpXgw6XbWbJlP7eP70rD+mFVckxj/AloghCRSSKSKCKrRGSyu6yviCwUkeUislhEBpWx7R9FJNn9+WMg4/TS92tTWLBhH5PGdCKmXqjT9hARA0Ou8zo0gzPJ0LMX9HV6tb+1hIycwE4ylJ6dz9TpaxjQtiHnDWgV0GMZczQBSxAi0hO4ChgE9AFOF5FOwOPA/araF7jHfVx620bAvcBgd/t7RaTWVcQWFhXzyFdJtIurz8WD28KOJbD2KzjxRqgXe/QdmCrRqH4YL1zcn13pudzywYqAtkc8PnMt6dn5PHhWT4KsYdp4LJAliG7AQlXNVtVCnBFgz8EZJjzaXScG/x3uxgGzVDXNnXtiFjA+gLF64v3F20nem8Vt47sSFhIE3z0K9Ro6jdOmWhnQtiF3TOzGrNV7+Ne8wEwytHxbOu8s2srlQ9vRvUX00TcwJsACmSASgREi0lhEIoGJQGtgMvCEiGwDnsT/sOEtAd9Wwe3uslojK6+Qp2et44SEhs6E89sWwfpZzm2t4d7cd2+O7IqTEpjQsxmPzVjLok1plbrvomLlrk9/Ib5BOFNO6VSp+zbmWAUsQahqEvAYztX/DGAFUAhcC0xR1dbAFOA1P5v7K1v7LdeLyNVuW8bilJSaM37gK3M2kJqV58wUJwLfPQyRcXDCVV6HZsogIjx+Xm/aNIrkhreXkpKZV2n7fuunLSTuOMBdp3cnKiK00vZrzPEIaCO1qr6mqv1VdQSQBiQDfwQ+dlf5AKeNobTtOKWNEq0oY+wnVX1FVQeq6sD4+PjKCz6Admfk8sq8jZzeu7lzj/vmH2Dj9zBsCoQ38Do8cwRREaG8eEl/MnIKmPTuMoqKj789IiUzjydmrmVoh8ac0dvG3DLVR6DvYmri/m4DnAu8g3OiH+mucjJO0ihtJnCqiDR0G6dPdZfVCk/PWktxMdw2viuownePQIOmMPAKr0Mz5dCteTQPnt2TBRv28VwlTDL06FdJ5BYU8cBZPZ3SpDHVRIXHYqqgj0SkMVAAXK+q+0XkKuA5EQkBcoGrAURkIHCNql6pqmki8iDws7ufB1S1cit9PZK06wAfLNnOlcPaOZ3iNs6BLfNhwuMQFul1eKaczh/Ymp83pfH8t+vp17Yho49xfuiFG/fx8bIdXD+6Ax2bWOnRVC9S1UMIBNLAgQN18eLFXodxRH947SdWbs9g7q2jiakXAv8eD+lb4aZlEBrhdXimAnLyizjnxR/YfSCXaTcNp2UFpwEtKCrmtOfncTCviG/+MpJ6YcEBitSYsonIElUd6O8560ldheasS2Fecio3jelETGQobJgN2xbCiJstOdRA9cKCeenSARQWKde/tZT8wuIKbf+fHzaxbk8W953Zw5KDqZYsQVSRomLlkWlJtG0cyR+GtD3c9hDTGvpd5nV45hi1i6vP4+f1Zvm2dB75qvyTDO3KyOHZb5IZ07UJp3S34dxN9WQJoop8uGQba/dk8tdxbqe4dTOdntMjboUQG2+nJpvYqzlXnNSO1xdsZtrKXeXa5sEvV1NUrNx3Zo8AR2fMsbMEUQWy8wt56ut19GsTy8RezdzSw8PQMAH6Xux1eKYS3D6hK/3axHLbRyvZmJJ1xHXnrEvhq192c8Pojs6NCsZUU5YgqsC/5m5ib2Yed53mdopbMw12r4SRt0GwdYqqDcJCgvjHxf0JDRaue2spOflFftfLLSji3s8SaRdXn6tHtq/iKI2pGEsQAbb3QC7/nLuBib2aMaBtIygudtoeGneEXud7HZ6pRC1i6/Hshf1YuyeTuz71P8nQP+dsZPO+bB44qwfhIdYwbao3SxAB9sw36ygoKuav47o6C5I+g72rYOTtEBzobiimqo3sHM+NJ3fio6XbeX/xrycZ2rLvIP/4fj2n9W7O8E41o9e/qdssQQTQ2t2ZvPfzNv4wJIGEuPpQXOSM2BrfFXqe63V4JkAmjenEsI5x3PPZKlbtdCYZUlXu+3wVoUHC3ad19zhCY8rHEkQAPTo9iQbhIdx4ckdnQeLHkLoWRt0OQVa9UFsFBwnPXtiX2MhQrntrKQdyC5i5ag/frU1hyimdaRZjfV5MzWAJIkDmJafw/doUbji5ozNtZFEhzJkKTXpAt7O8Ds8EWFyDcP5xcX+278/hL++t4IEvVtGlaRR/HJrgdWjGlJsliAAoKlYe+WoNrRrW47ITE5yFv7wP+9bD6DsgyN72umBgQiNuH9+Vb5L2sDMjl4fO6UlosH32puawVtIA+GTZDpJ2HeD5i/oRERoMRQUw5zFo1hu6nu51eKYKXTm8Hdv3ZxNdL5QTEhp5HY4xFWIJopLl5Bfx5My19Gkde3hs/+Vvw/7NcNF7YMM51ykiwv1n9fQ6DGOOiZV3K9lr8zey+0AufyuZKa4wH+Y+AS0HQOdxXodnjDHlZgmiEqVk5vHS9xsY16Mpg9q51QnL3oCMbTD6Tis9GGNqFEsQlejZb9aRV1jszBQHUJALc5+E1oOhwxhvgzPGmAqyNohKkrwnk3d/3salg9vQPt6dGWzpfyFzJ5zzspUejDE1jpUgKsnU6WuIDA3mpjGdnAX52TDvKWg7DNqN8DY4Y4w5BpYgKsGCDanMXrOX60Z3pHGDcGfh4n9D1h5rezDG1FiWII5TcbHyyFdJtIytx59OSnAW5h+E+c9A+1GQcJKH0RljzLGzBHGcPluxg8QdB7h1XBenUxzAolcgOxVG/83b4Iwx5jhYgjgOuQVFPDFjLb1axnBmnxbuwgPww3PQ8RRoPcjbAI0x5jgENEGIyCQRSRSRVSIy2V32nogsd382i8jyMrad4m6XKCLviEi1GwLz3z9sYmdGLndO7EZQkNvOsOifkLPfGXPJGGNqsIDd5ioiPYGrgEFAPjBDRKap6gU+6zwFZPjZtiVwE9BdVXNE5H3gQuD1QMVbUfuy8njxuw2M7daUEzs0dhbmpMOCv0OXiU7PaWOMqcECWYLoBixU1WxVLQTmAOeUPCkiApwPvFPG9iFAPREJASKBnQGMtcKem51MTkERt0/oenjhwpcgN8OZ78EYY2q4QCaIRGCEiDQWkUhgItDa5/nhwB5VTS69oaruAJ4EtgK7gAxV/drfQUTkahFZLCKLU1JSKv1F+LMhJYu3ftrKRYNa07GJ2ykuOw0WvgjdzoDmfaokDmOMCaSAJQhVTQIeA2YBM4AVQKHPKhdRRulBRBoCZwHtgBZAfRG5tIzjvKKqA1V1YHx81czz+9j0NdQLDWby2M6HF/74AuRlwqg7qyQGY4wJtIA2Uqvqa6raX1VHAGlAMoBbbXQu8F4Zm44FNqlqiqoWAB8DQwMZa3n9tHEfX6/ew7WjOhBX0inu4D5Y+DL0OAea2nzDxpjaIdB3MTVxf7fBSQglJYaxwBpV3V7GpluBISIS6bZVjAGSAhlreZR0imseE8EVJ7U7/MQPz0JhDoyyO5eMMbVHoPtBfCQiq4EvgOtVdb+7/EJKVS+JSAsR+QpAVX8CPgSWAr+4cb4S4FiP6ouVO1mxPYNbTu1CvTC3U1zWXlj0L+j1e4jvfOQdGGNMDRLQ0VxVdXgZyy/3s2wnTkN2yeN7gXsDFlwF5RYU8fiMtXRvHs05/VoefmL+M1CUDyNv8y44Y4wJAOtJXU7/XbCZHek5/O00n05xB3bBz69Bn4ugcQdvAzTGmEpmCaIc9h/M54Xv1jO6SzwndYw7/MT8p0GLYOSt3gVnjDEBYgmiHJ7/NpmDeYXcMbHb4YXp22DJ69DvUmiY4FVoxhgTMJYgjmJT6kHe+HELF5zQhs5Now4/Me8pUIXht3gXnDHGBJAliKN4fMYawkKCmHJKp8ML92+GZW/AgD9CbOsytzXGmJrMEsQRLN6cxvTE3VwzsgNNonwGk537BEgwDL/Zu+CMMSbALEGUQVV5aFoSTaPDuXK4T6e4fRtg+Tsw8AqIbuFdgMYYE2CWIMow7ZddLN+Wzs2ndCEyzKe7yJzHITgMhk3xLjhjjKkCliD8yCss4rEZa+jaLIrfDWh1+ImUdfDL+zDoSohq6l2AxhhTBSxB+PHGj1vYlpbDnRO7EVzSKQ5gzlQIqQcnTfYuOGOMqSKWIEpJz87n79+uZ0TneEZ09hk+fM9qSPwYBv8f1I8rewfGGFNLWIIo5YVv15OZW8CdE7v++ok5UyGsAQy90ZvAjDGmilmC8LF1Xzb//XEzvx/Qmq7Nog8/sWslrP4MTrwOIht5Fp8xxlQlSxA+Hpu5hpCgIP5yaqlhu7+fCuExMOQ6bwIzxhgPWIJwLdmyn2krd3HViPY0jfbpFLdjKaydBkNvgHqx3gVojDFVzBIETqe4R75KIq5BOP83ov2vn/z+UajXEAZf401wxhjjEUsQwMxVu1myZT83n9qZ+uE+neK2/QzJX8PQmyAiuuwdGGNMLVTnE0R+YTFTp6+hc9MG/N63UxzAdw9DZBwMutqb4IwxxkMBnXK0JigsLmZcj2ac2KExIcE++XLLAtj4HZz6EIQ38C5AY4zxSJ1PEJFhIb+eCKjEd49Ag6Yw8M9VH5QxxlQDdb6Kya9Nc2HzPBj2FwiL9DoaY4zxREAThIhMEpFEEVklIpPdZe+JyHL3Z7OILC9j21gR+VBE1ohIkoicGMhYD1F1Sg9RLWDA5VVySGOMqY4CVsUkIj2Bq4BBQD4wQ0SmqeoFPus8BWSUsYvngBmqep6IhAFVcym/4VvY+iOc9hSERhx9fWOMqaUCWYLoBixU1WxVLQTmAOeUPCkiApwPvFN6QxGJBkYArwGoar6qpgcwVkdJ6SGmNfT7Q8APZ4wx1VkgE0QiMEJEGotIJDAR8J3AeTiwR1WT/WzbHkgB/iMiy0TkVRGp7+8gInK1iCwWkcUpKSnHF3HyLNixGEbcAiHhx7cvY4yp4QKWIFQ1CXgMmAXMAFYAhT6rXISf0oMrBOgPvKSq/YCDwO1lHOcVVR2oqgPj4+P9rVLegJ1+D7Ftoe8lx74fY4ypJQLaSK2qr6lqf1UdAaQByQAiEgKcC7xXxqbbge2q+pP7+EOchBE4a7+CXcth5G0QHBrQQxljTE0Q6LuYmri/2+AkhJISw1hgjapu97edqu4GtolIF3fRGGB1wAItLnbaHhp1gN4XHH19Y4ypAwLdUe4jEWkMFADXq+p+d/mFlKpeEpEWwKuqOtFddCPwlnsH00bgTwGLMulz2JMI5/4Lgut830FjjAECnCBUdXgZyy/3s2wnTkN2yePlwMCABVeiuMgZsTWuC/T8XcAPZ4wxNYVdLhdkQ6sToONYCAr2OhpjjKk2LEGER8FZL3gdhTHGVDs2FpMxxhi/LEEYY4zxyxKEMcYYvyxBGGOM8csShDHGGL8sQRhjjPHLEoQxxhi/LEEYY4zxVSsYNAAABjhJREFUS1TV6xgqjYikAFuOcfM4ILUSw6ksFlfFWFwVY3FVTG2Mq62q+p0roVYliOMhIotVNfBjP1WQxVUxFlfFWFwVU9fisiomY4wxflmCMMYY45cliMNe8TqAMlhcFWNxVYzFVTF1Ki5rgzDGGOOXlSCMMcb4ZQnCGGOMX3UuQYjIeBFZKyLrReR2P8+Hi8h77vM/ifx/e3cXIlUdxnH8+0u3sla00khca3u7SKN0A7GkkOwiKjRoQylNg256oaSLwuiFui+iF1AqYU0ryzQ2USq1DC98yW1NTQmToCXBsFqzF3P16eL8zel4ZvakM/8zMc8HBs7MeWbOM8/uf58z/5n9j1rrJK/Zkn6U1B0u90XIaYGkfZK2l9kvSS+FnL+S1FbrnHLmNUlSb0mtno6U1yhJn0raKWmHpEcyYqLXLGde0Wsm6UxJmyRtDXk9mxETfTzmzCv6eCw59gBJX0pakbGvuvUys4a5AAOAb4FLgNOBrcDoVMwDwLywPR1YUid5zQZeiVyvG4A2YHuZ/bcAqwABE4CNdZLXJGBFAb9fI4C2sD0Y+Cbj5xi9Zjnzil6zUIPmsN0EbAQmpGKKGI958oo+HkuO/SjwVtbPq9r1arRXEOOB3Wa2x8z+At4BpqZipgIdYXspMFmS6iCv6Mzsc+CnCiFTgYWW2AAMlTSiDvIqhJntNbOusP0rsBMYmQqLXrOceUUXanAwXG0Kl/SnZqKPx5x5FUJSC3Ar8HqZkKrWq9EaxEjg+5LrPZw4UP6JMbM+oBc4rw7yArgjTEsslTSqxjnlkTfvIlwbpghWSRoT++Dhpf04krPPUoXWrEJeUEDNwnRJN7AP+MTMytYr4njMkxcUMx5fBB4DjpbZX9V6NVqDyOqk6TODPDHVlueYHwKtZnYVsJrjZwlFKqJWeXSRrC9zNfAy8EHMg0tqBt4H5pjZgfTujLtEqVk/eRVSMzM7YmZjgRZgvKQrUyGF1CtHXtHHo6TbgH1mtqVSWMZtJ12vRmsQPUBpp28BfigXI2kgMITaT2f0m5eZ7TezQ+Hqa8A1Nc4pjzz1jM7MDhybIjCzlUCTpGExji2pieSP8GIzW5YRUkjN+suryJqFY/4CfAbcnNpVxHjsN6+CxuNEYIqk70imoW+UtCgVU9V6NVqD2AxcLuliSaeTvInTmYrpBGaF7XZgrYV3fIrMKzVPPYVkHrloncA94ZM5E4BeM9tbdFKSLjg27yppPMnv+f4IxxXwBrDTzF4oExa9ZnnyKqJmkoZLGhq2BwE3AbtSYdHHY568ihiPZjbXzFrMrJXkb8RaM5uRCqtqvQae7B3/j8ysT9JDwEcknxxaYGY7JD0HfGFmnSQD6U1Ju0k67/Q6yethSVOAvpDX7FrnJeltkk+3DJPUAzxD8oYdZjYPWEnyqZzdwO/AvbXOKWde7cD9kvqAP4DpEZo8JGd4M4FtYf4a4AngwpLciqhZnryKqNkIoEPSAJKG9K6ZrSh6PObMK/p4LKeW9fKlNpxzzmVqtCkm55xzOXmDcM45l8kbhHPOuUzeIJxzzmXyBuGccy6TNwjn6oCS1VRPWJ3TuSJ5g3DOOZfJG4Rz/4GkGeG7ArolzQ+Luh2U9LykLklrJA0PsWMlbQgLui2XdE64/TJJq8PCeF2SLg0P3xwWftslaXGEVYSdq8gbhHM5SboCmAZMDAu5HQHuBs4GusysDVhH8p/dAAuBx8OCbttKbl8MvBoWxrsOOLbUxjhgDjCa5LtBJtb8STlXQUMtteHcKZpMsijb5nByP4hkOeijwJIQswhYJmkIMNTM1oXbO4D3JA0GRprZcgAz+xMgPN4mM+sJ17uBVmB97Z+Wc9m8QTiXn4AOM5v7rxulp1JxldavqTRtdKhk+wg+Pl3BfIrJufzWAO2SzgeQdK6ki0jGUXuIuQtYb2a9wM+Srg+3zwTWhe9h6JF0e3iMMySdFfVZOJeTn6E4l5OZfS3pSeBjSacBh4EHgd+AMZK2kHyD17Rwl1nAvNAA9nB85daZwPywCudh4M6IT8O53Hw1V+dOkaSDZtZcdB7OVZtPMTnnnMvkryCcc85l8lcQzjnnMnmDcM45l8kbhHPOuUzeIJxzzmXyBuGccy7T30EL/C36OzmDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best configuration is: \n",
      "{'lr': 0.028245913732173278, 'net▁hidden_conv': 11, 'net▁hidden_fc▁choice': 0, 'wd': 0.00017160776862349322}\n",
      "Validation accuracy achieved under this configuration = 98.87\n"
     ]
    }
   ],
   "source": [
    "myscheduler.get_training_curves(plot=True,use_legend=False)\n",
    "best_config = myscheduler.get_best_config()\n",
    "best_acc = myscheduler.get_best_reward()\n",
    "print('The best configuration is: \\n{}'.format(best_config))\n",
    "print('Validation accuracy achieved under this configuration =', best_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameter search via Asynchronous BOHB \n",
    "\n",
    "When training neural networks, it is often more efficient to early-terminate unpromising trials, and Hyperband scheduling can save a lot of wall-clock time. However, Hyperband relies entirely on randomly sampled configurations. As it completes more training jobs under different hyperparameter configurations, a good AutoML system should rather learn which sorts of configurations tend to work well and hone in on these promising regions of the search space (as done in Bayesian optimization). [BOHB](https://arxiv.org/abs/1807.01774) is a recent hyperparameter-search strategy that combines the early-termination strengths of Hyberband with the better-than-random configuration proposals of Bayesian optimization. \n",
    "\n",
    "AutoGluon provides an improved [asynchronous BOHB implementation](https://arxiv.org/abs/2003.10865) that can also be used to tune your custom model like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "max_t = 5, as inferred from train_fn.args\n",
      "scheduler_options: Key 'resume': Imputing default value False\n",
      "scheduler_options: Key 'grace_period': Imputing default value 1\n",
      "scheduler_options: Key 'reduction_factor': Imputing default value 3\n",
      "scheduler_options: Key 'brackets': Imputing default value 1\n",
      "scheduler_options: Key 'type': Imputing default value stopping\n",
      "scheduler_options: Key 'keep_size_ratios': Imputing default value False\n",
      "scheduler_options: Key 'maxt_pending': Imputing default value False\n",
      "scheduler_options: Key 'searcher_data': Imputing default value rungs\n",
      "scheduler_options: Key 'do_snapshots': Imputing default value False\n",
      "\n",
      "scheduler_options: Key 'visualizer': Imputing default value none\n",
      "scheduler_options: Key 'training_history_callback_delta_secs': Imputing default value 60\n",
      "scheduler_options: Key 'delay_get_config': Imputing default value True\n",
      "\n",
      "search_options: Key 'random_seed': Imputing default value 31415927\n",
      "search_options: Key 'opt_skip_init_length': Imputing default value 150\n",
      "search_options: Key 'opt_skip_period': Imputing default value 1\n",
      "search_options: Key 'profiler': Imputing default value False\n",
      "search_options: Key 'opt_maxiter': Imputing default value 50\n",
      "search_options: Key 'opt_nstarts': Imputing default value 2\n",
      "search_options: Key 'opt_warmstart': Imputing default value False\n",
      "search_options: Key 'opt_verbose': Imputing default value False\n",
      "search_options: Key 'opt_debug_writer': Imputing default value False\n",
      "search_options: Key 'num_fantasy_samples': Imputing default value 20\n",
      "search_options: Key 'num_init_random': Imputing default value 10\n",
      "search_options: Key 'num_init_candidates': Imputing default value 250\n",
      "search_options: Key 'initial_scoring': Imputing default value thompson_indep\n",
      "search_options: Key 'first_is_default': Imputing default value True\n",
      "search_options: Key 'debug_log': Imputing default value False\n",
      "search_options: Key 'opt_skip_num_max_resource': Imputing default value False\n",
      "search_options: Key 'gp_resource_kernel': Imputing default value matern52\n",
      "search_options: Key 'resource_acq': Imputing default value bohb\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HyperbandScheduler(terminator: HyperbandStopping_Manager(reward_attr: accuracy, time_attr: epoch, reduction_factor: 3, max_t: 5, brackets: [Bracket: Iter 3.000: None | Iter 1.000: None])\n"
     ]
    }
   ],
   "source": [
    "bohb_scheduler = ag.scheduler.HyperbandScheduler(\n",
    "    ag_train_mnist,\n",
    "    resource={'num_cpus': 4, 'num_gpus': 1},\n",
    "    searcher='bayesopt',\n",
    "    num_trials=4, # small value used here for quick demo, you should increase this to see more realistic performance.\n",
    "    time_attr='epoch',\n",
    "    reward_attr='accuracy')\n",
    "print(bohb_scheduler)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Starting Experiments\n",
      "Num of Finished Tasks is 0\n",
      "Num of Pending Tasks is 4\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0804edb1a946478b953d0c6ecc58a6c1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=4.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lr: 0.0447213595, wd: 0.0002236068\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d84e807faa0445ff9843800c3f6c7dfc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=5.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lr: 0.028245913732173278, wd: 0.00017160776862349322\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2c8f4abbad3047afb530551b4f7448b8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=5.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lr: 0.18869688885602873, wd: 0.00023865057332689288\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7ab8f10ed76c4eb7aa276b4fd2afdbf4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=5.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "lr: 0.14193763233841836, wd: 0.00037124134442677336\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "28a7ec516b4e4db48f90051162c79e42",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=5.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAdo0lEQVR4nO3de7gcVZnv8e9v73CVS8AEiAQTxCiCygARUeY4aHAEFYKPqCgo8TKMjhdQR0UHRefMzGHmjOL1gBxQozDcURAF5a6OEg0XFQhKRIFAIAEJN28k+50/1uqV2p3und6b7q6dvX+f59lJXVZVvb2qq96qVdVVigjMzMwABuoOwMzMxg8nBTMzK5wUzMyscFIwM7PCScHMzAonBTMzK5wU+kjSv0h6QNJ9dccyWUkalPSYpKd3s2y3SfqapA/3YTkHSPp5r5fTKUmbSgpJM+uOpZsk/UDSazss+4CkfXsdUztOCiOQ9DtJf8w7hvslfVXSFmOc107AB4HdImKH7kY6ceW6b/wNVdbHY5KOGO38ImJNRGwREXd1s+xoSHpW0+cKSY9X+l8QEQsi4j+6udxWIuKKiNij18vpFUknSjqtadiwxJLLPJHrdpWkH0l6QaX8LElnS/p9LnOdpFe0Wd4BlfX0eF5OdV1u12q6iHhJRFzQzc/eK04K63dwRGwB7AW8ADh+tDOQNAWYBTwYESvGOP2klHfKW+R1cBd5feS/M5vLbwh1FRG/rnymaXnwsyuf62d1xjdBLcz1vR1wPXAeQN6J/zewCtg1jz8ZOF/Swc0zyUm0se72BtZUv6PN27ekAUnq6SfrMieFDkXEPcClwHMBJG0t6XRJyyXdk5uGBvO4BZL+W9JJkn4PXANcDjwtH018LZc7RNIt+ejlGknPaSwvn6V8RNIvgMclTcnDPiTpF/ko5XRJ20u6VNKjkq6QtE1lHudJuk/Sw/n0dffKuK9J+pKk7+RpF0napTJ+d0mX56On+yV9LA8fkHScpN9IelDSuZK2bVVnkpZIenWlf0o+Nd4rH82dkeexStLPJG0/2vWS6/0cSWdJehQ4UtKL8tHeqrx+Pi9po0oMIWl27j8jj2/U4U8k7Tzasnn8QZJ+nev7C/k7sGC0nynP62xJx+fuAyUtlXR8rr97JL1S0vzKevhgZdpBSR+XdEcuf6akqW2Wc6CkpZX++yS9X9LN+XOcKWnjEeL8e0m/yt+T70jasTLuZEnLJD0i6aeqNInkuj0hx/hIXv/VM+iD8md7SNJJY6nDViLiL8DXgVlKZ/0fAu6PiHdGxIqI+ENELAT+E/j0WJYhabGkT0r6KfAHYPs87PA8fre8Pf5e0go9iRaIXnBS6JBS888rgRvzoIXAauCZwJ7A3wLvqEzyQuAO0pHHy4GDgHvz0cQCSc8CzgKOBaYD3wW+3bQBvhF4FTA1IlbnYa/N83sWcDApUX2MdMQ5ALyvMv2lwJwcww1A85H1G4FPAdsAS4F/zZ91S+AK4DLgafkzXpmneR9wKPA3edxDwJfaVNtZeRkNrwAeiIgbgKOArYGdgKcC7wT+2GY+6/Ma4L/y/M4hrZdjSHWyH3Ag8PcjTP8m4OPAtqSzkf892rJKR5znknYy04DfAvuM8fO0Mgt4AtgBOBH4CnAY8HzgAOBfKzvkD5G+j38NzMzTjWbHehgwj7TeX0j6zOvIO7ljSd/D7UnbxhmVIj8BnkdavxcB5zWSM/BR0vfob4GpwNHAnyrTHkTarvYC3ipp/1HE35akTUnfvaUR8RhpWzq/RdFzgTmSZo1xUUcCRwBbAStbjP8Eqc72AHYHjhvjcrovIvzX5g/4HfAY6dTyTuD/AZuRVuafgc0qZd8IXJ27FwB3Nc1rf2BZpf/jwLmV/gHgHmD/yrLf1iKeIyr9FwAnV/rfC3yrzWeZCgSwde7/GnBaZfwrgdsqn+XGNvNZAsyr9M8g7XSmtCj7TOBRYPPcfybwidz9NuDHwPNHuT4OaBr2L8BV65nuH4HzcveUXA+zc/8ZwCmVsocAN4+h7NuAH1bGCVgOLFhPbJvmZcxsGn42cHzuPhB4GBjI/dPzNHtUyt8CHJi7fwvsVxm3M+mIVS2WfyBpB9novw84rNL/eeCzbWK/uun7uFH+LmzfoqxyDM/O/XcCrxihPuZWhl0MHNsmhhNJ2+Kqpr9Sp01lVpDO2p+fxy1rtY5Yu73sPcK62xVY3WL4YuDDLYYd3mY+RzZ9dx4A9u10u+j237hvfx0HDo2IK6oDJD2PtAEs19rmwgHg7kqxancrTyNtGABExJCku4EdK2VazeP+SvcfW/RvkWMcJB35v460ExnKZaaRdjCQdgANf2hMSzp6/02buGcB35Q0VBm2hpQo76kWjIilkpYAB0v6Nmknumce/Y28nLNz08YZwD9FxBNtljuSYfUkaVfSqf/ewOaknfuiEaZvVw+jKfu0ahwREZKWVWL6FWvX7csj4icjLKOVlRHRqPPGGdU6617pC7kT8F1J1addDpCO2B/oYFnNn3Fam3KzgFMkVc8UV5POTu6X9FHSAdIOpB3spsA0Sb8m1UW771irGEZaJ9+IiHKWns8Gms86h5WpeIB0YNNsRmX8WLTd/pUugH8WeDHpczXvO2rl5qOxuZt05DEtIqbmv60iYvdKmfU9fvZe0kYFQGVjru5Yn8wjbN8EzCc1LWwNzG4sqoNp7wZ2GWHcQZXPPTUiNo10zaWVRhPSfODWiFgKEBFPRMSnImI30sbxauAtHcTWSnM9fRm4GXhmRGxFOlXv9cW+5aSdIVDWZ0nwEVG9kDzahNCxSIea9wAva7GOxrqDa+du0lF2dTmbRcT1kl5OOnN9Demoe1vSjlqVGNt9x/rpClJzWbPXA7+JiDtbjOvESNvup0kHZrvl7+c76f33s2NOCmMQEcuB7wOflrSV0sXXXST9zShmcy7wKknzcjvrB0mJ5sddCnPLPL8HSUfL/zaKaS8BdpB0rKRNJG0p6YV53Cmk9utZAJKmS5o/wrzOJrUbv4vU7k+e7qWSnpfPaB4hNTusGUWMI9mStNE9rnTxfqTrCd1yCbCXpIOV7oA6hnSGVodTgBPzdTAkbacWd9J0aTnHS3p2Xs42Wnsv/pakdboS2Bj4Z9KZQsNpwL9JeoaSPdtdDO+x/wvMkHRKrqfNJL2FdF3mH3u0zC1JzaqPKN3E8P4eLWdMnBTG7i2kL/utpIut59P6NLSliPgVqS3xC6RT1INJt1v+pUvxfZ3UPHVPjvG6UcT2KOkC3MGk0/jbgZfm0Z8jtfF+X+lun+tIFyPbzWs56YLji0kXgRt2INXZI6TrFNcy/CLlk/FB0sXER0lnDeeMXPzJi4j7gTcAnyEl4l1IF17/3Otlt/AfpCPgq/I6+jHpgm1XRcRZwBeBCyU9AtxE+t4AfBv4AamJ6A7Sd7x6wfVE4DvAVaTvwCnAJt2OcX3yetuPdCbzK1Kc7wZeHxHf6tFijydtT4+QtoHzerScMVG+sGFmXZTPgO4lXbT9Yd3xmHXKZwpmXZLv+d9a0iaku8tWAz+tOSyzUXFSMOuev2ZtU8mBpDvX6mg+MhszNx+ZmVnhMwUzMys26B+vTZs2LWbPnl13GGZmG5Trr7/+gYhoecv0Bp0UZs+ezeLFi+sOw8xsgyKp7Y/y3HxkZmaFk4KZmRU9SwqSvpKfFX5zZdi2Ss/ovz3/v00eLqXn1C9VeldA1399aWZm69fLM4Wvke7VrjoOuDIi5pCez994hvhBpOf+zyE9V/3kHsZlZmZt9CwpRMQPgN83DZ5PejkN+f9DK8O/Hsl1wFRJHT9HyMzMuqPf1xS2zw9IazworfGS6x0Z/jzxZQx/r0Ah6ej8arvFK1e2eqGRmZmN1Xi50NzqWeItf2odEadGxNyImDt9el1PJjYzm5j6/TuF+yXNiIjluXloRR6+jPSCmYaZpCdMmlnNhoaCNRGsGQqGGv8Psc6wYeMjWDNE6V7dPL5pnkND6SgwIvL/jaVH6a4Oj3WGrz2GjEjjS3eLclH+WTuvaDd9ZcbDyjT3l+4RYmkaXn3KUIltfbHkcvOesz177NT9V1D0OylcTHrO/Yn5/4sqw98j6WzSs/kfbjQzTTbVjWX1ULBmTepfPTRUNrwyrk1/J2WHyvChpmmr43J/jqehcVrXeBOppLWneo1huaOUaZ6mzfjGAHVQtrr8qlbTrW/5a6ddu/zWy9SIMVenGWrUazTtWHP32mGkHeeayg62MX5YWVoMG+3OmmHfhXXmU4atnY+NPxJst9WmG1ZSkHQW6WX10/K7ak8gJYNzJb0duIv0/mCA75JeHL+U9D7Wt/YqLoCf372KRb99cNhOt/XOcygf7QyNuBNulB0aouyQm8evyRt9dUe7es3Q8P6h4UcOdRscEIMSgwNiQGmHOOxIi3WPZKrjqkdirca3mtdkU63jRj2v7R7+f3X88GEq8xkYgI0GBpqGVdbjgBgUDA4MMDjACMtRGl+ZfmCgeZ5U5tlqOWkeAxJTBpvjbP6MqVxDSuSVpF5J5tWkvHYSNSV/DUvSrQ4CpOEHAessc+2s2xxgaN2DF9G2XLtY2n+2FgdIzUcxPdCzpBARb2wzal6LskF621FfXHfHg/yfS28bNqy6MU4ZGCjdqV/r9De+6IMDA2l8/nJvstGUNK5RNm8MqT+VHWia55TqfAfWlk0b7UDL5Q/vHrlsI9ZSVmJwsDKfSn819n58AdupnkrD+hNJc+KhMn7USWmEZLa+5Vf/a95Zr7sDHzev5TUrNuhnH43Vgv1mc+S+s9buPL2BjjvlKGmd1eL1ZNZLkzIpbDJlkE0m5Sc3MxvZeLkl1czMxgEnBTMzK5wUzMyscFIwM7PCScHMzAonBTMzK5wUzMyscFIwM7PCScHMzAonBTMzK5wUzMyscFIwM7PCScHMzAonBTMzK5wUzMyscFIwM7PCScHMzAonBTMzK5wUzMyscFIwM7PCScHMzAonBTMzK5wUzMyscFIwM7PCScHMzAonBTMzK5wUzMyscFIwM7PCScHMzAonBTMzK5wUzMyscFIwM7PCScHMzAonBTMzK2pJCpLeL+kWSTdLOkvSppJ2lrRI0u2SzpG0cR2xmZlNZn1PCpJ2BN4HzI2I5wKDwOHAvwMnRcQc4CHg7f2Ozcxssqur+WgKsJmkKcDmwHLgZcD5efxC4NCaYjMzm7T6nhQi4h7gP4G7SMngYeB6YFVErM7FlgE79js2M7PJro7mo22A+cDOwNOApwAHtSgabaY/WtJiSYtXrlzZu0DNzCahOpqPDgB+GxErI+IJ4ELgxcDU3JwEMBO4t9XEEXFqRMyNiLnTp0/vT8RmZpNEHUnhLmBfSZtLEjAPuBW4GjgslzkKuKiG2MzMJrU6riksIl1QvgH4ZY7hVOAjwAckLQWeCpze79jMzCa7Kesv0n0RcQJwQtPgO4B9agjHzMwy/6LZzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrOgoKUi6QNKrJDmJmJlNYJ3u5E8G3gTcLulESbv2MCYzM6tJR0khIq6IiCOAvYDfAZdL+rGkt0raqJcBmplZ/3TcHCTpqcAC4B3AjcDnSEni8p5EZmZmfTelk0KSLgR2Bb4BHBwRy/OocyQt7lVwZmbWXx0lBeCLEXFVqxERMbeL8ZiZWY06bT56jqSpjR5J20j6h7EuVNJUSedLuk3SEkkvkrStpMsl3Z7/32as8zczs7HpNCn8XUSsavRExEPA3z2J5X4OuCwidgX2AJYAxwFXRsQc4Mrcb2ZmfdRpUhiQpEaPpEFg47EsUNJWwEuA0wEi4i854cwHFuZiC4FDxzJ/MzMbu06TwveAcyXNk/Qy4CzgsjEu8xnASuCrkm6UdJqkpwDbNy5g5/+3azWxpKMlLZa0eOXKlWMMwczMWuk0KXwEuAp4F/BuUvPOh8e4zCmkW1lPjog9gccZRVNRRJwaEXMjYu706dPHGIKZmbXS0d1HETFE+lXzyV1Y5jJgWUQsyv3nk5LC/ZJmRMRySTOAFV1YlpmZjUKnzz6ak+8WulXSHY2/sSwwIu4D7pb07DxoHnArcDFwVB52FHDRWOZvZmZj1+nvFL4KnACcBLwUeCugEacY2XuBMyVtDNyR5zdAum7xduAu4HVPYv5mZjYGnSaFzSLiSkmKiDuBT0r6ISlRjFpE3AS0+tHbvLHMz8zMuqPTpPCn/Njs2yW9B7iHNncHmZnZhqvTu4+OBTYH3gfsDRzJ2vZ/MzObINZ7ppB/qPb6iPgQ8Bip/d/MzCag9Z4pRMQaYO/qL5rNzGxi6vSawo3ARZLOI/3YDICIuLAnUZmZWS06TQrbAg8CL6sMC8BJwcxsAun0F82+jmBmNgl0+ua1r5LODIaJiLd1PSIzM6tNp81Hl1S6NwVeA9zb/XDMzKxOnTYfXVDtl3QWcEVPIjIzs9p0+uO1ZnOAp3czEDMzq1+n1xQeZfg1hftI71gwM7MJpNPmoy17HYiZmdWv0/cpvEbS1pX+qZL8DmUzswmm02sKJ0TEw42eiFjFGB+bbWZm41enSaFVuU5vZzUzsw1Ep0lhsaTPSNpF0jMknQRc38vAzMys/zpNCu8F/gKcA5wL/BF4d6+CMjOzenR699HjwHE9jsXMzGrW6d1Hl0uaWunfRtL3eheWmZnVodPmo2n5jiMAIuIh/I5mM7MJp9OkMCSpPNZC0mxaPDXVzMw2bJ3eVvpPwI8kXZv7XwIc3ZuQzMysLp1eaL5M0lxSIrgJuIh0B5KZmU0gnT4Q7x3AMcBMUlLYF/gJw1/PaWZmG7hOrykcA7wAuDMiXgrsCazsWVRmZlaLTpPCnyLiTwCSNomI24Bn9y4sMzOrQ6cXmpfl3yl8C7hc0kP4dZxmZhNOpxeaX5M7PynpamBr4LKeRWVmZrUY9ZNOI+La9ZcyM7MN0Vjf0WxmZhOQk4KZmRVOCmZmVjgpmJlZ4aRgZmaFk4KZmRW1JQVJg5JulHRJ7t9Z0iJJt0s6R9LGdcVmZjZZ1XmmcAywpNL/78BJETEHeAh4ey1RmZlNYrUkBUkzgVcBp+V+kZ64en4ushA4tI7YzMwms7rOFD4LfBgYyv1PBVZFxOrcvwzYsdWEko6WtFjS4pUr/aBWM7Nu6ntSkPRqYEVEXF8d3KJoy9d9RsSpETE3IuZOnz69JzGamU1Wo372URfsBxwi6ZXApsBWpDOHqZKm5LOFmfgprGZmfdf3M4WI+GhEzIyI2cDhwFURcQRwNXBYLnYU6ZWfZmbWR+PpdwofAT4gaSnpGsPpNcdjZjbp1NF8VETENcA1ufsOYJ864zEzm+zG05mCmZnVzEnBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrOh7UpC0k6SrJS2RdIukY/LwbSVdLun2/P82/Y7NzGyyq+NMYTXwwYh4DrAv8G5JuwHHAVdGxBzgytxvZmZ91PekEBHLI+KG3P0osATYEZgPLMzFFgKH9js2M7PJrtZrCpJmA3sCi4DtI2I5pMQBbNdmmqMlLZa0eOXKlf0K1cxsUqgtKUjaArgAODYiHul0uog4NSLmRsTc6dOn9y5AM7NJqJakIGkjUkI4MyIuzIPvlzQjj58BrKgjNjOzyayOu48EnA4siYjPVEZdDByVu48CLup3bGZmk92UGpa5H/Bm4JeSbsrDPgacCJwr6e3AXcDraojNzGxS63tSiIgfAWozel4/YzEzs+H8i2YzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKJwUzMyucFMzMrHBSMDOzwknBzMwKRUTdMYyZpJXAnWOcfBrwQBfD6RbHNTqOa/TGa2yOa3SeTFyzImJ6qxEbdFJ4MiQtjoi5dcfRzHGNjuMavfEam+ManV7F5eYjMzMrnBTMzKyYzEnh1LoDaMNxjY7jGr3xGpvjGp2exDVprymYmdm6JvOZgpmZNXFSMDOzYsInBUkHSvqVpKWSjmsxfhNJ5+TxiyTNHidxLZC0UtJN+e8dfYrrK5JWSLq5zXhJ+nyO+xeS9honce0v6eFKfX2iDzHtJOlqSUsk3SLpmBZl+l5fHcZVR31tKumnkn6e4/pUizJ93x47jKuW7TEve1DSjZIuaTGu+/UVERP2DxgEfgM8A9gY+DmwW1OZfwBOyd2HA+eMk7gWAF+soc5eAuwF3Nxm/CuBSwEB+wKLxklc+wOX9LmuZgB75e4tgV+3WI99r68O46qjvgRskbs3AhYB+zaVqWN77CSuWrbHvOwPAP/Van31or4m+pnCPsDSiLgjIv4CnA3MbyozH1iYu88H5knSOIirFhHxA+D3IxSZD3w9kuuAqZJmjIO4+i4ilkfEDbn7UWAJsGNTsb7XV4dx9V2ug8dy70b5r/lOl75vjx3GVQtJM4FXAae1KdL1+proSWFH4O5K/zLW3ThKmYhYDTwMPHUcxAXw2tzkcL6knXocU6c6jb0OL8pNAJdK2r2fC86n7XuSjjKraq2vEeKCGuorN4XcBKwALo+ItvXVx+2xk7ignu3xs8CHgaE247teXxM9KbTKmM1HAJ2U6bZOlvltYHZEPB+4grVHA3Wro746cQPpeS57AF8AvtWvBUvaArgAODYiHmke3WKSvtTXeuKqpb4iYk1E/BUwE9hH0nObitRSXx3E1fftUdKrgRURcf1IxVoMe1L1NdGTwjKgmtFnAve2KyNpCrA1vW+mWG9cEfFgRPw59/5/YO8ex9SpTuq07yLikUYTQER8F9hI0rReL1fSRqQd75kRcWGLIrXU1/riqqu+KstfBVwDHNg0qo7tcb1x1bQ97gccIul3pCbml0k6o6lM1+troieFnwFzJO0saWPShZiLm8pcDByVuw8Drop81abOuJranQ8htQuPBxcDb8l31ewLPBwRy+sOStIOjbZUSfuQvtsP9niZAk4HlkTEZ9oU63t9dRJXTfU1XdLU3L0ZcABwW1Oxvm+PncRVx/YYER+NiJkRMZu0j7gqIo5sKtb1+pryZCYe7yJitaT3AN8j3fHzlYi4RdI/A4sj4mLSxvMNSUtJGfbwcRLX+yQdAqzOcS3odVwAks4i3ZkyTdIy4ATShTci4hTgu6Q7apYCfwDeOk7iOgx4l6TVwB+Bw/uQ3PcD3gz8MrdHA3wMeHolrjrqq5O46qivGcBCSYOkJHRuRFxS9/bYYVy1bI+t9Lq+/JgLMzMrJnrzkZmZjYKTgpmZFU4KZmZWOCmYmVnhpGBmZoWTgllNlJ5Uus6TL83q5KRgZmaFk4LZekg6Mj9v/yZJX84PT3tM0qcl3SDpSknTc9m/knRdfnDaNyVtk4c/U9IV+QF0N0jaJc9+i/yAtdskndmHJ/SajchJwWwEkp4DvAHYLz8wbQ1wBPAU4IaI2Au4lvQLa4CvAx/JD077ZWX4mcCX8gPoXgw0HnWxJ3AssBvp/Rr79fxDmY1gQj/mwqwL5pEefvazfBC/GenxykPAObnMGcCFkrYGpkbEtXn4QuA8SVsCO0bENwEi4k8AeX4/jYhluf8mYDbwo95/LLPWnBTMRiZgYUR8dNhA6eNN5UZ6XsxITUJ/rnSvwduk1czNR2YjuxI4TNJ2AJK2lTSLtO0clsu8CfhRRDwMPCTpf+Xhbwauze8yWCbp0DyPTSRt3tdPYdYhH5WYjSAibpV0PPB9SQPAE8C7gceB3SVdT3rb1RvyJEcBp+Sd/h2sfSrqm4Ev5ydcPgG8ro8fw6xjfkqq2RhIeiwitqg7DrNuc/ORmZkVPlMwM7PCZwpmZlY4KZiZWeGkYGZmhZOCmZkVTgpmZlb8D8EQcXBkgX4vAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The best configuration is: \n",
      "{'lr': 0.028245913732173278, 'net▁hidden_conv': 11, 'net▁hidden_fc▁choice': 0, 'wd': 0.00017160776862349322}\n",
      "Validation accuracy achieved under this configuration = 98.87\n"
     ]
    }
   ],
   "source": [
    "bohb_scheduler.run()\n",
    "bohb_scheduler.join_jobs()\n",
    "\n",
    "bohb_scheduler.get_training_curves(plot=True,use_legend=False)\n",
    "best_config_bohb = bohb_scheduler.get_best_config()\n",
    "best_acc_bohb = bohb_scheduler.get_best_reward()\n",
    "print('The best configuration is: \\n{}'.format(best_config))\n",
    "print('Validation accuracy achieved under this configuration =', best_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "[**AutoGluon Documentation** (autogluon.mxnet.io)](https://autogluon.mxnet.io/api/autogluon.task.html)\n",
    "\n",
    "Bose A. [**Handwritten Digit Recognition Using PyTorch**](https://towardsdatascience.com/handwritten-digit-mnist-pytorch-977b5338e627). *Towards Data Science*, 2019.\n",
    "\n",
    "Falkner et al. [**BOHB: Robust and efficient hyperparameter optimization at scale**](http://proceedings.mlr.press/v80/falkner18a). In: *ICML*, 2018.\n",
    "\n",
    "Klein et al. [**Model-based Asynchronous Hyperparameter and Neural Architecture Search**](https://arxiv.org/abs/2003.10865). *Arxiv*, 2020."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "00941cf88c6540069e6e12a39813ba4a": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "03596f38e5b94ce38bcec48c041ac2b6": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_81dc697f0bed42799cb7a1af4f41fe59",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_242deb2579924cf185a972ba8b3baf85",
       "value": 1
      }
     },
     "036ad6426ec44afaa43002a046351c55": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_95e6604946e84d47a05ad4cb95567424",
       "max": 2,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_9b9620f2a1d741e7b738c514571e41bc",
       "value": 2
      }
     },
     "04134add6b3842c4897434a65316cd6a": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "0c2a0ef39f0c4c5ba03eabbb5b2ea86f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "13fcf9528f9548bb8caa660385be9120": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "1d20e314643d48c1a1a3efc366e641a9": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "20ad0bc1503148129ed9e5622f257f2e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "242deb2579924cf185a972ba8b3baf85": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "2586115495b14dc493a9f546bc579995": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "29d76a6c4bd6496081a4ab6aabd342c0": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_036ad6426ec44afaa43002a046351c55",
        "IPY_MODEL_ce3545b63e2841f2b960ca99e7ff61f9"
       ],
       "layout": "IPY_MODEL_d571e4ee37b141bb9af8252df746fc82"
      }
     },
     "309886d3f1c348a899782259fb4dc99b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_3f58792c2787450a974e2b8f7ef77ebf",
        "IPY_MODEL_e5bb3c6fbab5456bbf4f224f683610a1"
       ],
       "layout": "IPY_MODEL_eef08f88c6bf4501ab5770a6c85ef423"
      }
     },
     "3ee7e11549ca4ed498c4e97e9a8f8c53": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "3f58792c2787450a974e2b8f7ef77ebf": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_df4b68e73d534e70b28baaa0c4dd8143",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_651900fc0448469bafb8dddea6970266",
       "value": 1
      }
     },
     "4ed1aa4a55e74526865467e517a86166": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "50ba05f0dd2b4428a3316810c9e4b823": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_86fa39d7f6e1408da221f74ccc920e4a",
        "IPY_MODEL_7059db0fde1a48afb39b1993ce75052b"
       ],
       "layout": "IPY_MODEL_3ee7e11549ca4ed498c4e97e9a8f8c53"
      }
     },
     "5353f30aeead41ea9f3635e8c51143c7": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "54354771feea42d1b536c4e74fb103cc": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_03596f38e5b94ce38bcec48c041ac2b6",
        "IPY_MODEL_6bfb24ccc9fd4a7cb0ef7e0ade58678d"
       ],
       "layout": "IPY_MODEL_8b9a6b909667420db4e7d6ef2445bd98"
      }
     },
     "5daa0c094a2542f0a21c4c3a4bd5026e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_66e91fa313e04584a9ed447b8e3ab5de",
       "max": 5,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_4ed1aa4a55e74526865467e517a86166",
       "value": 5
      }
     },
     "651900fc0448469bafb8dddea6970266": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "66e91fa313e04584a9ed447b8e3ab5de": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "6bfb24ccc9fd4a7cb0ef7e0ade58678d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_bb466454cea741149b88222e74bdc317",
       "placeholder": "​",
       "style": "IPY_MODEL_e288ee57c64d4c068d7d0ae64d990c17",
       "value": " 1654784/? [00:00&lt;00:00, 4253383.63it/s]"
      }
     },
     "6d43bbb8f2994ea5acea8cf467e5f317": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "7059db0fde1a48afb39b1993ce75052b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_bea51f1599004332836249602d08fb63",
       "placeholder": "​",
       "style": "IPY_MODEL_d8adedc4352743b6a551d8369cc80327",
       "value": " 5/5 [01:36&lt;00:00, 19.38s/it]"
      }
     },
     "7943b2de9cf041beabe152d891433480": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "7a9e49a68e444ed6948c6b4fb19bb14b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_efa1302e547f40c1b3d3b82b8b132d3d",
        "IPY_MODEL_d70e77a3ae9440da925717e07cebaa36"
       ],
       "layout": "IPY_MODEL_5353f30aeead41ea9f3635e8c51143c7"
      }
     },
     "7c083fd862d74e888d8473ce33426846": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "81dc697f0bed42799cb7a1af4f41fe59": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "86fa39d7f6e1408da221f74ccc920e4a": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "100%",
       "description_tooltip": null,
       "layout": "IPY_MODEL_ea58e23e71074f929cb4b902ab2ebbf4",
       "max": 5,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_97e1b2bf23814db4b57b20b2e3b52567",
       "value": 5
      }
     },
     "89e19c7b67f4426087e74d1527d8cda8": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "8b9a6b909667420db4e7d6ef2445bd98": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "8cd97237ca61436189b57cc5e6856d94": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "9182fb7c7c42405f9d6aae7a3ea7b2df": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "926a2265f096480a9c646d129bb12e33": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_9182fb7c7c42405f9d6aae7a3ea7b2df",
       "placeholder": "​",
       "style": "IPY_MODEL_20ad0bc1503148129ed9e5622f257f2e",
       "value": " 5/5 [01:31&lt;00:00, 18.28s/it]"
      }
     },
     "95e6604946e84d47a05ad4cb95567424": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "97e1b2bf23814db4b57b20b2e3b52567": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "9a2ea1ca84f147d3bc07d3ccbee77217": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "9b9620f2a1d741e7b738c514571e41bc": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "bb466454cea741149b88222e74bdc317": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "bc49a547839948ca825be34a40b03ec3": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_5daa0c094a2542f0a21c4c3a4bd5026e",
        "IPY_MODEL_926a2265f096480a9c646d129bb12e33"
       ],
       "layout": "IPY_MODEL_8cd97237ca61436189b57cc5e6856d94"
      }
     },
     "bea51f1599004332836249602d08fb63": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "bed97dcd78c94f1787c8ee69811c02ca": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_13fcf9528f9548bb8caa660385be9120",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_0c2a0ef39f0c4c5ba03eabbb5b2ea86f",
       "value": 1
      }
     },
     "c04c52d679cd4b568be5a32b4c95576d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_7943b2de9cf041beabe152d891433480",
       "placeholder": "​",
       "style": "IPY_MODEL_89e19c7b67f4426087e74d1527d8cda8",
       "value": " 32768/? [00:00&lt;00:00, 67429.52it/s]"
      }
     },
     "c46ce69aaa8a45328091edc43b5844b7": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "ce3545b63e2841f2b960ca99e7ff61f9": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_6d43bbb8f2994ea5acea8cf467e5f317",
       "placeholder": "​",
       "style": "IPY_MODEL_1d20e314643d48c1a1a3efc366e641a9",
       "value": " 2/2 [01:31&lt;00:00, 45.81s/it]"
      }
     },
     "d571e4ee37b141bb9af8252df746fc82": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "d70e77a3ae9440da925717e07cebaa36": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_2586115495b14dc493a9f546bc579995",
       "placeholder": "​",
       "style": "IPY_MODEL_9a2ea1ca84f147d3bc07d3ccbee77217",
       "value": " 8192/? [00:00&lt;00:00, 42927.94it/s]"
      }
     },
     "d8adedc4352743b6a551d8369cc80327": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "dcba547b291e4457b88fbdbe3f7da380": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HBoxModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HBoxModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HBoxView",
       "box_style": "",
       "children": [
        "IPY_MODEL_bed97dcd78c94f1787c8ee69811c02ca",
        "IPY_MODEL_c04c52d679cd4b568be5a32b4c95576d"
       ],
       "layout": "IPY_MODEL_00941cf88c6540069e6e12a39813ba4a"
      }
     },
     "df4b68e73d534e70b28baaa0c4dd8143": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "e288ee57c64d4c068d7d0ae64d990c17": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "DescriptionStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "description_width": ""
      }
     },
     "e5bb3c6fbab5456bbf4f224f683610a1": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "HTMLModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "HTMLModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "HTMLView",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_c46ce69aaa8a45328091edc43b5844b7",
       "placeholder": "​",
       "style": "IPY_MODEL_7c083fd862d74e888d8473ce33426846",
       "value": " 9920512/? [00:01&lt;00:00, 9373779.14it/s]"
      }
     },
     "ea58e23e71074f929cb4b902ab2ebbf4": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "ed9eba42000d4c12b14d06ba66b663d6": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "ProgressStyleModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "StyleView",
       "bar_color": null,
       "description_width": "initial"
      }
     },
     "eef08f88c6bf4501ab5770a6c85ef423": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "1.2.0",
      "model_name": "LayoutModel",
      "state": {
       "_model_module": "@jupyter-widgets/base",
       "_model_module_version": "1.2.0",
       "_model_name": "LayoutModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/base",
       "_view_module_version": "1.2.0",
       "_view_name": "LayoutView",
       "align_content": null,
       "align_items": null,
       "align_self": null,
       "border": null,
       "bottom": null,
       "display": null,
       "flex": null,
       "flex_flow": null,
       "grid_area": null,
       "grid_auto_columns": null,
       "grid_auto_flow": null,
       "grid_auto_rows": null,
       "grid_column": null,
       "grid_gap": null,
       "grid_row": null,
       "grid_template_areas": null,
       "grid_template_columns": null,
       "grid_template_rows": null,
       "height": null,
       "justify_content": null,
       "justify_items": null,
       "left": null,
       "margin": null,
       "max_height": null,
       "max_width": null,
       "min_height": null,
       "min_width": null,
       "object_fit": null,
       "object_position": null,
       "order": null,
       "overflow": null,
       "overflow_x": null,
       "overflow_y": null,
       "padding": null,
       "right": null,
       "top": null,
       "visibility": null,
       "width": null
      }
     },
     "efa1302e547f40c1b3d3b82b8b132d3d": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "1.5.0",
      "model_name": "FloatProgressModel",
      "state": {
       "_dom_classes": [],
       "_model_module": "@jupyter-widgets/controls",
       "_model_module_version": "1.5.0",
       "_model_name": "FloatProgressModel",
       "_view_count": null,
       "_view_module": "@jupyter-widgets/controls",
       "_view_module_version": "1.5.0",
       "_view_name": "ProgressView",
       "bar_style": "success",
       "description": "",
       "description_tooltip": null,
       "layout": "IPY_MODEL_04134add6b3842c4897434a65316cd6a",
       "max": 1,
       "min": 0,
       "orientation": "horizontal",
       "style": "IPY_MODEL_ed9eba42000d4c12b14d06ba66b663d6",
       "value": 1
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}