chapter02_supervised-learning/linear-regression-gluon.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Linear regression with ``gluon``\n",
    "\n",
    "Now that we've implemented a whole neural network from scratch, using nothing but ``mx.ndarray`` and ``mxnet.autograd``, let's see how we can make the same model while doing a lot less work. \n",
    "\n",
    "Again, let's import some packages, this time adding ``mxnet.gluon`` to the list of dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import mxnet as mx\n",
    "from mxnet import nd, autograd, gluon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set the context\n",
    "\n",
    "We'll also want to set a context to tell gluon where to do most of the computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data_ctx = mx.cpu()\n",
    "model_ctx = mx.cpu()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build the dataset\n",
    "\n",
    "Again we'll look at the problem of linear regression and stick with the same synthetic data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "num_inputs = 2\n",
    "num_outputs = 1\n",
    "num_examples = 10000\n",
    "\n",
    "def real_fn(X):\n",
    "    return 2 * X[:, 0] - 3.4 * X[:, 1] + 4.2\n",
    "    \n",
    "X = nd.random_normal(shape=(num_examples, num_inputs))\n",
    "noise = 0.01 * nd.random_normal(shape=(num_examples,))\n",
    "y = real_fn(X) + noise"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the data iterator\n",
    "\n",
    "We'll stick with the ``DataLoader`` for handling our data batching."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batch_size = 4\n",
    "train_data = gluon.data.DataLoader(gluon.data.ArrayDataset(X, y),\n",
    "                                      batch_size=batch_size, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define the model\n",
    "\n",
    "When we implemented things from scratch, \n",
    "we had to individually allocate parameters \n",
    "and then compose them together as a model. \n",
    "While it's good to know how to do things from scratch, \n",
    "with `gluon`, we can just compose a network from predefined layers. \n",
    "For a linear model, the appropriate layer is called `Dense`. \n",
    "It's called a *dense* layer because every node in the input \n",
    "is connected to every node in the subsequent layer. \n",
    "That description seems excessive \n",
    "because we only have one (non-input) layer here, \n",
    "and that layer only contains one node!\n",
    "But in subsequent chapters we'll typically work \n",
    "with networks that have multiple outputs, \n",
    "so we might as well start thinking in terms of layers of nodes. \n",
    "Because a linear model consists of just a single `Dense` layer, we can instantiate it with one line.\n",
    "\n",
    "As in [the previous notebook](linear-regression-scratch.ipynb), \n",
    "we have an input dimension of 2 and an output dimension of 1. \n",
    "the most direct way to instantiate a ``Dense`` layer with these dimensions\n",
    "is to specify the number of inputs and the number of outputs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net = gluon.nn.Dense(1, in_units=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it! We've already got a neural network. \n",
    "Like our hand-crafted model in the previous notebook, \n",
    "this model has a weight matrix and bias vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Parameter dense4_weight (shape=(1, 2), dtype=None)\n",
       "Parameter dense4_bias (shape=(1,), dtype=None)\n"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(net.weight)\n",
    "print(net.bias)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, `net.weight` and `net.bias` are not actually NDArrays.\n",
    "They are instances of the `Parameter` class.\n",
    "We use `Parameter` instead of directly accessing NDAarrays for several reasons. \n",
    "For example, they provide convenient abstractions for initializing values.\n",
    "Unlike NDArrays, Parameters can be associated with multiple contexts simultaneously.\n",
    "This will come in handy in future chapters when we start thinking about distributed learning across multiple GPUs.\n",
    "\n",
    "In `gluon`, all neural networks are made out of Blocks (`gluon.Block`).\n",
    "Blocks are just units that take inputs and generate outputs.\n",
    "Blocks also contain parameters that we can update. \n",
    "Here, our network consists of only one layer, \n",
    "so it's convenient to access our parameters directly. \n",
    "When our networks consist of 10s of layers, this won't be so fun.\n",
    "No matter how complex our network, \n",
    "we can grab all its parameters by calling `collect_params()` as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dense4_ (\n",
       "  Parameter dense4_weight (shape=(1, 2), dtype=None)\n",
       "  Parameter dense4_bias (shape=(1,), dtype=None)\n",
       ")"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.collect_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The returned object is a `gluon.parameter.ParameterDict`. \n",
    "This is a convenient abstraction for retrieving and manipulating groups of Parameter objects.\n",
    "Most often, we'll want to retrieve all of the parameters in a neural network:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "mxnet.gluon.parameter.ParameterDict"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(net.collect_params())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize parameters\n",
    "Once we initialize our Parameters, we can access their underlying data and context(s),\n",
    "and we can also feed data through the neural network to generate output.\n",
    "However, we can't get going just yet. \n",
    "If we try invoking your model by calling ``net(nd.array([[0,1]]))``, \n",
    "we'll confront the following hideous error message:\n",
    "\n",
    "```RuntimeError: Parameter dense1_weight has not been initialized...```\n",
    "\n",
    "That's because we haven't yet told ``gluon`` what the *initial values* for our parameters should be!\n",
    "We initialize parameters by calling the `.initialize()` method of a ParameterDict. \n",
    "We'll need to pass in two arguments. \n",
    "\n",
    "* An initializer, many of which live in the `mx.init` module. \n",
    "* A context where the parameters should live. In this case we'll pass in the `model_ctx`. Most often this will either be a GPU or a list of GPUs. \n",
    "\n",
    "*MXNet* provides a variety of common initializers in ``mxnet.init``.\n",
    "To keep things consistent with the model we built by hand, \n",
    "we'll initialize each parameter by sampling from a standard normal distribution, \n",
    "using `mx.init.Normal(sigma=1.)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net.collect_params().initialize(mx.init.Normal(sigma=1.), ctx=model_ctx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deferred Initialization\n",
    "\n",
    "When we call ``initialize``, ``gluon`` associates each parameter with an initializer.\n",
    "However, the *actual initialization* is deferred until we make a first forward pass. \n",
    "In other words, the parameters are only initialized when they're needed. \n",
    "If we try to call `net.weight.data()` we'll get the following error:\n",
    "\n",
    "``DeferredInitializationError: Parameter dense2_weight has not been initialized yet because initialization was deferred. Actual initialization happens during the first forward pass. Please pass one batch of data through the network before accessing Parameters.``\n",
    "\n",
    "Passing data through a `gluon` model is easy. \n",
    "We just sample a batch of the appropriate shape and call `net` \n",
    "just as if it were a function. \n",
    "This will invoke `net`'s `forward()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "[[-1.33219385]]\n",
       "<NDArray 1x1 @cpu(0)>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_data = nd.array([[4,7]])\n",
    "net(example_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that `net` is initialized, we can access each of its parameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "[[-0.25217363 -0.04621419]]\n",
      "<NDArray 1x2 @cpu(0)>\n",
      "\n",
      "[ 0.]\n",
      "<NDArray 1 @cpu(0)>\n"
     ]
    }
   ],
   "source": [
    "print(net.weight.data())\n",
    "print(net.bias.data())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Shape inference\n",
    "\n",
    "Recall that previously, we instantiated our network with `gluon.nn.Dense(1, in_units=2)`. \n",
    "One slick feature that we can take advantage of in ``gluon`` is shape inference on parameters. \n",
    "Because our parameters never come into action until we pass data through the network,\n",
    "we don't actually have to declare the input dimension (`in_units`). \n",
    "Let's try this again, but letting `gluon` do more of the work:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "net = gluon.nn.Dense(1)\n",
    "net.collect_params().initialize(mx.init.Normal(sigma=1.), ctx=model_ctx)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll elaborate on this and more of ``gluon``'s internal workings in subsequent chapters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define loss\n",
    "\n",
    "Instead of writing our own loss function we're just going to access squared error by instantiating ``gluon.loss.L2Loss``. Just like layers, and whole networks, a loss in gluon is just a `Block`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "square_loss = gluon.loss.L2Loss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimizer\n",
    "Instead of writing stochastic gradient descent from scratch every time, we can instantiate a ``gluon.Trainer``, passing it a dictionary of parameters. Note that the ``SGD`` optimizer in ``gluon`` also has a few bells and whistles that you can turn on at will, including *momentum* and *clipping* (both are switched off by default). These modifications can help to converge faster and we'll discuss them later when we go over a variety of optimization algorithms in detail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "trainer = gluon.Trainer(net.collect_params(), 'sgd', {'learning_rate': 0.0001})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Execute training loop\n",
    "\n",
    "You might have noticed that it was a bit more concise to express our model in ``gluon``. For example, we didn't have to individually allocate parameters, define our loss function, or implement stochastic gradient descent. The benefits of relying on ``gluon``'s abstractions will grow substantially once we start working with much more complex models. But once we have all the basic pieces in place, the training loop itself is quite similar to what we would do if implementing everything from scratch. \n",
    "\n",
    "To refresh your memory. For some number of ``epochs``, we'll make a complete pass over the dataset (``train_data``), grabbing one mini-batch of inputs and the corresponding ground-truth labels at a time. \n",
    "\n",
    "Then, for each batch, we'll go through the following ritual. So that this process becomes maximally ritualistic, we'll repeat it verbatim:\n",
    "\n",
    "* Generate predictions (``yhat``) and the loss (``loss``) by executing a forward pass through the network.\n",
    "* Calculate gradients by making a backwards pass through the network via ``loss.backward()``. \n",
    "* Update the model parameters by invoking our SGD optimizer (note that we need not tell ``trainer.step`` about which parameters but rather just the amount of data, since we already performed that in the initialization of ``trainer``).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 0, loss: 3.44980202263\n",
      "Epoch 1, loss: 2.10364257665\n",
      "Epoch 2, loss: 1.28279426137\n",
      "Epoch 3, loss: 0.782256319318\n",
      "Epoch 4, loss: 0.477034088909\n",
      "Epoch 5, loss: 0.290909814427\n",
      "Epoch 6, loss: 0.177411796283\n",
      "Epoch 7, loss: 0.108197494675\n",
      "Epoch 8, loss: 0.0659899789031\n",
      "Epoch 9, loss: 0.040249745576\n"
     ]
    }
   ],
   "source": [
    "epochs = 10\n",
    "loss_sequence = []\n",
    "num_batches = num_examples / batch_size\n",
    "\n",
    "for e in range(epochs):\n",
    "    cumulative_loss = 0\n",
    "    # inner loop\n",
    "    for i, (data, label) in enumerate(train_data):\n",
    "        data = data.as_in_context(model_ctx)\n",
    "        label = label.as_in_context(model_ctx)\n",
    "        with autograd.record():\n",
    "            output = net(data)\n",
    "            loss = square_loss(output, label)\n",
    "        loss.backward()\n",
    "        trainer.step(batch_size)\n",
    "        cumulative_loss += nd.mean(loss).asscalar()\n",
    "    print(\"Epoch %s, loss: %s\" % (e, cumulative_loss / num_examples))\n",
    "    loss_sequence.append(cumulative_loss)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing the learning curve\n",
    "Now let's check how quickly SGD learns the linear regression model by plotting the learning curve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x7efc87a7f0f0>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAggAAAF7CAYAAAC3onORAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl4lOW9//H3NztJIOwBQlgUEBAEJCBWq7iDVkGrqN2s\ntcXTalf9tWp7ao/V9pzWLnpOa8WlajfBBUVF0apxaQVBQFYRRFnCvhOQhCTf3x/zBAcSZICZeSaT\nz+u65soz97PMd+4LzSfPct/m7oiIiIhEywi7ABEREUk9CggiIiLSgAKCiIiINKCAICIiIg0oIIiI\niEgDCggiIiLSgAKCiIiINKCAICIiIg0oIIiIiEgDCggiIiLSQFbYBYStffv23qNHj7gdb9euXRQU\nFMTteNI49XNyqJ+TR32dHOpneOeddza5e4dDbdfsA0KPHj2YNWtW3I5XXl7OyJEj43Y8aZz6OTnU\nz8mjvk4O9TOY2YpYttMlBhEREWlAAUFEREQaUEAQERGRBhQQREREpIGkBQQzyzOzt83sXTNbaGb/\nFbQ/ZGYfmtnc4DU4aDczu9vMlpnZPDM7MepYV5nZ0uB1VVT7UDObH+xzt5lZsr6fiIhIOknmUwxV\nwJnuXmlm2cCbZvZ8sO7/ufvjB2w/GugdvE4C7gFOMrO2wK1AGeDAO2Y2xd23Btt8A5gBTAVGAc8j\nIiIihyVpZxA8ojJ4mx28/FN2GQM8Euw3HWhtZp2B84CX3H1LEApeAkYF61q5+3R3d+ARYGzCvpCI\niEgaS+o9CGaWaWZzgQ1EfsnPCFbdEVxG+J2Z5QZtJcCqqN1XB22f1r66kXYRERE5TEkdKMnda4HB\nZtYamGxmA4CbgXVADjAB+BFwWyLrMLPxwHiA4uJiysvL43bsysrKuB5PGqd+Tg71c/Kor5ND/Ry7\nUEZSdPdtZvYqMMrd7wyaq8zsz8CNwfsKoDRqt65BWwUw8oD28qC9ayPbN/b5E4iEEcrKyjyeo2pp\nlK7kUD8nh/o5edTXyaF+jl0yn2LoEJw5wMxaAOcA7wX3DhA8cTAWWBDsMgX4SvA0wwhgu7uvBaYB\n55pZGzNrA5wLTAvW7TCzEcGxvgI8nazvJyIikk6SeQahM/CwmWUSCSaT3P1ZM3vFzDoABswF/iPY\nfipwPrAM2A1cDeDuW8zs58DMYLvb3H1LsPwt4CGgBZGnF5L6BMOyDZUs3Vq73+kNERGRpihpAcHd\n5wFDGmk/8yDbO3DdQdY9CDzYSPssYMDRVXrkvjdxDjt2VPONi8OqQEREJD40kmIcjSsrZeXOOhZU\nbA+7FBERkaOigBBHYwaVkJUBk2atOvTGIiIiKUwBIY6K8rMpK87kqTkV7NlbG3Y5IiIiR0wBIc4+\nW5LNjj01TFu4LuxSREREjpgCQpz1a5dB1zYtdJlBRESaNAWEOMsw47Khpfxr2WZWbdkddjkiIiJH\nRAEhAS4t64oZPPbO6kNvLCIikoIUEBKgpHULTu3VnsdnraK27tMmrBQREUlNCggJcvmwUtZs38O/\nlm0KuxQREZHDpoCQIOf0L6Z1fjYTdbOiiIg0QQoICZKblcnYwSW8tHA9W3dVh12OiIjIYVFASKBx\nZaVU19bx1NxGZ50WERFJWQoICdS/SysGlhQxceYqInNPiYiINA0KCAk2blgp763byYKKHWGXIiIi\nEjMFhAS7aFAXcrMymDhrZdiliIiIxEwBIcGKWmQzekAnnp67RhM4iYhIk6GAkATjykrZuaeGFxZo\nAicREWkaFBCSYMQx7Sht24KJMzUmgoiINA0KCEmQkWGMG1rKW8s3s3KzJnASEZHUp4CQJJ8fWj+B\nk84iiIhI6lNASJIurVtwWu8OPP7Oak3gJCIiKU8BIYnGlZWydvse3li6MexSREREPpUCQhKd3b8j\nbfKzmaQJnEREJMUpICRRblYmFw/pykuL1rNFEziJiEgKU0BIsnHDurK31pk8RxM4iYhI6lJASLK+\nnVoxqGsRj83SBE4iIpK6FBBCcFlZZAKneau3h12KiIhIoxQQQnDR4C7kZWcwUTcriohIilJACEGr\nvGzOH9CZZ+au4eNqTeAkIiKpRwEhJJeVlbKzqobnF6wNuxQREZEGFBBCMuKYtnRvl68xEUREJCUp\nIITEzLhsaFemL9/Cis27wi5HRERkPwoIIbp0aCkZhs4iiIhIylFACFGnojxO76MJnEREJPUoIIRs\nXFkp63dU8fr7msBJRERSR9ICgpnlmdnbZvaumS00s/8K2nua2QwzW2ZmE80sJ2jPDd4vC9b3iDrW\nzUH7EjM7L6p9VNC2zMxuStZ3Oxpn9SumXUGOLjOIiEhKSeYZhCrgTHcfBAwGRpnZCOB/gN+5ey9g\nK3BNsP01wNag/XfBdphZf+AK4HhgFPBHM8s0s0zgD8BooD9wZbBtSsvJyuDiISX8c/F6NldWhV2O\niIgIkMSA4BGVwdvs4OXAmcDjQfvDwNhgeUzwnmD9WWZmQfuj7l7l7h8Cy4DhwWuZuy9392rg0WDb\nlDduWKkmcBIRkZSS1HsQgr/05wIbgJeAD4Bt7l4TbLIaKAmWS4BVAMH67UC76PYD9jlYe8rrU9yS\nwaWtmThTEziJiEhqyErmh7l7LTDYzFoDk4G+yfz8emY2HhgPUFxcTHl5edyOXVlZeUTHG9xqLw8t\nrObBp1/h2NaZcasnXR1pP8vhUT8nj/o6OdTPsUtqQKjn7tvM7FXgZKC1mWUFZwm6AvXn2SuAUmC1\nmWUBRcDmqPZ60fscrP3Az58ATAAoKyvzkSNHxuNrAVBeXs6RHG/onr1MfP9lltV15JqRA+NWT7o6\n0n6Ww6N+Th71dXKon2OXzKcYOgRnDjCzFsA5wGLgVeDSYLOrgKeD5SnBe4L1r3jk/PsU4IrgKYee\nQG/gbWAm0Dt4KiKHyI2MUxL/zeKjZV425w/szDPvrmF3dc2hdxAREUmgZN6D0Bl41czmEfll/pK7\nPwv8CPiBmS0jco/BA8H2DwDtgvYfADcBuPtCYBKwCHgBuM7da4MzENcD04gEj0nBtk3GuLKuVFbV\nMHX+urBLERGRZi5plxjcfR4wpJH25USeQDiwfQ9w2UGOdQdwRyPtU4GpR11sSIb3bEuPYAKnS4d2\nDbscERFpxjSSYgoxMy4rK+XtD7fw4SZN4CQiIuFRQEgxlw7tSobBYxpZUUREQqSAkGKKW+VxxnEd\nefyd1dTU1oVdjoiINFMKCCnosrJSNuys4jVN4CQiIiFRQEhBZ/XrSPtCTeAkIiLhUUBIQdmZkQmc\nXl68gY07NYGTiIgknwJCirp8WCk1dc5TmsBJRERCoICQonp1bMmJ3VozcZYmcBIRkeRTQEhh48pK\nWbahktkrt4VdioiINDMKCCnsc4O60CI7U2MiiIhI0ikgpLDC3CwuOCEygdOuKk3gJCIiyaOAkOIu\nH1bKrupaps5fG3YpIiLSjCggpLiy7m04pn2BxkQQEZGkUkBIcfUTOM38aCsfbKwMuxwREWkmFBCa\ngM8PLSEzw3hs1uqwSxERkWZCAaEJ6NgyMoHTE7M1gZOIiCSHAkITMa6sKxt3VlG+RBM4iYhI4ikg\nNBFn9O1I+8JcJupmRRERSQIFhCYiOzODz59YwivvbWDDzj1hlyMiImlOAaEJuayslNo6Z/JsTeAk\nIiKJpYDQhPTqWEhZ9zaawElERBJOAaGJGVdWyvKNu5i9cmvYpYiISBpTQGhiLjihM/k5mUycqZsV\nRUQkcRQQmpiC3Cw+d0Jnnp23lkpN4CQiIgmigNAEXT6slN3VtUydpwmcREQkMRQQmqATu7Xh2A4F\nGhNBREQSRgGhCTIzxpWV8s6KrSzboAmcREQk/hQQmqhLTuwaTOCkswgiIhJ/CghNVIeWuZzZtyNP\nzK5gryZwEhGROFNAaMIuLytlU2UVr763IexSREQkzSggNGEjj+tAh5a5TNJlBhERiTMFhCYsKzOD\nz5/YlVeXbGTDDk3gJCIi8aOA0MSNK+tKbZ3zhCZwEhGROFJAaOKO6VDIsB5teEwTOImISBwlLSCY\nWamZvWpmi8xsoZl9N2j/mZlVmNnc4HV+1D43m9kyM1tiZudFtY8K2paZ2U1R7T3NbEbQPtHMcpL1\n/cI0rqyU5Zt2MWuFJnASEZH4SOYZhBrgBnfvD4wArjOz/sG637n74OA1FSBYdwVwPDAK+KOZZZpZ\nJvAHYDTQH7gy6jj/ExyrF7AVuCZZXy5MF5zQmQJN4CQiInGUtIDg7mvdfXawvBNYDJR8yi5jgEfd\nvcrdPwSWAcOD1zJ3X+7u1cCjwBgzM+BM4PFg/4eBsYn5NqklPyeLCwd14TlN4CQiInESyj0IZtYD\nGALMCJquN7N5ZvagmbUJ2kqA6D+JVwdtB2tvB2xz95oD2puFccNK+XhvLc++uybsUkREJA1kJfsD\nzawQeAL4nrvvMLN7gJ8DHvz8DfC1BNcwHhgPUFxcTHl5edyOXVlZGdfjxcrd6VJg3P/KQjrtXp70\nz0+2sPq5uVE/J4/6OjnUz7FLakAws2wi4eBv7v4kgLuvj1p/H/Bs8LYCKI3avWvQxkHaNwOtzSwr\nOIsQvf1+3H0CMAGgrKzMR44ceXRfLEp5eTnxPN7huDpzOXdMXUzX/kPp1bFlKDUkS5j93Jyon5NH\nfZ0c6ufYJfMpBgMeABa7+2+j2jtHbXYxsCBYngJcYWa5ZtYT6A28DcwEegdPLOQQuZFxikee8XsV\nuDTY/yrg6UR+p1Rz8YklZGWYblYUEZGjlsx7EE4BvgycecAjjb8ys/lmNg84A/g+gLsvBCYBi4AX\ngOvcvTY4O3A9MI3IjY6Tgm0BfgT8wMyWEbkn4YEkfr/QtS/M5ax+HXlSEziJiMhRStolBnd/E7BG\nVk39lH3uAO5opH1qY/u5+3IiTzk0W5cPK2XawvW8vHgDowZ0CrscERFpojSSYpo5rXcHilvl8pgm\ncBIRkaOggJBmPpnAaQPrNYGTiIgcIQWENDSurJQ6h8ffWR12KSIi0kQpIKShHu0LGN6zrSZwEhGR\nI6aAkKYuLyvlo827efvDLWGXIiIiTZACQpo6f2BnCnOzmDRLlxlEROTwKSCkqRY5mVw4qAtT569l\n5569YZcjIiJNjAJCGrs8mMDpmXfXhl2KiIg0MQoIaWxQ1yL6FBcySWMiiIjIYVJASGNmxriyUuau\n2sb763eGXY6IiDQhCghp7uIhJWRnGpM0gZOIiBwGBYQ0164wl7P7FfPknAqqazSBk4iIxEYBoRkY\nN6yULbuqeXnx+rBLERGRJkIBoRk4rXcHOrXK082KIiISMwWEZiAzw7h0aFdee38j67ZrAicRETk0\nBYRm4rKyrtQ5PDFbIyuKiMihKSA0E93bFTDimLZMmrWKujpN4CQiIp8upoBgZhlmlhH1vpOZfd3M\nTklcaRJvlw8rZcXm3czQBE4iInIIsZ5BeA74NoCZFQKzgF8D5Wb2lQTVJnE26vjOtMzN4jHdrCgi\nIocQa0AoA14Jli8BdgAdgW8ANyagLkmAFjmZXDS4C1MXrGWHJnASEZFPEWtAKAS2BcvnApPdfS+R\n0HBsIgqTxLh8WCl79tbxzLtrwi5FRERSWKwBYSVwipkVAOcBLwXtbYHdiShMEmNgSRF9O7XU0Msi\nIvKpYg0IvwX+AqwGKoDXg/bTgPkJqEsSpH4Cp3dXb+e9dTvCLkdERFJUTAHB3e8FTga+Bpzq7vWD\n+n8A/GeCapMEuXhICTmZGUyaqTERRESkcTGPg+Dus9x9srtXAphZtrs/5+7/Slx5kghtCnI4p38x\nk+espqqmNuxyREQkBcU6DsJ3zOzzUe8fAD42syVmdlzCqpOEGTeslK279/Ly4g1hlyIiIiko1jMI\n3wE2ApjZacA44AvAXOA3iSlNEunUXu3pUpTHRN2sKCIijYg1IJQAHwbLFwKPufsk4GfAiATUJQlW\nP4HT60s3smbbx2GXIyIiKSbWgFA/MBLAOcDLwfJeIC/eRUlyXFZWijs88Y5uVhQRkf3FGhBeBO4z\ns/uBXsDzQfvxfHJmQZqY0rb5nNKrHX+ZvkIjK4qIyH5iDQjXAf8COgCXunv9bD8nAv9IRGGSHD88\nry8bK6u4c9qSsEsREZEUkhXLRu6+g2CypgPab417RZJUg0pb89XP9OChf3/E2CElnNitTdgliYhI\nCoh5HAQzyzWzr5nZnWb2azP7qpnlJrI4SY4bzj2OTq3yuPmJ+eytrTv0DiIikvZiHQehP7CUyJDL\nJxF5cuH3wPtm1i9x5UkyFOZm8fMxA1iyficTXl8edjkiIpICYj2DcBcwB+jm7p91988C3YB3iQQF\naeLO7l/M+QM7cffLS/lo066wyxERkZDFGhBOAW4J7kUA9t2X8GPg1FgOYGalZvaqmS0ys4Vm9t2g\nva2ZvWRmS4OfbYJ2M7O7zWyZmc0zsxOjjnVVsP1SM7sqqn2omc0P9rnbzCzG7yfArRceT05mBj9+\naj7uHnY5IiISolgDwh6gdSPtRcG6WNQAN7h7fyKXKK4LLl3cBLzs7r2JjK9wU7D9aKB38BoP3AOR\nQAHcSuRSx3Dg1vpQEWzzjaj9RsVYmwDFrfL40ei+/GvZZp6cXRF2OSIiEqJYA8IzRMZBOMXMMoPX\nqcC9wJRYDuDua919drC8E1hMZITGMcDDwWYPA2OD5THAIx4xHWhtZp2B84CX3H2Lu28FXgJGBeta\nuft0j/z5+0jUsSRGXxjejaHd23D7c4vYsqs67HJERCQksQaE7xK5SfENImcM9gCvAe8D3zvcDzWz\nHsAQYAZQ7O5rg1XrgOJguQSInihgddD2ae2rG2mXw5CRYfzykoFUVtVw+3OLwi5HRERCEus4CNuA\nMWbWG+gbNC9292WH+4FmVgg8AXzP3XdE3ybg7m5mCb/4bWbjiVy2oLi4mPLy8rgdu7KyMq7HC8uo\nHlk8ObuCYzM2c3z7zLDLaSBd+jnVqZ+TR32dHOrn2MUUEOq5+1IiZxKOiJllEwkHf3P3J4Pm9WbW\n2d3XBpcJ6ucfrgBKo3bvGrRVACMPaC8P2rs2sn1j32MCMAGgrKzMR44c2dhmR6S8vJx4Hi8sI06p\nZcFdbzDpQ2famM+Sl51aISFd+jnVqZ+TR32dHOrn2B00IJjZ3bEexN2/c6htgicKHiBy5uG3Uaum\nAFcB/x38fDqq/Xoze5TIDYnbgxAxDfhF1I2J5wI3u/sWM9thZiOIXLr4CvC/sX4H2V9edia/uHgg\nV943nbtfXsoPR/U99E4iIpI2Pu0MwsAYjxHrJYFTgC8D881sbtB2C5FgMMnMrgFWAOOCdVOB84Fl\nwG7gaoAgCPwcmBlsd1vU3BDfAh4CWhCZUKp+Uik5Aicf247LhnZlwuvLuXBQF/p1bhV2SSIikiQH\nDQjufkY8P8jd3wQONi7BWY1s70QmiWrsWA8CDzbSPgsYcBRlygFuOb8fr7y3gZufnM8T3/wMmRka\nWkJEpDmIeS4GaZ7aFOTw0wv7M3fVNv46fUXY5YiISJIoIMghXTSoC6f16cCvpy1h7faPwy5HRESS\nQAFBDsnMuGPsAGrq6rj16YVhlyMiIkmggCAxKW2bz/fP7sOLi9bzwoJ1YZcjIiIJpoAgMbvm1J70\n79yKW6csYOeevWGXIyIiCRRzQDCzYjO70czuMbP2QdspZtYzceVJKsnKzOCXlwxk484qfj1tSdjl\niIhIAsUUEMxsKLAE+CJwDVD/QPw5wB2JKU1S0aDS1lz1mR78ZfoK3lmxNexyREQkQWI9g3AncJe7\nDwGqotqnERkASZqRG849js6t8rjlyfnsra0LuxwREUmAWAPCUD6ZkjnaWj6ZfVGaicLcLG4bM4Al\n63cy4fXlYZcjIiIJEGtA+Bho00h7Xz6ZXEmakbP7F3P+wE7c9fJSPty0K+xyREQkzmINCE8Dt5pZ\nbvDezawH8D9EZmeUZuhnFx5PblYGP548n8jI2CIiki5iDQg3Am2BjUA+8CaRSZS2AT9JTGmS6jq2\nyuOm0X359webeXJ2ozNri4hIE/Vpsznu4+47gFPN7EzgRCLBYra7/zORxUnqu3JYNybPruD25xYx\n8rgOtCvMPfROIiKS8g5roCR3f8Xd73T3XykcCEBGhvGLSwZSWVXDHc8tDrscERGJk5jOIJjZTw+y\nyoE9RC43vODumsmnGepT3JJvnn4sd7+yjEtO7MqpvduHXZKIiBylmAICcBnQDSgA1gRtXYBdRO5L\nKAU2mNnp7q7n3pqhb53Ri2fnreWWyfOZ9r3TaJGTGXZJIiJyFGK9xPAbYCbQw927uXs3oAcwA7iN\nSFh4H/htIoqU1JeXnckdFw9k5Zbd3P3K0rDLERGRoxRrQLgV+IG7r65vCJZ/CNzm7puBHwMnx79E\naSpOPrYd48q6MuH15SxeuyPsckRE5CjEGhCKgbxG2nOBjsHyeiKPQEozdsv5/WjdIpubn5xPbZ3G\nRhARaapiDQj/BO41s2FmlhG8hgH3AC8F2wwEPkxEkdJ0tM7P4acX9mfuqm38dfqKsMsREZEjFGtA\n+DqRMwQziEzWVAVMD9q+EWyzk8iAStLMXTSoC6f16cCvXniPtdv1YIuISFMUU0Bw9w3uPgroB3w+\nePVz99HuviHY5lV3fzFxpUpTYWbcMXYAte7c+vTCsMsREZEjcLgDJS1x9ynB6/1EFSVNX2nbfL5/\ndh9eXLSeFxasC7scERE5TLGOg4CZ9QEuJTIeQk70Onf/WpzrkjRwzak9eXruGm6dsoDP9GpHq7zs\nsEsSEZEYxXQGwcwuAOYBFwJfA44DzgcuBjRsnjQqKzODX14ykI07q7hz2pKwyxERkcMQ6yWG24D/\ncveTidyg+GUiAyX9EyhPSGWSFgaVtuaqz/TgL9NX8M6KrWGXIyIiMYo1IBwHTAyW9wL57r6HSHD4\nXiIKk/Rxw7nH0blVHrc8OZ/qmrqwyxERkRjEGhB28slASWuBXsFyFtAm3kVJeinMzeK2MQNYsn4n\n972hqTpERJqCWAPCDODUYPk54DdmdivwZ+CtRBQm6eXs/sVcMLAzd728lA837Qq7HBEROYRYA8IP\niAyMBPAz4EUiYyEsIzKIksgh3Xphf3KzMvjx5Pm4axhmEZFUdsiAYGZZQF+gAsDdd7v7N939BHe/\n1N1XJrpISQ8dW+Vx0+i+/PuDzTwxuyLsckRE5FMcMiC4ew3wJNAy8eVIurtyWDfKurfhjucWsbmy\nKuxyRETkIGK9xPAun9yYKHLEMjKMX14ykMqqGu54bnHY5YiIyEHEGhB+RuTGxLFmVmpmbaNfCaxP\n0lDv4pZ88/RjeXJOBW8s3Rh2OSIi0ohYA8JzRKZzfhL4CNgYvDYFPw/JzB40sw1mtiCq7WdmVmFm\nc4PX+VHrbjazZWa2xMzOi2ofFbQtM7Obotp7mtmMoH2ime03HLSklm+d0Ytj2hfw48kL+Li6Nuxy\nRETkALEGhDOiXmdGverfx+IhYFQj7b9z98HBayqAmfUHrgCOD/b5o5llmlkm8AdgNNAfuDLYFuB/\ngmP1ArYC18RYl4QgLzuTX1wykJVbdnP3K0vDLkdERA4Q02RN7v7a0X6Qu79uZj1i3HwM8Ki7VwEf\nmtkyYHiwbpm7Lwcws0eBMWa2mEhQ+UKwzcNELovcc7R1S+KMOKYd48q6MuH15Vw0qAv9OrcKuyQR\nEQnEPN2zmQ00s/8zs+fNrHPQNtbMhhxlDdeb2bzgEkT9qIwlwKqobVYHbQdrbwdsC564iG6XFHfL\n+f1o3SKbm56cT22dxkYQEUkVMZ1BMLNzgSnA80T+Um8RrDoW+Cow9gg//x7g54AHP39DZLbIhDKz\n8cB4gOLiYsrLy+N27MrKyrgerzm47FjjT/O2cetf/snZ3WObElr9nBzq5+RRXyeH+jl2MQUEIr+8\nf+DufzSznVHt5cANR/rh7r6+ftnM7gOeDd5WAKVRm3YN2jhI+2agtZllBWcRordv7HMnABMAysrK\nfOTIkUf6FRooLy8nnsdrDk53Z9GemUz+YAvfGnMqXVq3OOQ+6ufkUD8nj/o6OdTPsYv1EsMAYGoj\n7VuAI37Msf5SReBioP4JhynAFWaWa2Y9gd7A28BMoHfwxEIOkRsZp3hk3N5XgUuD/a8Cnj7SuiS5\nzIw7xg6g1p2fPr1QwzCLiKSAWAPCFhq/pn8ikev9h2Rm/yAysdNxZrbazK4BfmVm881sHpEnIr4P\n4O4LgUnAIuAF4Dp3rw3ODlwPTAMWA5OCbQF+BPwguKGxHfBAjN9NUkBp23x+cE4f/rl4PdMWrgu7\nHBGRZi/WSwx/B35tZuOI3C+QZWanA3cSmdHxkNz9ykaaD/pL3N3vAO5opH0qjZzNCJ5sGH5guzQd\nXzulJ0/NWcOtUxbymV7taZUX2/0IIiISf7GeQfgJ8CGwAigk8pf9K8CbNPJLXORIZGVm8MtLBrJx\nZxW/fmFJ2OWIiDRrMQUEd9/r7l8E+gDjiIw30Nfdv+zuGgZP4mZQaWu++pme/HXGCt5ZsTXsckRE\nmq2YAkIw3kG2u3/g7o+7+yR31/B3khA3nNuHLkUtuOXJ+VTX1IVdjohIsxTrJYa/A+vM7E9mdkoi\nCxIpyM3itjHHs2T9Tu57Y3nY5YiINEuxBoRi4EYiAyO9ZmbLzex2M+ubuNKkOTurXzEXDOzMXS8v\n5cNNu8IuR0Sk2Yn1HoSd7v5ndz8H6Ab8H5FJlBaa2cxEFijN160X9ic3K4MfT56vsRFERJIs5rkY\n6rn7GiIB4ZfAPCJjIYjEXcdWedw0ui///mAzT8w+6MCYIiKSAIcVEMzsDDO7H1gP3A/MBs5ORGEi\nAFcO60ZZ9zbc/twiNldWhV2OiEizEetTDL82s1VERjXsQGSio07ufo27v5rIAqV5y8gwfnnJQHZV\n1XD7c4vDLkdEpNmI9QzCZ4BfAJ3dfYy7P+bu+nNOkqJ3cUu+ObIXk+dU8MbSjWGXIyLSLMR6k+Ip\n7n6Pu29JdEEijfnWyGM5pn0BP568gI+rNTaXiEiixToXA2aWRWSug25ATvQ6d38kznWJ7CcvO5Nf\nXDKQKyY0GLNpAAAcZ0lEQVRM566XlzLi0DNCi4jIUYgpIATjHTwD9AQMqA323QtUAQoIknAjjmnH\n5WWl3PfGckpOzgu7HBGRtBbrPQi/B94BioDdQD+gDJgLfD4xpYk0dPP5fWmTn83986vYsWdv2OWI\niKStWAPCMOB2d98F1AFZ7j4b+CHwm0QVJ3Kg1vk5/M/nT2D1zjq+fP8Mtu2uDrskEZG0FGtAMCJn\nDgA2AiXB8mqgV7yLEvk0Z/Ur5vohuSxeu5Mr75uh8RFERBIg1oCwABgULL8N/MjMTgf+C1iWiMJE\nPs2Qjlncf1UZH26q5IoJ09mwY0/YJYmIpJVYA8IdRM4iAPyEyJMMrwLnAt9JQF0ih3Ranw48dPVw\nKrZ9zOUTprNm28dhlyQikjZiHQdhmrs/GSwvd/d+QHug2N3LE1ifyKcacUw7/nLNcDbtrGLcvW+x\nasvuQ+8kIiKHdNiTNdVz9y2uKfYkBQzt3pa/feMkdu6p4bI/vcXyjZVhlyQi0uQdcUAQSSUndG3N\no+NHsLe2jnH3Tuf99TvDLklEpElTQJC00a9zKyZeO4IMgysmTGfhmu1hlyQi0mQpIEha6dWxJZOu\nPZkW2ZlcOWE6c1dtC7skEZEmSQFB0k6P9gVMvHYErfNz+NL9M5j5keYYExE5XAoIkpa6tsln0rUn\n07FVLl954G3+tWxT2CWJiDQpCgiStjoV5TFx/Ml0b5fP1Q/N5NUlG8IuSUSkyVBAkLTWoWUu//jG\nCPoUFzL+kVlMW7gu7JJERJoEBQRJe20Kcvjb10cwoKSIb/1tNs+8uybskkREUp4CgjQLRS2y+cs1\nJzG0exu+++gcHn9nddgliYikNAUEaTYKc7N4+OrhfObY9tz42Lv8bcaKsEsSEUlZCgjSrLTIyeT+\nq8o4s29Hfjx5AQ+++WHYJYmIpCQFBGl28rIz+dOXhjJ6QCdue3YRfyzXjOUiIgdSQJBmKScrg/+9\ncghjBnfhVy8s4XcvvY/mHhMR+URW2AWIhCUrM4PfjhtMblYGd728lD01tdw0qi9mFnZpIiKhU0CQ\nZi0zw/jvS04gJyuDe19bTtXeOn76uf5kZCgkiEjzlrRLDGb2oJltMLMFUW1tzewlM1sa/GwTtJuZ\n3W1my8xsnpmdGLXPVcH2S83sqqj2oWY2P9jnbtOfgRKjjAzj52MG8PVTe/LQvz/ix0/Np65OlxtE\npHlL5j0IDwGjDmi7CXjZ3XsDLwfvAUYDvYPXeOAeiAQK4FbgJGA4cGt9qAi2+UbUfgd+lshBmRk/\nvqAf15/Ri3+8vYobH3uXmtq6sMsSEQlN0gKCu78OHDit3hjg4WD5YWBsVPsjHjEdaG1mnYHzgJfc\nfYu7bwVeAkYF61q5+3SP3Gn2SNSxRGJiZtx43nHceG4fnpxTwXcfnctehQQRaabCvgeh2N3XBsvr\ngOJguQRYFbXd6qDt09pXN9IuctiuP7M3edmZ3P7cYqpr6/i/LwwhNysz7LJERJIq7ICwj7u7mSXl\nwq+ZjSdy6YLi4mLKy8vjduzKysq4Hk8al+h+7gV8uX8Of1m0ns///iW+PSSX3Mzmd1uL/j0nj/o6\nOdTPsQs7IKw3s87uvja4TFA/H28FUBq1XdegrQIYeUB7edDetZHtG+XuE4AJAGVlZT5y5MiDbXrY\nysvLiefxpHHJ6OeRwMCZq/jRk/P48wd5PHDVMApyw/5PJrn07zl51NfJoX6OXdgDJU0B6p9EuAp4\nOqr9K8HTDCOA7cGliGnAuWbWJrg58VxgWrBuh5mNCJ5e+ErUsUSO2Lhhpfz+8sHM/GgrX3nwbXbs\n2Rt2SSIiSZHMxxz/AbwFHGdmq83sGuC/gXPMbClwdvAeYCqwHFgG3Ad8C8DdtwA/B2YGr9uCNoJt\n7g/2+QB4PhnfS9LfmMEl/N+VQ5i3ehtfun8G23ZXh12SiEjCJe18qbtfeZBVZzWyrQPXHeQ4DwIP\nNtI+CxhwNDWKHMzogZ35U1YG3/zrbK68bwZ/vWY47Qpzwy5LRCRhwr7EINJknNWvmAe+WsaHmyq5\nfMJ0NuzYE3ZJIiIJo4Agchg+27sDD109nLXbPmbcvW9Rse3jsEsSEUkIBQSRwzTimHY8cs1JbN5V\nzbg/vcXKzbvDLklEJO4UEESOwNDubfjHN0awq7qGcfe+xQcbK8MuSUQkrhQQRI7QgJIiHh0/gpq6\nOi6/dzpL1u0MuyQRkbhRQBA5Cn07teLR8SeTmQFXTHiLBRXbwy5JRCQuFBBEjlKvjoVMuvZk8nOy\nuPK+6cxZuTXskkREjpoCgkgcdG9XwMRrR9C2IIcv3T+Dtz88cOJSEZGmRQFBJE66tsln0rUn06ko\nj6sefJt/LdsUdkkiIkdMAUEkjopb5THx2pPp3i6fqx+ayavvbTj0TiIiKUgBQSTO2hfm8o9vjOC4\n4paM/8ssXliwLuySREQOmwKCSAK0Kcjhr18/iYElRVz399k8Pfegs4+LiKQkBQSRBClqkc0j15zE\n0O5t+N7EuUyauSrskkREYqaAIJJAhblZPHz1cE7t1Z4fPjGPax6ayUebdoVdlojIISkgiCRYi5xM\nHrhqGDeP7sv05Zs593ev8+tp77G7uibs0kREDkoBQSQJcrIyuPb0Y3nlxpFccEJn/vDqB5x552tM\neXcN7h52eSIiDSggiCRRcas8fnf5YB7/j5NpV5jDd/4xh8snTGfx2h1hlyYish8FBJEQlPVoy5Tr\nT+WOiwewdP1OLrj7DX769AK27a4OuzQREUABQSQ0mRnGF0/qzqs3juRLI7rz1+krOOPOcv42YwW1\ndbrsICLhUkAQCVnr/BxuGzOAZ7/9WXoXt+THkxcw5g9v8s4KzecgIuFRQBBJEf27tGLi+BHcfeUQ\nNu2s5vP3vMX3J85lw449YZcmIs2QAoJICjEzLhrUhZdvOJ3rzjiW5+at5Yw7y7n3tQ+orqkLuzwR\naUYUEERSUEFuFv/vvL68+P3TGHFMO375/HuMuut1Xnt/Y9iliUgzoYAgksJ6tC/gga8O489fHYY7\nXPXg23z94Vms3Lw77NJEJM0pIIg0AWf07cgL3/ssPxrVl39/sImzf/cav3lxiUZjFJGEUUAQaSJy\nszL55shjeeWGkYwe0In/fWUZZ//mNZ6dp9EYRST+FBBEmphORXncdcUQHvuPk2mdn8P1f5/DlfdN\n5711Go1RROJHAUGkiRrWoy3PfPtUfj52AO+t28kFd7/Jz6YsZPvuvWGXJiJpQAFBpAnLzDC+PKI7\nr94wkiuHl/LIWx9xxm/K+cfbKzUao4gcFQUEkTTQpiCH28cO5Jlvn8qxHQq4+cn5jP3Dv5i9cmvY\npYlIE6WAIJJGju9SxKRrT+auKwazYeceLvnjv7lh0rts2KnRGEXk8CggiKQZM2PM4BJeuWEk3xx5\nLFPereDMO1/jvteXazRGEYmZAoJImirIzeJHo/ry4vdPZ1iPNtwxdTGj73qd1zUao4jEQAFBJM31\nbF/An68ezgNXlVFT53zlwbcZ/8gsVm3RaIwicnApERDM7CMzm29mc81sVtDW1sxeMrOlwc82QbuZ\n2d1mtszM5pnZiVHHuSrYfqmZXRXW9xFJRWf1K+bF75/G/zvvON5Yuomzfvsav31xCR9X14Zdmoik\noJQICIEz3H2wu5cF728CXnb33sDLwXuA0UDv4DUeuAcigQK4FTgJGA7cWh8qRCQiNyuT687oxSs3\nns6o4ztx9yvLOPu3rzF1/lqNxigi+0mlgHCgMcDDwfLDwNio9kc8YjrQ2sw6A+cBL7n7FnffCrwE\njEp20SJNQeeiFtx95RAmjh9By7wsvvW32Xzx/hm8v35n2KWJSIpIlYDgwItm9o6ZjQ/ait19bbC8\nDigOlkuAVVH7rg7aDtYuIgdx0jHtePbbp/LzMcezcM0ORt/1Bv/1zEK2f6zRGEWau6ywCwic6u4V\nZtYReMnM3ote6e5uZnE7/xmEkPEAxcXFlJeXx+vQVFZWxvV40jj1c3yVArefnM0TS52H/vURj8/8\niEv75DCkqEr9nCT6N50c6ufYpURAcPeK4OcGM5tM5B6C9WbW2d3XBpcQNgSbVxD5/1m9rkFbBTDy\ngPbyg3zeBGACQFlZmY8cObKxzY5IeXk58TyeNE79nBgXngsLKrbzsykL+fOCrbzSKpNrzzqGCwZ2\npig/O+zy0pr+TSeH+jl2oV9iMLMCM2tZvwycCywApgD1TyJcBTwdLE8BvhI8zTAC2B5cipgGnGtm\nbYKbE88N2kTkMAwoKeKx/ziZ310+iOpa55bJ8xl2xz+59i+zeGHBWqpq9NSDSHOQCmcQioHJZgaR\nev7u7i+Y2UxgkpldA6wAxgXbTwXOB5YBu4GrAdx9i5n9HJgZbHebu29J3tcQSR9mxsVDutJ621I6\n9DmRyXMqeHruGqYtXE+rvCwuOKEzYweXMKxHWzIyLOxyRSQBQg8I7r4cGNRI+2bgrEbaHbjuIMd6\nEHgw3jWKNFdmxoCSIgaUFHHz6L78+4PNPBWEhX+8vYqS1i0YM7gLY4eU0Ke4ZdjlikgchR4QRKRp\nyMrM4LQ+HTitTwdur67hpUXrmTyngntfX84fyz+gf+dWXDykhIsGd6G4VV7Y5YrIUVJAEJHDlp+T\nxZjBJYwZXMKmyiqefXcNk+eu4Y6pi/nF84s55dj2jB1SwnnHF9MyTzc3ijRFCggiclTaF+by1VN6\n8tVTerJ8YyVPzV3DU3MquPGxd/nJUxmc078TYwd34bQ+HcjODP2+aBGJkQKCiMTNMR0K+cE5ffj+\n2b2ZvXIbT82p4Nl5a3jm3TW0Lcjhcyd0ZuyQEoaUtia4MVlEUpQCgojEnZkxtHsbhnZvw39+rj+v\nv7+RyXMrmDhzFY+8tYLu7fIZO7iEsUNK6Nm+IOxyRaQRCggiklA5WRmc3b+Ys/sXs3PPXl5YsI6n\n5lZw9ytLuevlpQwubc3FQ0r43AmdaVeYG3a5IhJQQBCRpGmZl81lZaVcVlbKuu17mPJuBZPnrOHW\nKQu57dlFnN6nA2MGd+Hc/p1okZMZdrkizZoCgoiEolNRHuNPO5bxpx3Le+t28NScNTw9t4JX3ttA\nQU4m5w3oxMVDSvjMse3J1GBMIkmngCAioevbqRU3jW7FD887jrc/2sJTcyp4bv5anpxdQceWuVw0\nKDIY0/FdWunmRpEkUUAQkZSRkWGMOKYdI45px88uOp5X39vA5DkVPPzWR9z/5of07ljI2CElXDSo\nC6Vt88MuVyStKSCISErKy85k9MDOjB7YmW27q3lu/lqemlPBr6ct4dfTljC8R1vGDinRTJMiCaKA\nICIpr3V+Dl88qTtfPKk7q7bs5um5FUyeU8Etk+fzsykLOaNvBy4eUsIZfTuSm6WbG0XiQQFBRJqU\n0rb5XH9mb647oxcLKnbw1Nz9Z5ocNaATI45px5BubejRLl/3LIgcIQUEEWmSzIyBXYsY2HX/mSan\nzl/HpFmrAWiTn83g0tYM6daGId1aM6i0Na00N4RITBQQRKTJi55psrbOWbphJ3NWbmPOyq3MWbmN\nV5dsBMAMenUoZEi31pzYrQ1DurWhV8dCPUYp0ggFBBFJK5kZRt9OrejbqRVXDu8GwPaP9zJv9bZ9\noeHFRev3nWUozM1iUGkRQ0ojZxkGl7bWiI4iKCCISDNQ1CKbz/buwGd7dwDA3flo8+59ZxjmrNrK\nPa99QG2dA9C9XT5Doi5N9O3UipwszUQpzYsCgog0O2ZGz/YF9GxfwCUndgXg4+pa5lds3xca/v3B\nZp6auwaA3KwMBpYUMaTbJ6Ghc1GLML+CSMIpIIiIAC1yMhnesy3De7YFImcZ1m7f88m9DKu28fBb\nK7jvjQ8B6NQqLwgMkdAwsKSIvGw9YinpQwFBRKQRZkaX1i3o0roFF5zQGYDqmjoWr92xLzDMWbmN\n5xesAyArw+jXuRUnRp1l6NZWj1lK06WAICISo5ysDAaVRh6X/GrQtqmyirnBfQxzVm7j8XdW8/Bb\nKwBoW5AT3MsQCQ0ndC2ipR6zlCZCAUFE5Ci0L8zl7P7FnN2/GIDaOuf99Tv3uzTx8nsbgMhjln06\nttzv0kSvDoVk6DFLSUEKCCIicZQZXGro17kVXzjpk8cs3w0uScxZtZXnF6zj0ZmrAGiZm8UJpUXk\nVlexNGM5pW3z6dY2n27t8inM1f+iJTz61ycikmBFLbL3DeQEkRsgP9y0a19gmLd6O3PX1/DKysX7\n7deuIOeTwBAVHLq1zae4VZ4GeJKEUkAQEUkyM+OYDoUc06GQzw+NPGZZXl7OkOGnsGrrblZs3s3K\nLZHXqi27mbtqG8/NX7tvnAaAnMwMurZpQWnbfLoHoaE+TJS21dkHOXr6FyQikiKK8rMpyi9iQElR\ng3U1tXWs2bZnX3CoDw8rtuxi9sqt7NxTs9/27Qpy9p1tiA4P3drm06lVnu57kENSQBARaQKyMjMi\nv/Db5Te6fvvuvfuCw4otu1gVLM9euZVn5zVy9qFti/0uXdSfiShtk0+Bzj4ICggiImmhKD+bgfmR\n2S0PtLe2jrXb9rBiy679zj6s3LKbd1Y0PPvQvrCRex+C+x+KW+rsQ3OhgCAikuayP+Xsg7uz/eO9\nDS9dbI6Eh2feXUPUyYf9zj50KMylbUHOvle7whzaFuTSLnifn5OpgaKaMAUEEZFmzMxonZ9D6/wc\nTujausH6vbV1rNn28ScBIriBctXW3by3didbdlVTXVvX6LFzsjL2hYW2BTnBci5tC7KDn/WhIrKu\nVV62zk6kEAUEERE5qOzMDLq3K6B7u4JG17s7lVU1bN21l827qtiyq5rNu6rZsquarVHLm3dV89Hm\nXWyprGZXdW2jx8rMMNrkZ0cFikiIaLMvXAQ/g1DRJj+H7EzNspkoCggiInLEzIyWedm0zMs+6A2U\nB9qzt5atu6vZXBkJD/UB4pNAEQkai9ftYMuuarbt3nvQY7XKy6Jd1KWO6DMW+wWNwsg6iZ0CgoiI\nJFVediadi1rEPGV2TW0d2z7eGwkS+0JFFVt27WXLrqp9Zynqx4zYuquamugbJ6JkGhS+9iKFuVkU\n5GaSn5O1b7kgJ4uC3MircL91WeTnZkaWc4JtcyPrcrMy0vY+CwUEERFJaVmZGbQvzKV9YS4UH3p7\nd2fHnpp9QWJfqNhdzaL3l9OuuAuVVbXsqqphV3UNu6pq2Lizat/yrqrag95XcaDMDCM/J3NfkCjI\nydwXMuqXC3OzyA+Cxb7tDggk9etaZKfOjZ0KCCIiklbMjKIW2RS1yKZn+/3vnShnNSNHDjjkMapr\n6thdXUNlEBg+CQ81VFbVRq0L1gdho7Kqlt1VNWzZtZtd1TXsrqqlsqqGqprYAocZ+52lKMjJ4uIh\nJXzt1J5H1BdHI+0CgpmNAu4CMoH73f2/Qy5JRESamJysDHKyIk93xMPe2jp2RwWNyqoadlfXRoWM\nGnZV136yrqqWymDb3OxwbsRMq4BgZpnAH4BzgNXATDOb4u6Lwq1MRESas+zMDIryMyjKzw67lJil\n2/Mhw4Fl7r7c3auBR4ExIdckIiLS5KTVGQSgBFgV9X41cNKBG5nZeGA8QHFxMeXl5XEroLKyMq7H\nk8apn5ND/Zw86uvkUD/HLt0CQkzcfQIwAaCsrMxHjhwZt2OXl5cTz+NJ49TPyaF+Th71dXKon2OX\nbpcYKoDSqPddgzYRERE5DOkWEGYCvc2sp5nlAFcAU0KuSUREpMlJq0sM7l5jZtcD04g85viguy8M\nuSwREZEmJ60CAoC7TwWmhl2HiIhIU5ZulxhEREQkDhQQREREpAEFBBEREWlAAUFEREQaUEAQERGR\nBhQQREREpAFz97BrCJWZbQRWxPGQ7YFNcTyeNE79nBzq5+RRXyeH+hm6u3uHQ23U7ANCvJnZLHcv\nC7uOdKd+Tg71c/Kor5ND/Rw7XWIQERGRBhQQREREpAEFhPibEHYBzYT6OTnUz8mjvk4O9XOMdA+C\niIiINKAzCCIiItKAAkKcmNkoM1tiZsvM7Kaw60lXZlZqZq+a2SIzW2hm3w27pnRmZplmNsfMng27\nlnRlZq3N7HEze8/MFpvZyWHXlI7M7PvB/zMWmNk/zCwv7JpSnQJCHJhZJvAHYDTQH7jSzPqHW1Xa\nqgFucPf+wAjgOvV1Qn0XWBx2EWnuLuAFd+8LDEL9HXdmVgJ8Byhz9wFAJnBFuFWlPgWE+BgOLHP3\n5e5eDTwKjAm5prTk7mvdfXawvJPI/0xLwq0qPZlZV+AC4P6wa0lXZlYEnAY8AODu1e6+Ldyq0lYW\n0MLMsoB8YE3I9aQ8BYT4KAFWRb1fjX5pJZyZ9QCGADPCrSRt/R74IVAXdiFprCewEfhzcCnnfjMr\nCLuodOPuFcCdwEpgLbDd3V8Mt6rUp4AgTZKZFQJPAN9z9x1h15NuzOxzwAZ3fyfsWtJcFnAicI+7\nDwF2AbqHKc7MrA2Rs7o9gS5AgZl9KdyqUp8CQnxUAKVR77sGbZIAZpZNJBz8zd2fDLueNHUKcJGZ\nfUTkktmZZvbXcEtKS6uB1e5efxbscSKBQeLrbOBDd9/o7nuBJ4HPhFxTylNAiI+ZQG8z62lmOURu\nfpkSck1pycyMyPXaxe7+27DrSVfufrO7d3X3HkT+Pb/i7vqLK87cfR2wysyOC5rOAhaFWFK6WgmM\nMLP84P8hZ6GbQQ8pK+wC0oG715jZ9cA0InfHPujuC0MuK12dAnwZmG9mc4O2W9x9aog1iRyNbwN/\nC/64WA5cHXI9acfdZ5jZ48BsIk9CzUEjKh6SRlIUERGRBnSJQURERBpQQBAREZEGFBBERESkAQUE\nERERaUABQURERBpQQBCRJsHMepiZm1lZ2LWINAcKCCIiItKAAoKIiIg0oIAgIjGxiB+a2Qdm9rGZ\nza+f8Cbq9P8XzOxNM9tjZu+Z2bkHHOM0M5sRrF9vZr8LRhCM/owbzGypmVWZ2Woz++UBpXQ3s5fM\nbLeZLTKzc5Lw9UWaHQUEEYnV7cA1wHVAf+CXwL1mdkHUNr8C7gYGAy8BT5tZCUDw83kiw9wOCY51\nZXCcer8A/jNoOx64jP2nUge4I/iMQUTmQXk0mN1TROJIQy2LyCGZWQGwCTjX3d+Iav890Af4FvAh\n8BN3vyNYlwG8B0xy95+Y2R3AOOA4d68LtvkqcC/QhsgfLJuITOH9p0Zq6BF8xn+4+71BWwmRGRE/\n6+5vxv+bizRfmqxJRGLRH8gDXjCz6L8qsoGPot6/Vb/g7nVmNiPYF6AfML0+HATeBHKAXsHxc4GX\nD1HLvKjlNcHPjrF9DRGJlQKCiMSi/nLkhUSmzo22F7CjPP7hnMrcu28nd4/M3qvLpSLxpv+oRCQW\ni4AqoLu7LzvgtSJquxH1Cxb5zT0cWBw0LQZGBJce6p0KVAMfBOurgLMS+D1EJEY6gyAih+TuO83s\nTuDO4Bf/60AhkUBQB7wYbPpNM3sfmE/kvoTuwD3Buj8C3wP+aGZ3AccA/w38n7vvBgjaf2lmVcFn\ntAOGunv9MUQkSRQQRCRW/wmsB24k8kt/BzCXyJML9W4CfgCcCKwALnb31QDuXmFmo4FfB/ttA/4O\n3BK1/83A1uCzugaf90jivpKIHIyeYhCRoxb1hMEwd58VbjUiEg+6B0FEREQaUEAQERGRBnSJQURE\nRBrQGQQRERFpQAFBREREGlBAEBERkQYUEERERKQBBQQRERFpQAFBREREGvj/dp/IjX+o6UAAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7efc908d94a8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot the convergence of the estimated loss function \n",
    "%matplotlib inline\n",
    "\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(num=None,figsize=(8, 6))\n",
    "plt.plot(loss_sequence)\n",
    "\n",
    "# Adding some bells and whistles to the plot\n",
    "plt.grid(True, which=\"both\")\n",
    "plt.xlabel('epoch',fontsize=14)\n",
    "plt.ylabel('average loss',fontsize=14)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, the loss function converges quickly to the optimal solution."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting the learned model parameters\n",
    "\n",
    "As an additional sanity check, since we generated the data from a Gaussian linear regression model, we want to make sure that the learner managed to recover the model parameters, which were set to weight $2,-3.4$ with an offset of $4.2$.\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The type of \"params\" is a  <class 'mxnet.gluon.parameter.ParameterDict'>\n",
      "dense5_weight \n",
      "[[ 1.7913872  -3.10427046]]\n",
      "<NDArray 1x2 @cpu(0)>\n",
      "dense5_bias \n",
      "[ 3.85259581]\n",
      "<NDArray 1 @cpu(0)>\n"
     ]
    }
   ],
   "source": [
    "params = net.collect_params() # this returns a ParameterDict\n",
    "\n",
    "print('The type of \"params\" is a ',type(params))\n",
    "\n",
    "# A ParameterDict is a dictionary of Parameter class objects\n",
    "# therefore, here is how we can read off the parameters from it.\n",
    "\n",
    "for param in params.values():\n",
    "    print(param.name,param.data())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion \n",
    "\n",
    "As you can see, even for a simple example like linear regression, ``gluon`` can help you to write quick and clean code. Next, we'll repeat this exercise for multi-layer perceptrons, extending these lessons to deep neural networks and (comparatively) real datasets. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next\n",
    "[Binary classification with logistic regression](../chapter02_supervised-learning/logistic-regression-gluon.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "For whinges or inquiries, [open an issue on  GitHub.](https://github.com/zackchase/mxnet-the-straight-dope)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}