weeks_02_03/Notebooks/Week 02 Task 01 - TensorFlow 2.x + Keras Crash Course.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"Lesson 02 Task 01 - TensorFlow 2.x + Keras Crash Course.ipynb","provenance":[{"file_id":"1UCJt8EYjlzCs1H1d1X0iDGYJsHKwu-NO","timestamp":1597001065699}],"collapsed_sections":[],"toc_visible":true},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"cell_type":"code","metadata":{"id":"W15eefa2Q5hA","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1613057445370,"user_tz":180,"elapsed":2588,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"f3033cf2-a48c-4555-b160-adaf6daa130f"},"source":["import tensorflow as tf\n","print(tf.__version__)"],"execution_count":1,"outputs":[{"output_type":"stream","text":["2.4.1\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"tbpWk7OLz7HH"},"source":["<font color=\"red\"><b>A note for the students</b></font>: This notebook's proposal is to be consumed using an informative perspective. The main goal is to present the big-picture of possible things to do."]},{"cell_type":"markdown","metadata":{"id":"zoDjozMFREDU"},"source":["# TensorFlow 2.x + Keras Overview for Deep Learning Researchers\n","\n","<font color=\"red\">Adapted from *@fchollet, October 2019*, last edition, February 2021.</font>\n","\n","---\n","\n","**This document serves as an introduction, crash course, and quick API reference for TensorFlow 2.x.**\n","\n","---\n","\n","TensorFlow and Keras were both released over six years ago (March 2015 for Keras and November 2015 for TensorFlow). That's a long time in deep learning years!\n","\n","In the old days, TensorFlow 1.x + Keras had a number of known issues:\n","- Using TensorFlow meant manipulating static computation graphs, which would feel awkward and difficult to programmers used to imperative styles of coding.\n","- While the TensorFlow API was very powerful and flexible, it lacked polish and was often confusing or difficult to use.\n","- While Keras was very productive and easy to use, it would often lack flexibility for research use cases.\n","\n","---\n","### TensorFlow 2.x is an extensive redesign of TensorFlow and Keras that takes into account over four years of user feedback and technical progress. It fixes the issues above in a big way.\n","\n","### It's a machine learning platform from the future.\n","---\n","\n","TensorFlow 2.x is built on the following key ideas:\n","\n","- Let users run their computation eagerly, like they would in Numpy. This makes TensorFlow 2.x programming intuitive and Pythonic.\n","- Preserve the considerable advantages of compiled graphs (for performance, distribution, and deployment). This makes TensorFlow fast, scalable, and production-ready.\n","- Leverage Keras as its high-level deep learning API, making TensorFlow approachable and highly productive.\n","- Extend Keras into a spectrum of workflows ranging from the very high-level (easier to use, less flexible) to the very low-level (requires more expertise, but provides great flexibility).\n","\n"]},{"cell_type":"markdown","metadata":{"id":"U71NYDeFkUhq"},"source":["# Part 1: TensorFlow basics"]},{"cell_type":"markdown","metadata":{"id":"l2e8-qrcl2kH"},"source":["## Tensors"]},{"cell_type":"markdown","metadata":{"id":"PX6JvH4h0zyY"},"source":["This is a [constant](https://www.tensorflow.org/api_docs/python/tf/constant) tensor:"]},{"cell_type":"code","metadata":{"id":"pGB6GDsfRFJs","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1612738913456,"user_tz":180,"elapsed":6153,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"128afe50-0eca-47e7-ab3d-8c28f9a8885c"},"source":["x = tf.constant([[5, 2], [1, 3]])\n","print(x)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["tf.Tensor(\n","[[5 2]\n"," [1 3]], shape=(2, 2), dtype=int32)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"RX2SB_2O1jx7"},"source":["You can get its value as a Numpy array by calling `.numpy()`:"]},{"cell_type":"code","metadata":{"id":"EwGyHOoq1oWn","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1612738923014,"user_tz":180,"elapsed":615,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"b10e449e-d8b3-4482-de43-afca88299399"},"source":["x.numpy()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([[5, 2],\n","       [1, 3]], dtype=int32)"]},"metadata":{"tags":[]},"execution_count":3}]},{"cell_type":"markdown","metadata":{"id":"PNkno66r1xvg"},"source":["Much like a Numpy array, it features the attributes `dtype` and `shape`:"]},{"cell_type":"code","metadata":{"id":"tSxtblSP13v2","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1612738927502,"user_tz":180,"elapsed":634,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"28d0a382-faa2-47f5-8f1c-f3217b230e11"},"source":["print('dtype:', x.dtype)\n","print('shape:', x.shape)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["dtype: <dtype: 'int32'>\n","shape: (2, 2)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"Oogzv3--2EF2"},"source":["A common way to create constant tensors is via `tf.ones` and `tf.zeros` (just like `np.ones` and `np.zeros`):"]},{"cell_type":"code","metadata":{"id":"9qDlfa8r2Lia","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1612738937643,"user_tz":180,"elapsed":707,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"3d5ca41a-c143-49a0-9eee-24804f4dcf53"},"source":["print(tf.ones(shape=(2, 1)))\n","print(tf.zeros(shape=(2, 1)))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["tf.Tensor(\n","[[1.]\n"," [1.]], shape=(2, 1), dtype=float32)\n","tf.Tensor(\n","[[0.]\n"," [0.]], shape=(2, 1), dtype=float32)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"qzGYEkdcmYbe"},"source":["## Random constant tensors"]},{"cell_type":"markdown","metadata":{"id":"Fk94gREJ2r-e"},"source":["This is all pretty [normal](https://www.tensorflow.org/api_docs/python/tf/random/normal):"]},{"cell_type":"code","metadata":{"id":"jqRrO-Puma7-","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1612738949041,"user_tz":180,"elapsed":657,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"60209acb-8afb-44fa-b5b4-61f202b19ebb"},"source":["tf.random.normal(shape=(2, 2), mean=0., stddev=1.)"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<tf.Tensor: shape=(2, 2), dtype=float32, numpy=\n","array([[ 0.5961736 , -1.8380253 ],\n","       [ 0.58525527, -0.52120256]], dtype=float32)>"]},"metadata":{"tags":[]},"execution_count":6}]},{"cell_type":"markdown","metadata":{"id":"wL0EMPT93SEU"},"source":["And here's an integer tensor with values drawn from a random [uniform](https://www.tensorflow.org/api_docs/python/tf/random/uniform) distribution:"]},{"cell_type":"code","metadata":{"id":"9syARhtj2wbx","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1612738971824,"user_tz":180,"elapsed":716,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"9047ffb6-5968-4b56-e6cf-c00fb11db0db"},"source":["tf.random.uniform(shape=(2, 2), minval=0, maxval=10, dtype='int32')"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<tf.Tensor: shape=(2, 2), dtype=int32, numpy=\n","array([[8, 4],\n","       [8, 4]], dtype=int32)>"]},"metadata":{"tags":[]},"execution_count":8}]},{"cell_type":"markdown","metadata":{"id":"I95066exmbDU"},"source":["## Variables"]},{"cell_type":"markdown","metadata":{"id":"cMflzgPM3Mim"},"source":["[Variables](https://www.tensorflow.org/guide/variable) are special tensors used to store mutable state (like the weights of a neural network). You create a Variable using some initial value."]},{"cell_type":"code","metadata":{"id":"8FvENXmBmcyT","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1612738997400,"user_tz":180,"elapsed":641,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"d5dd2a04-b8e6-4a77-ced1-5fde2627e7fd"},"source":["initial_value = tf.random.normal(shape=(2, 2))\n","a = tf.Variable(initial_value)\n","print(a)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["<tf.Variable 'Variable:0' shape=(2, 2) dtype=float32, numpy=\n","array([[ 0.01360766, -0.05510926],\n","       [-1.4898207 ,  1.6995021 ]], dtype=float32)>\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"YRFwVySi3biu"},"source":["You update the value of a Variable by using the methods `.assign(value)`, or `.assign_add(increment)` or `.assign_sub(decrement)`:"]},{"cell_type":"code","metadata":{"id":"EOCsCNvc3mNR"},"source":["new_value = tf.random.normal(shape=(2, 2))\n","a.assign(new_value)\n","for i in range(2):\n","  for j in range(2):\n","    assert a[i, j] == new_value[i, j]"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"xrSjwl_056j8"},"source":["added_value = tf.random.normal(shape=(2, 2))\n","a.assign_add(added_value)\n","for i in range(2):\n","  for j in range(2):\n","    assert a[i, j] == new_value[i, j] + added_value[i, j]"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"rAIqYQmOl_wR"},"source":["## Doing math in TensorFlow"]},{"cell_type":"markdown","metadata":{"id":"5bmtTepn6SvG"},"source":["You can use TensorFlow exactly like you would use Numpy. The main difference is that your TensorFlow code can run on GPU and TPU."]},{"cell_type":"code","metadata":{"id":"pCZGHQ_XmHuZ"},"source":["a = tf.random.normal(shape=(2, 2))\n","b = tf.random.normal(shape=(2, 2))\n","\n","c = a + b\n","d = tf.square(c)\n","e = tf.exp(d)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"CciCn-7Pvb7S","executionInfo":{"status":"ok","timestamp":1612739103370,"user_tz":180,"elapsed":687,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"87e943c8-a199-4f6e-90fb-ab0425e4d18b"},"source":["e"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<tf.Tensor: shape=(2, 2), dtype=float32, numpy=\n","array([[   1.1601362,    2.3242807],\n","       [   7.3489366, 1122.7648   ]], dtype=float32)>"]},"metadata":{"tags":[]},"execution_count":15}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"fh8LXbqtvdSR","executionInfo":{"status":"ok","timestamp":1612739110454,"user_tz":180,"elapsed":663,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"07f1400a-eaaa-4638-f8d3-91bd09475a3c"},"source":["print(e)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["tf.Tensor(\n","[[   1.1601362    2.3242807]\n"," [   7.3489366 1122.7648   ]], shape=(2, 2), dtype=float32)\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"qdMo_1ffvgw3","executionInfo":{"status":"ok","timestamp":1612739147840,"user_tz":180,"elapsed":650,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"ff5b972a-3c15-4b0e-e3ba-04cb280d79ba"},"source":["e.numpy()"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["array([[   1.1601362,    2.3242807],\n","       [   7.3489366, 1122.7648   ]], dtype=float32)"]},"metadata":{"tags":[]},"execution_count":18}]},{"cell_type":"markdown","metadata":{"id":"ZTJUvSBe4rbP"},"source":["For <font color=\"red\"> advanced topics </font> related to use of TensorFlow 2.x in <font color=\"red\">Linear Algebra</font> please see [this notebook](https://nbviewer.jupyter.org/github/adhiraiyan/DeepLearningWithTF2.0/blob/master/notebooks/02.00-Linear-Algebra.ipynb)."]},{"cell_type":"markdown","metadata":{"id":"5Feq3qWoBVQW"},"source":["## Computing gradients with `GradientTape`"]},{"cell_type":"markdown","metadata":{"id":"JdsmOcrJBWXe"},"source":["Oh, and there's another big difference with Numpy: you can automatically retrieve the gradient of any differentiable expression.\n","\n","Just open a [`GradientTape`](https://www.tensorflow.org/api_docs/python/tf/GradientTape), start \"watching\" a tensor via `tape.watch()`, and compose a differentiable expression using this tensor as input:"]},{"cell_type":"code","metadata":{"id":"FkEAY45IBjPv","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1612741882963,"user_tz":180,"elapsed":669,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"62183465-5842-41d2-bf93-85e2fdadb1ed"},"source":["a = tf.random.normal(shape=(2, 2))\n","b = tf.random.normal(shape=(2, 2))\n","\n","with tf.GradientTape() as tape:\n","  tape.watch(a)  # Start recording the history of operations applied to `a`\n","  c = tf.sqrt(tf.square(a) + tf.square(b))  # Do some math using `a`\n","  # What's the gradient of `c` with respect to `a`?\n","  dc_da = tape.gradient(c, a)\n","  print(dc_da)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["tf.Tensor(\n","[[-0.8598253 -0.763847 ]\n"," [-0.750971   0.6864368]], shape=(2, 2), dtype=float32)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"W8UpqFx_DDbV"},"source":["By default, variables are watched automatically, so you don't need to manually `watch` them:"]},{"cell_type":"code","metadata":{"id":"OtH3FuvDDOAY","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1612741911407,"user_tz":180,"elapsed":734,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"5f0517aa-5b5a-4741-c864-0b2c2470074b"},"source":["a = tf.Variable(a)\n","\n","with tf.GradientTape() as tape:\n","  c = tf.sqrt(tf.square(a) + tf.square(b))\n","  dc_da = tape.gradient(c, a)\n","  print(dc_da)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["tf.Tensor(\n","[[-0.8598253 -0.763847 ]\n"," [-0.750971   0.6864368]], shape=(2, 2), dtype=float32)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"dFlBGjuEDbt-"},"source":["Note that you can compute higher-order derivatives by nesting tapes:"]},{"cell_type":"code","metadata":{"id":"cjPcY0OIDhEA","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1612741929789,"user_tz":180,"elapsed":640,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"b490ad98-f576-4004-c873-1eee31e6370f"},"source":["with tf.GradientTape() as outer_tape:\n","  with tf.GradientTape() as tape:\n","    c = tf.sqrt(tf.square(a) + tf.square(b))\n","    dc_da = tape.gradient(c, a)\n","  d2c_da2 = outer_tape.gradient(dc_da, a)\n","  print(d2c_da2)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["tf.Tensor(\n","[[0.11678296 0.5655415 ]\n"," [0.31122163 0.70031565]], shape=(2, 2), dtype=float32)\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"KC5RgwGeBP-9"},"source":["## An end-to-end example: linear regression"]},{"cell_type":"markdown","metadata":{"id":"Owbx4mlEErNN"},"source":["So far you've learned that TensorFlow is a Numpy-like library that is GPU or TPU accelerated, with automatic differentiation. Time for an end-to-end example: let's implement a linear regression, the FizzBuzz of Machine Learning. \n","\n","For the sake of demonstration, we won't use any of the higher-level Keras components like `Layer` or `MeanSquaredError`. Just basic ops."]},{"cell_type":"code","metadata":{"id":"Uhitqoj2FH8U"},"source":["input_dim = 2\n","output_dim = 1\n","learning_rate = 0.01\n","\n","# This is our weight matrix\n","w = tf.Variable(tf.random.uniform(shape=(input_dim, output_dim)))\n","# This is our bias vector\n","b = tf.Variable(tf.zeros(shape=(output_dim,)))\n","\n","def compute_predictions(features):\n","  return tf.matmul(features, w) + b\n","\n","def compute_loss(labels, predictions):\n","  return tf.reduce_mean(tf.square(labels - predictions))\n","\n","def train_on_batch(x, y):\n","  with tf.GradientTape() as tape:\n","    predictions = compute_predictions(x)\n","    loss = compute_loss(y, predictions)\n","    # Note that `tape.gradient` works with a list as well (w, b).\n","    dloss_dw, dloss_db = tape.gradient(loss, [w, b])\n","  w.assign_sub(learning_rate * dloss_dw)\n","  b.assign_sub(learning_rate * dloss_db)\n","  return loss"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"qC1fp3BYJeXo"},"source":["Let's generate some artificial data to demonstrate our model:"]},{"cell_type":"code","metadata":{"id":"6ocAkrliMVAQ","colab":{"base_uri":"https://localhost:8080/","height":282},"executionInfo":{"status":"ok","timestamp":1612742637451,"user_tz":180,"elapsed":1810,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"3365b5f9-fc81-4500-ffbe-3eaec6d1c40d"},"source":["import numpy as np\n","import random\n","import matplotlib.pyplot as plt\n","%matplotlib inline\n","\n","# Prepare a dataset.\n","num_samples = 10000\n","negative_samples = np.random.multivariate_normal(\n","    mean=[0, 3], cov=[[1, 0.5],[0.5, 1]], size=num_samples)\n","positive_samples = np.random.multivariate_normal(\n","    mean=[3, 0], cov=[[1, 0.5],[0.5, 1]], size=num_samples)\n","features = np.vstack((negative_samples, positive_samples)).astype(np.float32)\n","labels = np.vstack((np.zeros((num_samples, 1), dtype='float32'),\n","                    np.ones((num_samples, 1), dtype='float32')))\n","\n","plt.scatter(features[:, 0], features[:, 1], c=labels[:, 0])"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<matplotlib.collections.PathCollection at 0x7fb557effba8>"]},"metadata":{"tags":[]},"execution_count":23},{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd3gUVReH3zuzNQkhhN47KB0FC4hgw44Kdqz42VDsIioqIhZAEaygiCKiUlQEFAUVUOldBKT3mkbKZuvM/f5YElh2NwmkkIT7Pg+PZvbOnbvJ7m/OnHuKkFKiUCgUirKLdqoXoFAoFIrCoYRcoVAoyjhKyBUKhaKMo4RcoVAoyjhKyBUKhaKMYzkVF61SpYps0KDBqbi0QqFQlFlWrFiRLKWsevzxUyLkDRo0YPny5afi0gqFQlFmEULsjHRcuVYUCoWijKOEXKFQKMo4SsgVCoWijKOEXKFQKMo4SsgVipMg9UAa//y5nuS9Kad6KQrFqYlaUSjKKkbAYMT9o5k3aQFWuxW/18/5PTrQf3w/bHbrqV6e4jRFWeSKUseBHYf4pP+XDOzxFpOGTSMzLetULymXia9/x/wpC/F5/LjSs/F5/CyasYLPnp94qpemOI0Rp6KMbYcOHaSKI1dEYt3CjQy4/DUCPoOAP4DNaSOmgpOPVwylSu3Kp3p59Krah4yUzLDjjlg70zMmIIQ4BatSnC4IIVZIKTscf1xZ5IpSg5SSt/t8iMflJeAPAOBz+8hIyWTci9+c4tUFyc50RzzuzfZhmmYJr0ahCKKEXFFqyEzN4sCOpLDjpmGy5OeVp2BF4bQ4v1nE443bNUDX9RJejUIRRAm5otRgdUTfLHTGOkpwJdHpO/JenHEOdEtQtDVdwxFrp9+H/zvFK1OczqioFUWpwRnroOMV7Vg2axUBv5F73B5jo0ffyzECBn9OXcy8SQtwVnBy9f2X0rrLmSW6xsZtGzBm9dtMHv4jm1Zso1Hb+tz8TA/qNq9d7NcO+APM+XI+v331J1a7lavvv5QLep6r/PIKtdmpKF1kpGby/OVD2PXfXjRdI+AL0Pn6c3jm80cYeO2bbFi0CY/LixBgc9q5bcD19B5446leNgAZKZlM+2AWK+b8Q7V6VbjxyWto3rFJkcxtGAYDug9hw5LNeLO9QHCD9eLeXXhy9INFcg1F6SfaZqcSckWpQ0rJllXbObAjicZt61OrcQ3+/mEJQ+96H4/LGzLWarcyYduHVK5Z6RStNkjaoXQeav8sWalZ+Lx+hBDYnFaeGvswF996QaHnXzxzBa/fPhJPlifkuM1p4+MVw6h3RvE/EShOPSpqRVFmEELQ9KxGdOl5LrUa1wBg4Y/LwkQcwGLVWfX72pJeYhjfvvUDGckZ+Lx+IHgz8mb7eL/v2NwInMKwfPaaMBEHEMCauf8Wen5F2Ub5yBVlgrhKsWi6hmmEhvgJIYitGHNK1iSl5L+lW3ClZ7N45ooQv34ORsBgz6b9NGhZt1DXqlQtHqvNgt8XelPQLBrxVeILNbei7KOEXFEmuLLPxfz8yW943b6Q45pF4+zubUt0Lcl7U5j2wS/MHD0bvy+AbtHwZIU/LQAEAgZxlWILfc3L7urKN2/9AKFvH13XOe+aswo9v6Jso1wrijJBw9b1eeS9PsFMz3gnMRWcxFepwFu/DAyrcXJodzIrf/uHgzvDY9ILy4JpS7mn2WNMGjotmKLv9uHO9BBpr0m36Jx5blOq1ErMPWaaJit/+4cZo2fz74L/Ip4Xiap1q9B7YC8sNgtWhxVHrJ3EmgkMnfMSdqe9yN6fomyiNjsVZQpXRjZr5q3DEeugzYVnYrEefagM+AMMvet9Fvy4DNuRglYdr2zPC18/USQFrTzZXm6qfl9EX30Omq7hjHMQ8Ado2Koer80YQELVigAcTkrn6W6vkLQ7BdMwEZqgYet6DJ39Es44Z9Q5jYDBSz2Gsvav9XiyvVhtVoQmeOOn52nbrVWh35ei7BBts1O5VhRlitj4GDr16BjxtS8HTWbR9OX4PX78nuCm47JfVvHZ8xN5eMQ9IWOzDrv48cNZLJ65ksQaCfR8/GradmuZ57XXzFuHpuf9ENu6yxnc8dJNJNasFBZJMuqhT9m75QDGMb70Lat28PnAb+k78t6QsaZpkpXmIibeyezx81gz7198R96T/8iG6pBb3+XbvZ+ojFJF0Qi5ECIBGAu0AiTQR0q5qCjmVigKyozRs8N86D63n58//S1EyLMOu3iw/TMcPpieK44r5vzD/UN7c90jV0adPz8Rt8fY6XpzZ9pdFG4lGwGDRTOXh4g4BEV5zoT5IUL+6xdz+bT/BFwZbnSrjuE3CPjCI1+8bh9bVm4/oVj1gzuTmPvt33hcXs656izOPLepSigqBxSVRT4K+EVKeaMQwgacmjACxWmNO0J4HhwtaKVpQSGe9v7PISIeHOPlk/5f0f2ei3LLAUgpWbfgP/ZuOUDjtg1o27VFntf3ur28/+hYxr34Nb2evIbbBtyQm8pvmibSjOzG9Gb7mPDqFOq3rIOU8P6jY/FmB29IkQQ8FwlCK7gI//HNX7zzv9GYhonhN5g6YibdbunE02MfVmJexim0kAshKgIXAvcASCl9hO2tKxTFT8tOzfln/vqw4806Ns4VcQgm1xwr4jn43D6e6voyb/0ykI3LtvD6rSNxZ3nQdA3dotOiUzNemPg4Q259FynJzbDMRQbFPyvNxYRBk9m9cS/PjnuEBdOW8e/fG6harzIHt4dvwPq9fr58dTLOOAd+rz9iGGMkHLF2mrRvWKCxrnQX7/xvNL5jnli82V7mT17IRbd25uzLSjbyR1G0FIVF3hBIAj4XQrQFVgCPSyldxw4SQjwAPABQr169IrisorSzYs4aJg37kZR9qZx1SRtuGXB9SARHDmv/2sC3b/3AgR2HaN3lTG57vifV61fNfX3VH2v5Zdxc/F4fF916AZ2u7xjRL/zIqD480WUgfk9QDHWLjtVuod8HoQWtKlVPiLrmrat30O/8Fzi4Iyk3Zt0IGBgBgzXz19GoTX0m7vyY+1o+GS7kx2CaknnfLmTzim0k70nFneXBms+Ga7Qnimg8P/HxkBtUXqyY8w8Wq47vuCq8HpeX37/+Swl5GacohNwCnAX0k1IuEUKMAgYALx07SEr5CfAJBKNWiuC6ilLMzDGzGf30l7lit3fLAf745i/GrH47pEHEH9/+zYj/fZzrSti7+QDzJi3ko+VDqdW4Bp89P5FpH8zKjRRZ9stq2l/Smld/6I8QAiklG5dtYc+m/RgBg4at6rF1zU7sMXbaX9KaB4bdSc1G1UPW1v2ebiz7dXWYvxpAmpL9Ww9GfE/SkEx7fxYdurfl8KH0fH8HpmGyf+vBXAs7Z5OyKGh6diPaX9y6wOOj+feFINf9oyi7FIWQ7wH2SCmXHPl5KkEhV5ym+Dw+Pnl2QojFavgNXBluvnlrGpfcfgELpy/Harfww/uzckUcgtZvdqab8a9M4p7Bt/L9qJ9C3CAel5dF05czqNdwHv/ofl66bii71u9BmjJko9NLMGLl2ocvzxVyv8/PvEkL+aDfZ2EZogXFNExmfPxrcEu/ABTUTZIfVrsFvzeAPcaG1WblufGPntD5Z3dvG/E92512Lruza5GsUXHqKLSQSykPCCF2CyGaSyk3ApcA4Y5KRblh04qtTBo6jb1bDtDqgjO4+dnrqFa3Su7ruzfui7h5ZvgNfv/qT379fC4+tw9N1zACka3iRTOWk1gjASMQWXAXzVjOhsWbyEp1haWt5+DN9jH6qS/4ZM07bFm9nee6v0Z2enahxXXL6h2FOv+EEND0rIb06Hslm5ZvoW7zWlx6Z1cqVIo7oWmcsQ4Gfvskr908AiEEhmEgNI1r+15Omwvz3sRVlH6KJCFICNGOYPihDdgG3CulTIs2XiUElV0Wz1zBkFtG4PP4kVJiserYY+18uPQtqtROZNKwH5k19neS96YW+loWmyXvqI0CoOkas7zfcFvdB0ndf7jQawJoclZDtq7aTknl0lntVr7ZPZqKedRUWT57DZOG/sCh3Sm07daS3i/2CtlnyCEjJZO/v1+Cx+Wl45XtSqSOuqLoUGVsFYVGSsnt9R4KE2lNE1zQ6zxS9qWyecW2iBEhpwp7rJ1GreuxYfHmIpuzYpUK+Dz+E96cPFmccQ7eX/Im9c+sE/H1n8f+xkdPfJHrytItGo44B6NXDqdGg2olskZFyaDK2CoKTdrBw6RH6CBvmpIVs9ewdc3OUiXiAF6Xt0hFHCA9OZO4xMIXwiooQhPUalw94mt+n58xz3wZuh8RMHFnepj42tSSWqLiFKOEXFFgnBWcRPMnWGyWIo3KKO0k7UopsrmEJrDYIm9X6VadColxvNrrbVbMWRP2+oHthyImGpmGyeq564psjYrSjRJyRYFxxjq4oOd5YfHQjlg7F9xwTpEUpjodqd+iTvR65VJycEcSS35ayaAbhjNp+I9IKXOrJlasEh+1cUWV2uEx+4ryiSqapQhh+7+72LB4M5VrVaJD97ZhMcZPfvIg2RnZrPp9LbpFx5PtRZqSNfPWgQhal9FS0XOw2HQsVkueVQRPJ9p0a8n0D36J+NqxUTuebC+fPT+RcS9MDHZROrsRbS5sQeO2Ddi6ZmfIE5EQgppNauDz+LA5bMX+HhSnFrXZqQCCzX1fv20kS39aCQI0XSc23smI+YPDEmoAJr/9I5/2/+qEr2Ox6Jx/fUc0TWP+5IVFsfQyT06M+MniiLVjmiZ+jz/E82V32mh6diPemfdqgTNAFaUbVcZWkSczx8xh6c+rQpJqPC4PT3QZSGx8DMl7U0moXpGbn+lBu4ta8dkLX5/wNaz2YB1tT5aHFuc3468IrdtORwoj4hBMktI0DaFpyGN+n163jy2rd7Dq97UqBb+co4RcAcDM0bPDaodIU5K6/3Bu/LU7y8Pop8bTsE09zCiJOnmR8+i/7JfVLP91TYG74yjyxzQj/z08WR7WL9ykhLyco563FEDBrUKv28d/S7YU+npKxEsGR6ydyrUqneplKIoZJeSlGMMw+PGjX7iv5ZP0bvAwHz0xjvTkjGK5VrdbO+dbnU9R9tAtOl1v6XSql6EoZpRrpRQhpcTj8mBz2NAtOm/f9zF/TV2c6/KYMXoOC6Yt49O1I4ipEL3HY0FYM28d4wZ+za4Ne6nTrBa3PX8DtRpXZ9d/e/ONOlGUDnSLFrUWDUDNRtV5ecrTxMarPi/lHWWRlxKW/LySuxo/yg2J93BdxbsY3udD5k9aEOK3DvgCpCdnMnv8vEJda/nsNbx49RusX7iJrDQX/y3ZzBu3jeSmZ3ucUMcZxanD5rDy4jdPRg0t1HSNj1YMLXDjCUXZRgl5KWD94k28dvM7HNhxCCNg4nX7+OPrvzGNcMvYm+1l9dx/C3W9MU+PD+tt6XX7GP/ypJPaxFSUPBabBSklFSrHQYR7b8PW9YirWHJlBBSnFuVaKQV8/fp3ITW5IXqvRovVQq3GNfKdc/vanaxftInKtRLpeEW7kMSeXf/tjXhO0u6iSztXFA26Nfh3O7YJhtVuJbFGAsPv/TAsqcpqt2C1W3lmXN8SXafi1KKEvBSwZ9O+Ao+1WHWufbh71NeNwJHEnlkrAdB1HWcFByPmD869AVSqkUBKEZSZVRQvsRVjeGnyU3z63Ffs3XIgt85N7SY12LN5f9jNX7NodLi8LVabla8GT6XbLZ3o0uu8Yu0AtHPDHia8OpkNSzZTq1ENeg/sRbuLWhXb9RSRUUJeCmjWoQn7tx7EzGOTUbfqVK5Zif5fPErNhpEr4UGwxdrSWatCvuRul4fBN73D6JXDAbj2wcsY/8pkFQJY2hEwe/w8Xv2hP0l7UtizcR8NW9dj1R9r+eKlSWHDzYDJ0lmrMQMGUgZ7ps4a9wdv/PxCxB6nhWXbPzt5/IKBeI+UaTi0M5kNSzbxzLi+dLu5c5FfTxEd5SMvBfQe2Aub057nGKvNwrgNI2nbrWWe42aO+S1iYs/2f3Yy/pVJHE7O4Nuh05SIlwFch7OZ++0C+nZ4jjrNanJFn4tp3rEJFSrFYY1SLdHwG7lp+h6Xl/WLNrF4xopiWd9nz0/E6/KERDl5s3189MQXUROUFMWDEvJSQP0z6/Dun4Npf0nrqFEjmq6x7Z9dYcellKyYs4bXbx/JqzcOJyNCvXAI1gz/+o3vubXWA6pYVRlCmhJ3ppsZH8/OPXbhTedH3OCMhCfLw4JpS4tlbRuWbI5Y1TgrLSvq51BRPCghLyU0ad+QYXNe5qxLIndGN/wGFRLD+zSOeWY8g3oOZ963C/j7+6WkJ2dEvRmYhhmxR6aidOP3BVi3YGPuzxUqxTFkxvNUSIwjpoKTmHgnjlg7dmd4KKKma8RVKp7olcQaCRGPCyGIUbHrJYrykZcyej5xDf8u2BjiHtF0jXpn1qZO05ohY/ds2sf0j34NSa83iqhru6J0UfeMWkDwCczn8dGmawsm7/+U9Ys24c5yk53lYcR9H4edZ7VZuLLPxcWyplsH3MDIhz4J+azanTYuu7ubqk1fwighL2Wcc2V7eg/sxVeDp2C1WzECBjUaVmfwj8+Fjf3zu8WFrpynKP3oVp3r+13Jt0N/YNLQH8nOdJNYI4H7h9+JaZiMfGAMmkXDfyRkVQiwOqxICX1H3UvD1vWLZV2X9O5Cyr5UvhryHYJgxFS3WzvTd+Q9xXI9RXRUPfISZse63SyYthRd17ig13lhVnYOWYddbFq+lYRqFWnYuh5ChLtLnrlkEGtUO69yjdAEL099hp3rdvPNmz+EWL82hxXTlBFzDiw2C9c8eCmPjLqv2Nfo8/g4uDOJxBoJxKokpGIlWj1yJeQlyITBU5g0dFrwi6cJdF3nvrdup+djV+eOkVLiyfZic1gRQpB2MJ3YijE4YsKjWnpV60NGstpUKq84Yu28OetFzjyvGT0r30t2pvuEzrc6rEzc/hGVqkf2ZSvKHtGEXG12lhDb/93FpKHT8Lp9GIaJ4TfweXx8NmAih3YlAbB45grubPQIN1S6m2vj7qBH/J3c2agvPSvfy9C73sfnORobnrwvFfcJfrEVZQuv28dHT3yOKyP7pBpbW20WNi7bWgwrU5Q2lJCXEH9/vyTXh3k8C39czr8L/mPILSM4uDMJI2Di9wbwZvvwewP4vX5+++pP7mj0CCn70/D7/Dx2/gtR51OUD6Qp2bZ2F8t+WY3VEXnzMK8iZ6ZhklhTWeOnA0rISwhN1yL6uRECoQm+em1qWCGr40k7cJgnOr/I3z8sJeuwC1ROT7nH8Bu8ded7eFyesNdsDistz28eNeyweoOqND2rUUksU3GKUUJeQlx443m5BZCOxe/10/7iVgWut5J2KJ2lP6/CnRn+xVaUUyS5lTCFJrDaLTQ5qyFDZj7PO/NfZcBXj9Hi/GZYbBZsDis2h5XmHRvz1q8vRTYeFOUOFX5YQtRtXptrHrqM79/9KeS4EPDeI2NpelYjDu1Mzjd1XpoSe4wNm9OKz33iflNF2cZqtTBoWn86Xt4u99gFN5zLBTecixEw2PXfXmIrxlCtbpWI57tdHj57fiJzxs/H7/Nzdvd2PDLqXmo0qFZSb0FRDBSZRS6E0IUQq4QQM4tqzvJG+qGMsNRq05BsWLKZK/pcHPER+XiMgEnd5rVU15fTlEDAIPNI+vvGZVuYPPxHfhn3B650F7pFp2GrelFFHOCFq17n509/JzvTjd8bYOlPK3jknAFkpmWV1FtQFANF6Vp5HNhQhPOVO4KlSMOPW6w60pS8PXcQiTXzbpRrBAzGvfg1Gakq7PB0REpJi87NGXLrCJ6+aBDjBn7Dh4+P47Z6D7F+0cY8z920YitbVm4PiYAxTYnX5eXXz+cW99IVxUiRCLkQog5wNTC2KOYrr7Tt1gJLhKp12RluBvUczuu3jyxQc2Wf24/hV9XlTkuk5I3bRrJ45gq82V4Mv4HH5cWd6WFQz+EYRvQSDTvX7Qn68o7D6/axacW24ly1opgpKot8JNAfUOoSgZT9aWxYspnud1+Es4IjYsiYETDYv/WgqpWiyBMpg1UHj28qAeDJ9rL5iCDv3bKf3776k5W/r80V99rNahKpXKHNaaNx2+JJ41eUDIXe7BRCXAMcklKuEEJ0y2PcA8ADAPXq1SvsZUsduzfuZfpHv3JwZxJnXdqGy+/phm7Reeuu91k8YwVWh5WA10+n6zqyYNqyk0rwUCiAqGGnniwvj53/IvYYG36vP9iYWUB8YgXenjuIM89tSr0Wddi+ZucxdVkENoeVK4qpsJaiZCh0ir4Q4k3gTiAAOIB44Hsp5R3RzilvKfpLfl7JazePIODzYwRM7DF2KlWvSOsuZzJ/8kJ8nqOirWkaUkrV2EFRYmiaoHG7Bny0fBiujGw+fHwc875dQMBv0ObCFjz+8f3UbV77VC+z1COlRLq/A9dYMNPA1hFR4WmEpWGJraFEaq0cscifkVJek9e40izkpmkyc8wcpn/4Cx6Xl843nEPvF3sRX7lCxPGGYXBrrQc4nBTq27baLRgBIzf+t6BYbBYsNgueLBUnroiOpmtYrDq+nCe7fD5mNoeVz/8bRbV6VYPDjxgTmqZSSQqKmTkCXOOBnNIYGogYROXpCEudPM+V3gXIzLfB2A56XUTckwjHiT8FqVorBWR4n4/45NkJ7Fy/h4M7k5j+0a/07fAc7qzIdU32bNqPJ4K/0u8NnLCIAwR8gYhZfApFDla7hUZt6vP6Ty9wx0s3IgrQLkjTtZDPqRCi2EVc+jdgpvXDTLoEM/VBpP+fYr1ecSLNLHB9zlERBzBBupGuT/I+1/sXMu1hCKwDmQ2BjcjDT2C6fy6y9RVpQpCUch4wryjnLEn2bT3An8e5QgK+AIeTMpj95Xyu63tF2DnOOAdmlEgBi80SscRoviivi+I4dKuGpulYrDoX334Bbbu1ZNyL35C6Pw17jC3f9n0xFZzUaRa5ZHJxIH2rkal3AV5AgrEHmbIIKn2MsJfBxszGNhBWkMf/ng3w5d0TVWYOBY43zjyQORScVxXJ8pRFfgyblm+NmEbvzfayeu6/Ec+pVrcKDdvUR9NDf5WOWDs3PHYV9hjb0TRplS2tOEk+Xjmcn91fMz1jAvVa1OGd/41mw+JNHNyZlKeIW2w6jhg7z33Zr0TdKDLzdYLilWOVSMCDzBhScmsI7ET6NyFlEQTTaTVBRqqFJMCST8RPYHvk4+YBpCyaoAeVon8MVepUjpKwY6FWoxoRz9m/7SBel/do13ARHN/97m7cP/QOut3SicE3vsOhXfmn3ysU0Vj+yxrqnVGblb/9w9gBE/F78haAitXiOfvSNlSvX5Wr7r+05FPw/esjHze2IWUAIYpPemRgOzKtLxh7QQT92FR8G2HvdNJzCr0q0t4NvPMJPmXkYEfEPpj3yXp1MPZEmLQiRSXBSsiPoWWn5lSunci+LQcwjaN3cYtV55qHLgsbbwQMnur6Min700IMD13XuO35GxBCsH/bIdKTM5SIKwpFekoGdzTsS2aqK18RB8hIzmTAhMdCimatW7iRX7+Yi9/rp9stnTnnyvbFV1RLSwAzKfy4iAXCn3qLCin9yNQ7wEwGZPB7KbODPuqqvyD0k3cviYThyPRB4DlSL0mLB3t3CPyL1KtFnzu2H2QMItS/7oS4h4vs969cK8cghODtPwbRsnNzrHYLdqeNavWq8NqMAdRsWD1s/Io5/+DKcCPNUJE2TZNfPp+LO8vN5wO/ztd/qVDkx59TFpGyNy2k1Vte2J22EJEYP2gyz3V/jV8++4PfJvzJkFtG8EbvUcVnYMTeBziPO+iAmDuL/OYhTRdm1jjMlNuRqb3BzCT80dpAZk8t1HWEcKIlDEVUXwZxz4KZAe7vkRlDkUndMV1fRjxPi7kBKvQHkQBYQFSAuH6ImHsKtZ5jURb5cVSuWYkR8waTdigdb7aX6vWrRv3gpexLDbHcc/B7A2xYtIl+kxayb8vB4l6yopxz7lVnsWb+ugKJbs36Xnrcm0abCxIwXZMRMT34ZdwCJr42NeR8j8vL4hnLWTNvHe0ualXkaxYx9yCNFMieAEIHGQBnL0TcY0V6HWm6kCm9wNhH+IbisfjAPFA0FzUzIOvt4JzHkjkcab8QYWkQdooW2xsZcxtIF4hYhChaG1oJeRQqVauY75gzz2sWMeUZYPmcNWhCKJeKolBoFo2bnu3B2r/zr0d3dtcMXv5sJxabxGI5CJmvc3j3SMYOqBPxc+jN9rJw+rLiEXKhIeKfRcY9HBRZvSZCi5yLURike0oBRBwgBmE7/8TnN1KQ7u/B2IOwdQRHd/DMIXLkgoH0/IKIeyjiXEJoQWu8GFCulULQoGVdzru2Q8RCWGbAJKDqpigKSaXqFY8YDHmP0zTJs+/txhFjYrHkDHbjcKRw3b2Rm5Zouk5MhePdH0WL0OIQ1mYRRVwGtmGm9cU82BEz6TJM17cnbvh4fid/EbeDpV5QhE8A6VuDTL4Ust4D9zfI9IHI5OuR0k3kslIy+ORxClBCXkien/gY9c44sfRmR6ydBq3qRipEp1DkoukaHS9vh81u5alPH8LutIWFueZQu5EXR0y4uNgdkguvTY94jm7VufSOC4t0zQVFBvYgU24E7+8g08HYCZlvBrMfTwS9MpGtYxtYmoClGcQ9ikj8FiHyr/efuz4pkelPBV0huVEq2WDsgsBOIgu5FeEID4ooCZRrpZDouk6T9g3ZvnZXga0Jj8vLjn93B6sgKteLIgIWq44zzsmdL98EQNebO1G/ZV1mjplNyr40tq7eTvK+tNwIFintWKyRLQN3Vrj4a5pGvw/uo06zWie9RjOwC9JfDWYsajUg/jk0e8HcF9L1CUg3oY8absj+Ehn3IEKLDx0vZcS9KhFzJ9Izl9CIEA30WojKP538xqq5D4xDEV7wgef7CMdtEHsvwtr85K5XSJRFHgGv28vGZVs4sCPSHzKcqx+8DJszcpfzvDg+2kVxelOnWS2s9qBtZbVbuXvwzbm1UQDqNq9FfOUK/DN/HQd3JpFQNZ5q9apQ94xaXPnAnViczTn+K+1168z8smrIMYvNwtA5L0ZfVeYAACAASURBVHHFvSdX8VBKiZnxDiRfCv6/QKaCsR7S7o4auRGGfxUQwfUorCEJNNK7EDPpKuTB5kEXTNbHuQk+0rca6RoPWiLBkMbYYMy4Xh9R6bNCRsdYie7PChz5d+y64xBxTxTieoVDWeTHMXPMbMY8OwFNEwT8Bk3bN2TQD8+SUDX65meL85rR5/Xb+Oz5r9GtumqMfJoSG29w2U2pNG7tZts6J3MmVyIrvWBfMd2qc3BnEn5vUCDcWR4+fe4rEmsm0qXnuQCM6vspf3z9J/EJLoxYnaTdKThi7Xz061DqNq+NNDogU+8MVuaTAAGk/XK2bgR7zH5sDkGzNlnc+FR32nY7ectRZk+E7Cj1RTLfwHTekX8Wqd4AApsIE0vpB70mUvqQnjmQPoBc14ZMh6yPkWYG0nIGZLxEbgkA9OB/K76HsHc5IRGXgd1I16fBm4veEBH3AMLaCmlpBoH1FKjNgnQH3UMRIlZKgiKtflhQSmv1w9Vz/2XgtW+FxOrqVp3mHRozasHrYeO3rN7O1HdmsGfTftp0bUGrzmfw7oNjOHwosk9SUX6pUc/Lez9txuY0ccZIPNkCr0fj8aubsn+nPd/zRZQIp7rNazFuwygOJ6Xz9h230+/N7cRXMhAarPozjrefasD5117MM+MeAQhaq/7lQbeAtS3CUheAQ9t+opL9JTQ9x0WhIRLeRxTQFZKDlBJ56DyQadEHJU5Hs50Rfq5xCMyDoDcEYzsypTehG5U2sLYG24XBG4X0ENFqx36k7kmkPqNxiGq/IbTEgr2fwFZkyk1HrhUg6G+3Iyp9AHpDZMqtIFPIX8ztiKpzEHrkDPCiIlr1Q2WRH8PUETPCEi4Mv8GW1TvYt/UAtRof/SMFa5C/g8/jR5qSrau3M+Wd6arg1WlK3yF7iUsw0I8kLTpiJFa7waNv7OHF3o3zPT+aQXVgRzA7MmPvDwz8ZAs2+9FxZ3XNZPAXWxn7Zm3MjDfAMysocM6bEbF9cjf3pJlGldgXjvikyf2MysMPQdV5CC3vPrG5a/SvQ7q+yFvEj71Azk9mNjL9SfAuAGELRnbEPQAJ70HGK0eyMA0gAP514M+7CBVoIKNFhGUh0wcjKo2MsP614P8X9Npg64wQOjJz+JENzeNqwqS/gqj6B+jVIJCa/3osTYtdxPNCCfkxpOyL/AG1WHUOH0rPFXIpJe8+MDqk3ZYKNTy9ObtrZq6I56Dr0L5LFkFxODl/bcMWTszk66hdY1PYxrjVBg3PzOblT/+GbC9wJHU/6yOkdz7E9AZLY/CvjmxgSBkU/5jb812HmT0dMgYSWmckCsZW4MzgJcxsZMo1R2uN5BSecn2KiB8CVech018Ez3fkiGj+mORpMXl/xnSdgxYbfF9S+pBpDwRdJ1IGE5S0SpD4NfiWRZ7LPIj0r4LAViI/FWggHIAAURFR6f0CrLv4UEJ+DB2vaMfO9XvC2rAZAZOGbY5WOEvZn0ZmaqTHOsXpSsAvsFjDBSHgFxSm7GXfwSuRAQ8CI+I0QggqJPjIFXEAPOBfgUxfF/xRq0JkAfaBGXQDSimDPmjhQAhHyCgpfZD5CgUTWSDjNaTjcoSwIlPvj1wwSrqRGW+CrfMxIl4QnBBzB/j/Af+S6MMyhyItdRH2LsisMeBbeXT9EjA8yMPPBmu/yMwIEwQg66MjLpwI79tyBiL2btCqge08hCi++jEF4bQX8k0rtrLyt7XEVoyh+93d+PWLeWSmZuZuOtlj7PR54zacsQ6S96bw3cifWDxjeUjNcoXij+8rcelNaSGuD59XMHdawknPWbW2j0YtsxF5iJzNAUJEKq8KucJlHiTyzcQOtvORvmXIw8+DuRcwkVp9SBiJZmsRHBb47wRX7gPfIkzfTggsiz5MJkPy5RRcxOMh5gaIewJBJjLp8uDNJyJupOtThL0LuKcSfhMyjrhw8rjJ+pYSFp0CgB0clyKcNxRw3cXPaSvkUkre7vMR86csIuAL5GZn9v/iETav3MbSn1eRWDOBXk9eS4fubdm7ZT+PnDMAj8urOt0rwvjk1Vo0PNNDgzNyanALdm2yM+aVk++FWaGigeEX4IgmdM5g3HJgA3m7PHKMDo3cTTvhBFs3pKgIKdcRInTmDkjtiRn/BvgWg2cu4Cr4wqVEepdGj2wJGRvJGj4e/ci6syD722Chqpg7oNJXkHoXEMVnH8jJaI1mdBnkfRPxgKgc7OqTG6duBS0BERO1JfEp4bSNWln44zLevGNUWGXC2IoxTDk4FqstNC781V5vs+DHpSr2+zRE0yQ1G/hwZWgcTs4rX0ByxlnZ1GvmYfdmBxtWxAACi9XE7pS4MjQQosAGqMVqMmntOuLiI0RMiARw3sO6VbVp1nAgVltBK2xqYDkTYu4O1ut2fQpk5zFecHI7+DqRfctFiRViHwPXB0S+kQmwdghubnp+IlzQC/DetNqI+BeQrs/BTAXHxYjY/xU4KqaoUVErx/HrF3MjlpeVUrL2zw2cdWkbfF4/Xw2ewqyxv4c1V1acHnS64jBPDN+DzSHRLZJ1S2N54+H6ZKRG+uoI/lsZy38rYwGw2k0eHryHy25KQ2iQcsDK+wNqs3ze0azFhKrxUT9bAb/GhwPr89x7+wlW2jMJht7FYhgQSHmPpg0ku7fYccTYqVrLh8UKQuQlTiaYWeD6CIwdBfgN5DVXXmJ9IiJ+JAa8IPHaIfjB9T7E3BOsssjxfXVl0H0S2BZs4iCTw1/PEys4LkMc+VeaOW0zO/N6Esl5bdANw/hu5E9KxE9TGrfK5rkPdlGxsoEz1sRml7Q6N4vXvtxWoPOfHbWLS29Mw+aQWG2SGvV8vDR2B01aH7WA8/xsCTD0KxCVJ4LjGtCbIbX6GIE0dC0VuzO4pnpNPRgG3H1eC0YPuQpEPJBHXRFzZwFFPD+KyuI2gPxj7SPjA//6YNSIFqmxgwnycB7hitF85A7QqkStZFjaOG2FvPvd3XDERv7wtL6wBdvX7uSfP9fjc0fbSFKUd274XzJWW3jIX4MzPNRrmncER6Wqfs7rnoHdGXq+zS65pd8h6jbx0OWawzRuFd2t4YxzcN8bt4OlZTBywtyDMDehaaFzWqxQubqfGvU8zPh0P0aFicFoizLF8db0CeBfhMQerBMeEYOofvQQq1wEk5VsXaHCU4gqP50yF8qJUm5cK/u3H+T7kT+xZdV2mrRvSK8nr8mzT2Gn6zrS6bqOLJi2DJ/Hh9VmQWiCgd8+ic1uZds/u6JWmlOcHtSo50OP8A0JBASVa/jZtdkR/uIRqtX24/cJ7MdtVGp6MOb8nEsyMAICXZdsW+/kxd6NyM48GsKmWzRGrxpOzYbVgzHhvr+PbLoRpWqmpHZDL5WqpqFlPpI7NpxjNjzLDQZkvEqhbgZAMLO0DVrC8KJYVIlSLoR8y6rtPNX1ZfxePwG/wX9LNvPr53MZ8edgmrRrGPEcTdMYMOEx/lu6hZVz/iG2Ygxdb+mU21CiVpMaamPzNGfVX3E0a5cdJsZWm2TrurzreO/ZZg+z5gFMA+xOE4sFcqzBJq3cPPrGHob1O5qrcP+wO3MbfkvPL3kIcxBnrOTJt/cgEYg8AxjKm4gfwdiSzwArQTdKXk/YXvDMQhrPIfQqIa9IYz94/w5G+9gvQmixhVxw0VIuTM4P+n2GO8uTm10Z8Bu4szx80G9cnucJITjz3Kb0HtiL6/tdGdIV6IxzmlCnWS00S7n4FSlOgulfVMGVruM/5rvvdgl++LRKxM1OISQ3P3KQb9esY9I/68hM1fG6j5rPphm0pi3HnWo7UjNc0+WReQQLpi3NfT1lX0q+1Y6FCFr7uq6Mj8hoYLsg/2HCFpbAZGZ9iEzqjswYgsx4CZnUGeldVEzrPDnKhUqtX7wp4vENUY5HYs+mfWxYshmfJ/itFUIw7LeXccZGf3xWlG8y0yw8fFkzfvqyMvt32ti42snIZ+vy+ZuRu6U/PHgvvZ88SKWqAaw2SZVaAYQmSdpnxZWhsez3ChhRGshomqRuEw+Va/iRUrJh8WayD2/mwL9P48tcWIzv8nRAB2EB39z8h0pfsJtQ7o+rIOsTguGN7mBdFpmNPNwXGSnj8xRRLlwrzlgH2Znh/rFom5nHkrQnhZd6vMWeTfvQLTrSlDzyfh8uv/siKlSKw6s2O09rDidb+fjlOnz8ct7j4ioGuKJ3akSf+JqFsQx/rD51m3joePHGsHNzrO33ftqMboG0JJ35PyZi99xIYkUPlipS9R85KewEY8flkcJY+WEDWxcQcblHpPt7IpcmEEFXi+PSollqISkXFvlVD1yKzRkabmVz2rjmgbxjP6WUDLhiCNvX7sKb7SM7w407y8P7j4xl47Kgz61K7bKxa604tdSo5yPgC9+FtFigxdkubn3sIG9/v5VIzdOFAN1ypGKiTVK1VoAbHz6EwJ1bv0W1BTwZdIJ+8YLsC2gE484XIw+di+meFTwsczJ1j0ceLQBWCigXQn7vkNs496r22BxWYivGYHNYOffqs7hnyK15nrd19Q4O7UzCNEL/0D63n+9H/QzAna/cpKJXFPlycI8Nqz3C5qYJNer5ubv/ASpWDkQU5OOtbSGiC7eUR/8p8iObgsW653QD8h9xnbgg/TlkYAvCcVWw69DxyADYOxftcgtBoV0rQoi6wJdAdYK/jU+klKMKO++JYLNbeXnKMxzcmcSeTfuo06wW1etXzfe8tEPp6JbwqmVSSvZtPcC7D47hvyWbadCyLtvX7lRfHkVUMtMszP0+ga7XHcYRc/SDIgQRrfBjORFr2+sRuLN0KlU9Nd3ayyeRarH4kdmTEBWeDza68P5J8MZgCf6LfxGhRe8aVtIUutaKEKImUFNKuVIIUQFYAVwvpVwf7ZzSUGsFICM1k9vqPJh/JcOTLTehOK3QLZK7n93Ptfem5Ha0z6/j2Ylw7FdVuVpKAL1RsASwVhls7SGwE7Q4hPM6hKXJKVlStForhf6YSSn3SylXHvn/TGADcPIl30qQ+MQK3P5iLxwx+WyKKhFXFAAjIBj3Zi1uaNaK+7s2L+ATXITH9ijkuFyUiJcQxjbwLwXvLMgcCt6/gt2MSuGjeZE6f4UQDYD2QFjFdyHEA0KI5UKI5UlJSUV52UJxS//rqNO8FkJT3w5FUSFwxJgRNz9zOKoFeSf6KEoLBpi7wP0DMqUX0rvgVC8ohCITciFEHPAd8ISUMqzogZTyEyllByllh6pV8/dflxQ/ffobuzfuVVmcipNCt0i69kjjmZG7uO/FfdRuFKyouX2DE18EIc8RcGVVl1VMgj09X8iz8F5JUyRx5EIIK0ERnyil/L4o5iwKPNle0pMySKyZEFZfPIdfx/0R0ntTcXoSnxgg4BNkZ0Vv2eWs4MCd5UHXdYyAQevzPfQftY2EqgFsdonfBz36JDO8Xz0Wz4lnyZxELu6ZnOsOkVIJeLnBTAm6WfTSYZQWRdSKAD4DNkgpRxR+SYXHCBiMfno8P4/9HQDTMGnStgE3PHE1XXqdGyLqpeimqjgFNGubzdMjd1G7YfBmvnpBLG8/Xi+sgYQ9xk7Px67m2r7dWb9oE1Urvk3jMzajaWauOFttYEXy1Lu72PpvPK3O9eRudioRL28YkcMSTxFF4VrpDNwJXCyEWH3k31VFMO9J89kLXzPrs9/xuX343D4CvgD/LdvC230+pN95L+DJDj7++jw+HHEnWwdZURrQdEnjVtnUbRItcSM6idX9DJ2ylQbNvVhtwWSc9hdkMWzq1pC5KiYG6Hn/AS7r+QeJCfPpfGUazVptQdfNiOLsjJW0Pj8dTTvauCTHIleUFwRSRo52k1Ii3TMxk3tgHuqMefgpZGBXsa6m0Ba5lPJvCtMmvIgJ+ANM/+jXiO4SvzfA7o37+PGDWdzS/3reuuv93AxORdmjQ7cM+n+wC6tVoumSpH02Bt3bgD1bC1Yf58rbU9At4bW9q9by0+ocF/8ujaNJ62yGTdmKxSqxO3fhT1mAEAa6Hj1bMFrIobLIywIWIjdcPp4AJF2AGXMHokJ/xDHJAtL1MbjGgDxSNsTzc7AUcZXpCL14AvrKXcqiO8uDEYiezeVz+/jjm79J3pvC4hkr8HtUYkVZpHpdLy+N3UHFRIOYCiaOGEntRl6GTd2aW0UwP+o09obVRjk6f9AQ6P/+bmLjzdwGERaLH00rp6VgFRRMxHPwQfYEZPozSPMwANJ0QdbooyIOBLsUuZFZBWhGfZKUOyGPS4glPjEuzzF2p50DO5KwOcpaFxVFDlfcnhpWslXTwBFjctaFBenMDuuWxuJ2hZvJVpvAmdiBOwZeTN0m4V9sZVkrjuIHz0zkoS6Yac8iM4YQ2UERAN+yYltFuRNyIQQPvXsP9pjIPQsdsXaufbg7dZrVzD+jU1FqqVLDjzXC9oYmKHD6+m9TK5F52BJSb9zrFuzb1ZjHxwzlrkG3o6n8AkWB8IL3R/B8R9RORXrdYrt6uRNygItu6cyg7/tTv2WdkONCE5zfowOX9O5CQtWKXH7vRdjzy+pUlEpW/lkBtyv846vpwU73BcGTrdPvyqbMnpTI4WSdg7utTHy3On0vdTLlnekYRixY2yFlufyaKEoUByLugWKbvdC1Vk6Goqy14vP6SU/KIKFafEhYoTvLzd1N+5GenJlb3VC36tRpWpNP/nkHTdMwTZPvRszku5E/kXXYhRDgcXmjXUpRirBYTUbN3EydJl4cR/zXbpdg7g+VGNW/8JaPxWahxfnNGDbrDkT6nXjdh7FYTDRduVZOX+wE3SYFbSghgueIGIh/Gc1Z+GC+aLVWymxjCdM0GT9oMt+/OxMpJZqucdvzPbn1uesRQjD324W4szwhJWoNv8GhXcms+n0tZ1/WFk3TuOmZHtz0TA98Xj/Xxt1xCt+R4kQI+DWeur4p196dzEU3pOFxa8wcX4W5PyTkeZ6ma2Fli3OITwzQqIWbpH02DuySNGi8CDPlO3SLDavNSsDvRYueL6Qo1zgg7gGEvRsyYxj4l5BvuKutG6LiK6BVR4ji/eCUWSGfNOxHvhsxE2/2UQt64pDviEuI4dqHLmf72p0RreuA32D3f/s4+7K2Icc1TQTrrRSkfLGiVOB1a0wdXY2po6vlO9Zis/DS5Kew2q0snLaUxT+vJHl3ypFXJfe/vI9r70nB7w3WScnpgSkEIH1oGtjsKhb89CMGMMB+MSL2QYSwIip/iZnxOmRPJqo/XDgRMbci9Folssoy6fyTUjJ5+I8hIg7gzfby9RvBCgGN2zbAERceT2yx6tRrUSfCcQvnXNke3apMrvKI4Q+wfe0uOl7ejsc/foA+r92W2wrwsptTufrOFOwOSVxFE4s12LEnkgtFuVXKOif6BwxAXD+QXmTy1ZiHLsI8/BQi5hZE4niIuRccVwMOgqJvC/6/owfYuxX14qNSJi1y0zTJSovcgy/tYDqHdiezZ/N+zICJECK3uI3FZqFGw2q0u6hlxHOfHPMgT174Mgd2HMLwK9O8PCElfPHSt2SkZHLXoJu5uPcFLJi2kK2rVnHjQ8k4Y5WpXe7RzwW5HcxDJ3CSD7JGEhJf7tmH9P6BSPwaLf55AKSZAZ7ZIDPB1hlhbVakS8+PMrvZeVfTR9m/9WDY8VpNapC6P42AL0DAb+Q2hbA5bXS7uRMPv3sPcQnRoxoMw2DQDcNY/NNKVYe8nGKxajwxwuTSXpuRZgAhfMrSPl1w3Aaeb4pmLlsntMQvimauAlJsjSVOFX3fvRd7WMNlK54sDx6XNyjikCvGzTo05tnPHyEuIRav28u0D2bx5IUvMfDaN1k6a1XuHLquc/uLN4bNrSid6BZJ5ysPc2u/g3S6Ij0s5T4S1/U5QJcr1yLIRtOUiJ9WeKYU3Vz+NUU3VyEpsxY5wOq5/zL+lUns3riPBi3rcv2jV/JG71H4vZETfb7a/hGVaiTwROcX2fXf3tx6LI5YOz2fuJp7X7std+z4QZOZNPQH/F6Vwl9aqd3Qw8iZm4mJC4YFerMFqUlWnuzRhPSUYChqpap+4ioa7N1mxzSDij157b9UrBzddaYqFSoKhF4XrervJXrJchd+CNDuola0u6hV7s+Hk9Kj11kRsHjmCmIqONm9cV9IUS2Py8vUd2bQo+8VVK5ZCYC7B91M97u68umAr1g0fTkBnxL00kRidT+jf9+E1S5zRdcZJ6lu9/HQq/sY/XJtXhizgxZnZ2MEBD6fYOQzdVj0a8U8RRyUiCsKggWcd4YckdIH3gUgM8B2LkKvUZKrKT8kVK1I5dqJJO1KDntNt+hYrDqLZiyPmvSzfPZqLr/7otyfazaqzsuTn2b+5IVMeG0qyXtSqHtGbUzTYNOybcX2PhT5c/ez+0NEPAeLFbpck06TNoJaDbKxWCQgcQIvjtmJaQplcSuKAAnuKcjY2xDCjvSvR6beC/gJFskKIGP7oFV4qkRWU66EHGDAl4/y9EWDwjYqhQCP28fy2asjnufz+Hnv4U+p0aAabbuGRrV0vbkTXW/ulPuzaZrcc8Zj7N8SvtmqKBnOuTQzqhhbbTr1ItQn1y1gEWoHW1EUGGDuAfdMpPN6ZNr9INNCh2SPR9rORdg7F/tqyuxmZzTaXNiSh0fcg8WqY7FZsDmt2BxWut7cic9f/AZ3ZvT0Wp/HzyvXD8Pvi15Ma/HMFfyv1ZNKxIsDQViYr8UWOa7fnRX5oyslHDpQB8MMV3llhSsiIzgpKZRupPdP8K8GGaGJtnQjs4soQiYfyp2QA/R8/GombPuQR0b14dH37mPCtg9ZPGNFWAJRJKSUrJm3PuJrS2etYsgtI9j9376iXvLpiwimzVvtVi66tTPnXNkei1VHt+g0PbsRHyx5i4pV48NO+3FcFXye0I+vlOD1CF663Ynhj5Jxlwcqa/N0QgA20JtCpa/A2pZgUs+JRKtZQK8O0kPURCMZOd+lqCl3rpUcqtSuzDUPXgZAyv40/CewWRltY3PsgK/wulWj5qLEEeNgyIznaNGpeW7RM5/HhxEwcMY5AXh2XF9eu3lEyO9+9qRa3PxkU6o4FiHRMPxeDu218tR1jUlLsvH9J1W4/r4UHDH5N4FQne1PRySgQdwjCEtDROVJyMBOZNpDYGyNMF4cc14OFkTMraDXBBlpA92JcFxT5CuPRLm0yI8nvnJcgetKGwGT+q3qsOqPtezfFuo+2bflQHEs77SmUvWKtOnaMqRypc1hyxVxgHOvPpvB0wdw5nlNqZAYR4vzmzF4+gtUa/kpouqv7D30NP1vasW9nc4kLSloUf36TSKb1jgwC9DMJ6fLveJ0wwPpTyGTumGmPgBa5TxqhttAqwM4QMSCiEckjEBYGiGEE+IHB1/jiCtQxIC1JTivLZF3Um4t8mOx2qz0fPKasCJbmq6hW3QCPj+6VUdoGq0uOIP7WjyJxWrB7/HTqG19hs55ibiKsVRvUI1dG/acwndSvqhcqxJvzHoRUQAVPeuS1px1Seuw40KvTZbrPLZv+B1wo2mSW/od5K5nDiI0JdCK/DABH/gWItOfQcTehfQtIbQYlhZ0oST+ipA7gv5wyxkIcdT40GKuQ9paIrOngJmKcFwK9ksQomQktkwnBJ0IpmnyzRvfM+WdGbgz3VSpU5kH37mL+MQKLP5pBXEJsZimyeRh08N86Y5YO2P/fZetq3fwRu+RITHoQhNIUzlXTxShCX7xfUvWYReHdiZTs1E1YisWrCHE8RgBg5tq/I/M1EyGTd1Km/NdSsAVJ4ENUXU+0j0Vsj4AoR/Te9MGWmVEwiiErW3YmdK/HgL/BS16a4cCGScnQ7SEoNNCyPdvP8gnz05gxew12GNsXHX/pfQe2AubPXRj464mj4a5U3Jo3K4+o1e+zdxJCxg74CsO7UwmoVpFKlSOY/eGvSXxNsoVzc5uRMM29Zn7zd9YbBYCvgDX9r2cB4bdiRatDX0UpJTsXPkhNnM01euolHvFyWJBVJ6GsDbDDOyH5CuB46JRRCyi6jyEVhEIJgHJtIfAt5ygHz0AWjwkvIdm61jkKyx3tVYKSnpyBo+eM4CF05bizvJw+FAG342YyZu3vxc21pUefYd513/7OLDjEBfd0pmJ2z/m18AkphwYS7W6lYtz+eUSu9NG7WY1mTdpAT6Pn+wMNz6Pn5mj5/DDqJ9OeD6Z9S71an9CjbpKxBWFIYAUwcbtwr8kciCKNMAz8+iPWR8eEXEPQXeMH8wUSL0TM+NNSspQLvdCPnPMbDwuL+Yx7g+v28fSWSvZs3l/yNh2F4f7YHPQdQ131tEY9Byr8dDO8CxSRXSant2IN2a9yKLpy2jQ7DAX90yjUcvg46s328uUd2YUaB7pX4+Z+iDmwU7gGkPB228pFHmQchVm5ntI4xDISBFqHqSRdPRH9xQif/ZMyP4W/CXjeSj3m50bFm/G5wlP8LHYLGxfu4s6TWvmHvvfm71ZPGN5xPE2h405E+bz25fzCfgCdL7+HPq82Vv1+CwgVpuFsy9vx2s/Pkd2xkGGTf6Xes28SDPYMHnDilhevrshmalZUeeQ0gTvr0jXBPCvJLhRpVAUITIbXGPB3gWEFeRxocgiBmE7+5jxeX3/PUj3dEQxuFiOp9xb5A1a1sViC79fGQGT2k1Ci9rUbFSd0SuHEZsQm7tZoekaNqeNxJoJ/Pj+LNIOppOZ5mLOV3/y6DkDOLt7G3RLePZhUT/i6xb9xJubFBV5XFfTNSw2S57v12Kz0KJTc/p/8QgTBk9h0dfX07CFB2esSUwFE0eMpEVHF3f330/zjk0iziGlRB7uh0x//oiVo0RcUVx4wDsfLO0JhhTm4ADLmWA7JuXefjF5fzFLxrVS7i3yHn0vZ/pHv4Yk+VhtFpq0a0CjNvXDxtc9ow7f7hnDbxP+ZMWcNVSrV4UW5zXj7fs+CrHUDb9BRkom9VvWpUKlWFwZbvxef25HoqJwjVkdVgy/gWmakFu0jwAAIABJREFU0as6FhNCBHuYCk3QtH1DKtVIYOWcf0KScqx2C7e/0JMLep7LhMFT+XPKorB54itXYOTfQ6jbvBbfjZzJ5OHTmPpvKlZ76C/I7pBccVsa7a69O/KCfEvB+zdReyQqFEWJsEGFp4M1x91TABMcNyBi70CIo/avqPAs0vs3yJQIkzgQzh4lstxyb5FXq1eV4X8Mokn7BrnWY5cbz+f1n1+Ieo4jxs41D17GK1Of4eER95CRmhUxxNDj8rJn434+/XcEN/fvQWxCDKIAv1FRwOQkv8cf7Ph+CqIb23ZrwbTD4/kuaRxPjX0Yv9ePYZggQOiCqnUq89Lkp7njpZv467slLJi2NOI8VruFOs2C7qtJQ6fh83ijNn+IiddpdnbjiK9J318oEVeUGNKPsDRAi70DrcqPaFVmoMX1QYjQSDehV0NUnQOOngTlVD/yXwfE3AjW4nerQBFZ5EKIK4BRBN/FWCnlW0Uxb1HRvENjPl4xHE+2N1hMy3pib7tW4+polnCFtsfYqHtGLRKqVuTMc5oy1T8T08hfdU9J7LkIumdsDhuGYWD4AhiB6O6Jjcu34oixs3fLAR7v9CIelyf3KcNmt3Jx7y6cf20HTNNk6ogZUXucutKz2bl+Dw1a1uVwUgbSFKxfEUOLDtmERhlqaI4LgaAbBd/8YHKF9B2xav7f3nmHyVVWf/xz7vTZlm2hpYH00BN6NaGDFOkE6SJd+k9ApEgVpKhBCU2FICg9BOkIAiIoqPQISIdkk91sm52dcs/vj3d2s5O9s3V2Z2bzfp4nT7Jz733fczczZ9573nO+pzw/vweLpV8iED0IcSoGdLY45ci4q9H0GRD/s9FeCe2EBNYbYTuXMewVuYj4gNnAHsD6wGEisv5wxx0JwtHQoJ2467os+OfHvYqEREzF6C5H7gjA60+8SecANz7dPhxoFz6/D8eXx6C4wtSt1+H6Fy7l4cbfsv0BW/X5ZJBKpvnmf4v4w5UP0tmRyAoVdcYSPHTT47S3xOjsSPQrRtYVP5+8vil//sV5E4m1OnR2mAOdcQekCqkwT0naeiXa9EPofNo49OYfQ+KlYdy8xTJQolB+Uvd7cTCIb2Wk7Bik/KRRdeKQn9DKFsCHqvqxqiaAe4F98zBuUTDnnN8z9/IHeq20J68/kRv++lMqa8y3dmVdJT6PVXtPxJH+nbNApCLMQed+Z9BfOv0RigZZc5PVCQQDHHXpIYTLwrlPVnP++6/914R3lsMf9PHVh98QjoaoXnlczmEq6yqYtN4EAE664WhCkSCfLghz7LbrMffGlXhxXg2LlsxC6p9C/BPQ1CcmbSsrjNIBqbfB37uizmLJG1KGjP8LTvmJWXHwUiAf1q4GfN7j5y8yr2UhIieIyD9E5B8NDQ3LHy5K2pvbmfebp3qvxh1h8tQJTJm6TGBn16N28sxe6cIf8lO90jiCfTR1nrbrxvz+w1/xQMMdHHfFLDbdOXde+1DY9ciduv89Ye1Vufn1q1l7+hq9znN8DmtPX4OalauZuM6qnhkpyc4U9RNrERF+cN2RBCOBXueEy0JcfP853RlAm83ckKufuohNvr0BTqCW99+aQd16dzJ52kXdlXIk/oZnFkBG38IIF1ks+cAPhIAwOBOQmt8hTu5FSTEzal87qjpHVaer6vT6+vrRmnZYfPNJg2fqorrKh29+kvXaylPGc+EfzuzlqH1+H+Mn1XHWLSdy5wc3UVYZzRnSeOfl93n3lQXdSoC5mmAEwwECoeUcZ0bXOxciwrTds1e0E9ZelV++ehW7HLUjwXCASEWYSHmYVdZYiR/feyYAh/5of4Lh7HsKRoJs990tGVdvnO9OB2/LJQ+cy1rT1iBSGWHl1cdz6I/2576vbu21ebnBtuty7bMX88evb+O65y5hg+2WewSVCqNx0YuM9nPlpTnv0WIZHC5EDkDq5iH1zyKBjQpt0JDJhyP/Euip/Tgh81rJs9Lkek9tchFhyga95S632XdzHlx8BzscsNWy/GoHNt99E2Ycvh3R8gg3/PWnTNlgkud88fZOHr35ie6fV1trFW/nLLDHcTMIRYJEKyMEQgG2238L1tps9Zz3Eq2MUFYZ7fW64zicd+ep3PrW9Zx5y4lcMf8C7nz/JupWM9ID626xFhf96WzGT67DH/ARDAfY9cgdOeeOk7PG2Xz3Tbn59Wt4dOnvueuj2Rx35eFEKyK95lN10c5X0dif0OTbvQ3NmZebgvh8U6xhseQFFzoeBFyTNuzGcFuuxF24Be7CzXCXnoemS6Nye9iiWWJ0GhcAMzEO/HXgcFV9J9c1hVA/HCo3nXwrT//+L1mKh6FokBv/ejlrbrrMcTZ8sYQ/XfcoLz7wKk3fLM2KK4eiQfY7dQ+Ov/oIAD769yecsd2PPatCV99oEnP+9XMAPnnnc07d8vys0I4/6GftaWtw08tX0NHWwZf//Yba1WqoHl/F4ZNOpOELr3xWOOT/9uX4q44Y8u9BVWlvjhGKBrO0wwc1htuILpkF7tfLujkEN0aqbzU7/fEnQdtQqYWW84GBNwOxWIZGEKk4C6LHoI2HQPJdoOuz7genHql/EpE+9pNGkVyiWcPeTVPVlIicCjyJST+8oy8nXmqc+stjqaqr4OFf/pn25hhTNpjIqb84LsuJf/3xQk7e/P+It8VJeaThdcYSPDL7CY698nAcxyGVSGVpv3QRDAfY6ZBlVWNTpk7kkgfP5frv/5rmxa2o6zJt100477enABApj2TZsfIa4z0dueN3OOz87w7r9yAilI8bmsxsF9p8AaQ/JctBJ9401Zqdz2ZOSmEeFEe3AMqyouKAhE21cGoBy5w4QAq0GToeh+jwPj8jzQohY5svXNf1lFi9ctaNvHDfK57OuQsR4bH2u7n/+nnck0np61noEy4LsfLq4/nFK1dkdccBsxpe/GUj0Ypwn5rdbzz7Fj/Z9+rsp4dIkJmztufMOScO4k7zj2onunBTvFfZQkGqniwWAkj9CxB/Am29BvBIpY3Mwqm6eNQt82KFlbHNJ7l0st987u0+nTiAonzzaQNzL3/AONoepzs+hwPO/A6zX7u6lxMH8yVQP6G238YLm83ckLNuO4nqlaoIhPwEI0F2O/bbnPqr4/q/uZFGU+R21taJW0YaH91t2JZD228F/yTw6uYjEfB7VxsXE2Nea2U0qKqtYOnC5j7PCYaDPHvXi55qWuoqyc5Er+yQoTDj0O3Y6eBtaFnSSrQySnD57JYRRt02o4siIQhu3l3SLE4Z6l/P5INn4WAFsCzDI2DSUt3/5TgeROoeR6USGrYjO3yShI570dBMcOog3cmyp0Yx10aKvyzGrsjzwEFn70O4LNTnOeoqVfWVnk2gHZ9DKNL39YPBcRzG1VeNuhN3Yw+ii7ZBm88xSoWLtkUTb3Qfl6qrTHpht6JchFyrJItl4CTB/QzvbCcfRPZF/JOQxEtGmnZ5NA7xx5GaPxj5WvzmusBmSO19Ay7VLyR2RZ4Hdj16J77471c8eON8HMchvlwBUSAcYPPdNmHnI3bgjgvu6XW9L+BjxuHbDWnujrYOnrjzed54+j+Mn1zPPifvxuT1Rr9oRlMfQcslQDwrUqJNx8P4VxAJI4F1oP5pNPYgpD8C3zrQ9rNRt9UyFvHaHI+AbyWk4twBjSC+OqT6FkyBuiKSv8XVSGM3O/NI29J2Pnnnc955+X3+cNVDuGmXVCrNlntuxnm/PYVIeYSXH36Nq2bdhONzUMBNpTnphqPZ+we7Dnq+1qY2Tp72fzQtWkpnLIHjcwiE/Pz43rPYau9p/Q+QR9zWa6H9TnptZkoZUnU1Et6t1zXqNqKLtgd6N/KwWIaHz8S3ax/G8Zu6DXVb0EXb0aujj0SQ6juyG0YUKSt08+VCkEqm+PrjhVTVVVJZm/1o1ra0nb/Pf4NUMsUWe25G9fiqIc1x+wVzeeCGx0h2ZjvPqvpK7vtqDj7f6IUt3OaLoOO+3gckglT8BIke4H1dwx5mdW6x5B0fRPbBqbqm+xW3Yz40/yjzUxrwQ/QwnMrzC2LhYBmxPHKLN/6An4nr9JKcAaB8XBkzZ20/7Dleeui1Xk4cTE/SLz74qlttcDSQ0Aw0Ps9oovREXQhtk/vCqqug8eCRNc6ygpKG+AvQY53kRPZCg9MzxWddcrNrF87EPGEdeQlTVtk7VRFMuKajLc4Lf3yFceOr2HCH9XKmTuaN0I4QmG4KKzSG2XgKQ9mxiG+V3Nc5dSNrl2XFxmOjUnwrQdmRBTBm5LCOvITZ77Q9uemkOVml/o7PoXxcGWfteDGBoB9Fqagu57rnLmGVNVbqPk/dRog/B6TMqsS3sscMA0fEgepboPMptGO+CalED0KCW/R9YeL1Yc1rseQmAtEcrQPHGNaRlzAzZ23PB69/yPxbnzFO21WilRHamtpJdiZJdppNxHh7JxftczW3vX0DAG7HE9B8nslpVwWuQCvOwik7Zlj2iPggvAcS3mNA52tyAcTnDWtOi8UQNpub2mEKezQBke8g0cMLbdioYDc7xwCLv1zC+699SM0q1dz2o7t568X3ep0Tigb59T9/xoS1ouiiHem1c08YqX0ACaw1Kja7sYeh5SeYkmhb2WnpYghyDTIRyo9BoodB6mNwvwT/usN+yixG7GbnGKZutVq229/Izna0eWuYOz4fsdY4xF/NrMSXPyOJxuchgbOGbIemF4P4+xXnVzcGLRfT+8vEYvFy4n05dz+Ed8Apyyh7BtYCRmcxUkzYys4xxo4Hbp3V3CJakebgUxZy9b1vs+bq16KpBeD5FOaCDi2fW5Nv4zbsjjbshC7aFnfJLDT9Te4Lkm/kaB5hsXjRl5tKLVPOXIGxK/Ixxr6n7s4zd7/Iwk8bcCTG7KcWUDs+SSiikHwOkmG8C3BCSHj3Qc+n6SVo4/dA25e9mHwDbTwc6p42cfOe52saTb5rUr8slgHRj6RxCZTQjzQl48hVlU/e+ZxYSwdrbbZ6XgSmxiKR8gizX7+a5+55CV/yTupXdQkEeq7A4yzrVZjCCFaFIXogEhx8c2PtuD+jbNiTNKQXo53PI+Gdl52rCbTxGEi+g20aYckPEYjkTiVUVeh8Bu14ADRtBLDCe/RaYJQ6JeHIv/54IRfufRUNny82pe2qnPGbE5hx2PCLasYioUiIPY6bibtkDiQ9HKaEofIiSH0CmkTCuw3JiQOQ/gxPDWfisPQM3KobcCK7ABiNleRbeMfG/Zg4qG0oYRkoQSOIFT0o5xnacqFpEagd5ufkaxD/M4z7VXdT8LFA0Tty13U5d+dLWfTZYrSH5vf13/8Nk9efyLc2nlI444qdXMU2mkb8U5HI/sOeQoLT0Pj83hWdACSg+Ww0+JTJIIg/Ss4NzqrrEDrR5q5MFoslFyGIfBcpP6nPzBRNvg8dj5H1ntMOSLxsCteCm4+8qaNE0W92vvvKB7Qsbs1y4gDJzhSP3vxkgawqDaTsKIxUbE8c8E/MX1lyeK/MF0auR1XXrIiAPsMpyfcgvB/4N8iPXZYxhgNSBaGZSM3tOFWX9p9emHgFzyc8jaGdL46IlYWi6B350oYWxEPD2027NH7dVACLSgcJbgEV52KKJcoxsp6rQ3A73MajcJsvRVO5xPgHOIeEkNr7IbBJjjOSqNtq/hnow0kn30W1GVJvDsseSylRBv5Nwb8tOLXgXx8qr4PKK8BZDaNbHwaJgn9NpP5pnOpf918t3IVUAV6a/MF+U2RLjaIPrUzdZh2Sid4ruVA0xJZ7Fb/sZKFxyo5AI/tD6l3UTULzmRCbiwlfvI52PAjVc5DQlkOeQ5xxUHkZuuQAPAuNQjtkjJmSexBdCIsPwHYLWpFoh9R7IL6MjOym3Uc0ciAk/2UaIvtXh8Dmg49ph3eF1p96pKA7EP7OsK0vJop+RV690jgOOie7A08wEmTlKfXscuQOBbSsdBCnDAluDvGHQFtYFoNOAR1oy4UMt8JXAmtB5LtkhXIkCuEZENgUTb4FbdfmHiD1FbifD8sGSykSB21Hm47HbbkOjT+LahoRQYKbItFDkOAWQ9qYFKcCqb7VrMylzDyVSjlS/QvEN34E7qVwlEyJ/quP/ZNHfvVnWpe2s8OBW/OdE3fxbFRsAU19Bp1/MX0zw7sgTg0A7sItQb3CUUFk/Ivd5w15XlVI/NWs8rtSvUIzAEEXz4T0Fzmu9GPTES2A+fL3TUFq7kGcaF6GVE1C8k1TURzYGMdXnZdxC0HJl+hvtfe0Ue96Uyp8/fFCkokUE9ZeBWI3Q9stmSMOtFyBVv3cpAA65ZD2cuSKySsfHiICoR2WhVK6Rk99CunFfVwYAW0d9vyWQhAEqQP9Kj/DaQxSH6HttyIVPzRt1xJvGiGswCZDy/92l6Bts7uVNt3AVNO1yv+t/NhcBJSMI7f05vMPvuTSA67jm/8tQhxhw63TXPbbt3Gc5dL3ms9GQy9B9EhovR7o6HEwYGRsnbIRtLSfx2LfFEi9NYLzW0aOBOiSgZ8uq2SeCvuq7O2EjkfQwIZo89ks01oJQfUtg6p5UE2hSw4BdxHdGSzJ/6BLDoX65xGnfOC2FzFFHyO3eJNMJDlrx4v57L0v6exIEG/vZLPtP0XVIwdbfND5AhI9AiJ7AqFMFksYAhsgVVfmxSZVRXtVeQK+idBXqljqPexbsZTJlfcfNKESMDFq35pQeTkDC6MpuvQMI/2gbZm/G9GmY0yIZMCmvZDZF+qZhqhG5jb+2MDHKXLsp6dEef3P/yLRkcjapPT5FM89IVUghYgPp+oqpP5ppOrnSO39OLX3Ic7geoaqKm77XbiLtsH9Zh3cht1xm05EF26MLpyKu3gfNLEsjVBEkHG/7GNEF8pOxj4gjjWSEDkCfJMBP0gNNJ9O/448nLnGq8pXofO5gZuQ/jyHGFyHCfmNEewnp0RZ8nUT6VT2G/3FR8ex6yFNRMqWT+FLm1ZsGcS3ct8rZA/UbUFjd0P8eRPPTn8BJDLDf2z+dJF6H208GuoeQvxrmDkD66DBGZDw+BAGtsSpOB3XbYSOewZll6WYUYjdTrdDTr3Wz/li9kv8U8G/BiRf8RgyBdo8cBP86y1rNJE1VRQJTB34OEXOsFbkInKtiLwvIv8RkYdEZGxl2RcxU7dZp1fo+e3Xynj+4TrS6SDmYEYcq/KiYWWkqNuKLtkP2n4NqX9nnHain6sSaPvtWa9I1SV4bqqmv8BtvwtS/x2yjZZiZTDaOQ5UXofU3I2Ed1oWlslCILj1wIcMbgH+tch+3wXAqTd55mOE4YZWngY2UNWNgAXA+cM3yTIQ1thoMlvuOY1QtEd+fTjE/Hu2QWrugrITofxUpG4+TnR4Xeo1djekGxicBkoa4n9BY/egbosZx23D84Ptfg6tV0PS9u9csQmCf4LJfgpuB4FNya5LiEBk/+6nvIEgIkj176Dse0ZKQqohciBS+ydExo6Cat7yyEVkf+BAVZ3V37m21Vt+SKfTPHH7czx2y9MkO5PMmLU9+5++J5GycF7ncZccbKrshkTEdA2qmYu2/Qo6n8qrbZaxRjVSfS0S2sFsnMfnoR2PAEEkerDRWhlDqoWDJVceeT4d+TzgPlW9O8fxE4ATACZNmjTt00/HzkbDWMdtOhk6nxneIL5vZeLqVtlwTBLYFlJvZ2LRHZmwSBXoUrLTXSMQ2hk6n8Q0OPHyPxGkbh7inzQKhpcWuRx5v6EVEXlGRN72+LNvj3MuxGxFz801jqrOUdXpqjq9vr5+qPdhKQBSdjRGwCjrVYzioRghrvC+JishF+nPsE2WS4EhrHYlikT3R+r/glRehJSfhoy7Eeqfg7LjTEhEopgGJoch465F6p/prvrtTRLt+OMw72PFot+sFVXdua/jInI0sDcwUwtR728ZcSS4BVpxHrRea3LSNQX+1ZHqW3pJiboNM03KV+9RyC11aykanNXAzbUfIpi1n7/H8RD4VoPw7ibmHD0w62wqTkfLT4D0IvDVI5KJeftWhvBuaOJvHlr2KUh/nd/7GuMMK/1QRHYHzgN2VPXsLGAZIxgVxe9C6l1wxiH+Nb1PjHwX2n5DtiNwwL+2SYGM3dHdrcUQ6FGib9cBBcWZgtQ/Ccl/ox3zoOMhTPgjmdEC382ssBN/N2miGofwXkjZMX1uHIqEwStMEtgM1GPzW6JIaLu83daKwLBi5CLyISavp6tG91VVPbG/6+xm59hFtRNtPBZS75h4qYRAwkjNveCbZFIS2+eY2KlvClJ5AerfGBq2wjryQhI0zbJ1iXGugamAQvobcKpHrJTdbf4xxOf1+HIPgX8SUvvQmMoqyRcjIpqlqjmWZZYVFZEQ1NxtWmkl3wLfKplMA/OhlPLjofx4VF1EMls06UUoAfrPTbeMDEEoOwUaD8o0AUkDKZBKCM9Eyk8HRsaRS+VPIbg5GptrQizhPZHoUdaJD5KSkbG1jB1UFVIfAgqZEI027JARNlqeqOks5H4D6a8oLbnbrphysTeUDgBBoN3jmA+kEql7HPHVjrJdluUZctaKxZJPNPkO2vBttPFAtPFgtGFHSP4bKi4mOzNGgChS94gpbkp/Tm4nLni39CowvvUZeRUMh94ZRWCyidaB4K5mD6JPXLJTBHuSNo0fYr8flpWWkcVqrVhGDXXb0cYjs7XHNYY2HYvUPw81d6HtcyD9KQSmI+XfB2cVWLwXfcfPfRBY14Ryion0O6MwiRptkvQCs/lIGKKHIBVnIhIwipSxudB+m5GP9a0OqY/I3ozu74khYTY4LUWLdeSW0aPzKe8sBU1DfD4SnYUEZ2cfSrxO/7HzELi5VpRjHQU6kPGvGblXqVi290CmRL3sCCg7wpzddjPadtMg53DAZ4tzihkbWrGMHunFeDvlOJpuyHFNjtezaAcJUpThlWExkOIcAd8ERHyIU5XlxD1xavEOxfRFECk7ZpDXWEYT68gt/aIaRzsexG25Co3d36+wv2kw4REKCU7H09lK1DSH9iK4ycCMTP2Xsfd2HkgiQggpO37gQ4b3NEVd/ZKJvTvjkXE3IYH1Bj6HZdQZa+98S57R9EK0YRe05TKI3Ym2XI4u3hlN9+7RqG4jbtMZ6MINTIOJppPR9MJlJwQ2gdBWZCnaEQb/hhDcxnN+8a0KgT6LizMkM12PVpS3tGOU/KquQQb6ZUems3zNb8FZlT5X5oEtkfonkPoXkfC3h22tZWRZUd71liGiLZeDu7hHGXUM3Ea0+ZLs8zSFLjk4o26YBFLQ+Ty65CDTQJeuTkGzoeJ847z9U6HiXKTm9j4V7aRmNgR3GoCxzeAbuMRpSRPcCRn/Ck5kj0FfKoGNzOZy9a2YtEMPUh8gvlX7D9VYigL7v2Tpm87n6Z3V4ELir9nhk86/gLuE7BTBtOmXGH+y+xURP07ZoTh1D+DUPYRT9j1EgmjqI9ymE3EXTsNtmIHbPrd7fKMpfTM4a/Vtq1QjFWcz+BhwF/2l6XlOysBzBkLg5GPTMJIpix+6do2IZFbyOb5AfasMeWzL6GMduaUfcr1Flns99XEm/W05NIamPupzBk19gS45yHxpdLWRa/0Z2nq16U6U+AeafBeC6/VhD5n0uolQdR04E/qcszcC435jurwPCtMPtX8cKDsdqZ8Pga0GOUdPBMqOQUJbDmOMLjLqlV5EDvR+3VKU2PRDS99E9oCOxzDhki78EN4lOxziXxMkbLqd90SiuQW2Mmj7rZkvgZ6bex0Q+z0au8fMR4z+N//SaNtsnOqb0OA0U2w00LJ//zSc8NaoXIM2/YDu8NCgcDDFNcsTgfKTccq/n/lxPzT5Wo5z+6ISau/CydfGY+JveH8xSkZ22FIq2BW5pU+k4gLwTwEpA4Lmb99EpPIn2SeGdjB9ELPWBj6jmtdfb8Tkm3g7zTSmcKWdgWVwqFFnBHBqMql2AyT1JqqKhLYy1aTRwzKr+sGELyJQdhb4NgAy/SalCspPR8pO6HHa3uBbdwBjO+CfDpHvwbg5yEqv5c+JA7hLwXNvQjNStpZSwa7ILX0iThXUzoPEK0Yfxb8GBLftFZ8V8UPtvWjLFZmYuBqxrMqL+hdA8q8BqQUMfoXay9pu7RYRgarL0KbTMKtyF/N2z7XK7lFE45+CVF6EVvwfuvQ06HyZga3sYxC/H6mbj0gos8kb6LWRKxKEuvvQ2EMQfxykAtyvMymUPSouJYyMuwrxTx7oL2BwBKeDJj0ORJGQzVQpJaxolqXgaPJddMmhgEeMfVCEkdq5SGBDo+kS+5OJt2ubCd0Ep5kGB53PkL2B64fQrjjVN3qO6i7eB1LvD8wEiSKVlyKRffs/twfqxtDWy6HjUSAF/rWQysuQ4KaDGmewuC3XQOwelmmthMG/JlJ7r1UgLEJGRMbWYskHElgfqmejLRcb/Wsc8E0wmisDVg6MINU3I4ENcdvvMt2MulbiEjVNDKJHwZJZZK/8faYysuonnqOqG8uslAeIxtDEvwbtyMWJIlVXopWXASnTjGEUkIrzMjKy95j9jfBeSPQg68RLDOvILaOKpj6D5NsmvS2wSXfYQULbQ92zJl1RIuA2oYv3MZkoA8IF32TUbYLWn5EVotAYJN+ApmNBG8iOt/sgciTi5Og3Kn1kdngShmGEQkT8jObHUkQgPAMJzxi1OS35x252WkYFVRd36Xno4r3QlgvRpmPQxXtmaayISEYvJIj4VoLqOQx4s1GCJs6c+DuIhyPUmAmz9IrDJyB+b+5hJQTBrT3s6JLOXe4jJAEkst/AbB4G2vk33Mbv4S7ayVTTpj4c8TktxYt15JY+UbcNTS5A3bbhjRP7A8SfADrNI7zGIP0J2nx2zmsksG5GDGsgEyTA/62M9rbXClpyvA6kPkY7X8ptR9VV4Fs1k7kTyoRqNobaxyCwOcahB8C/LlIzF3HGDczmIeJ2PGZSJBN/B/cr6HwCXXIgmlwwovNaiheuYHnZAAAH9ElEQVQbWrF4ouqirZmNMPGDptDoIUjF+UOrKIzdTe/NzDQk3kDdRs/QhkgIjR7r0bAZjFPOhEgkApGDEacG9Vw9g0mdLAdd4nEshTadDDV3IsFpve3wjYe6pyDxEqQ+N9rngWkmLFF7V3d7tJF24GD+X2i9guzfpQvagbb+HKm5ZcRtsBQf1pFbPNH2WyH2B8wKOhNvjv0RdaqR8lOGMmCOA+JdEdp1tPx01KmC9lvBbQL/WhA9HhLPQecr4FRB9Bgkeqg5X4JQfRvadBwmjKImxa7iDCSwEdp4HN7ZMXG07SakxrsTjogPQjuaVuPLH3Mq+rrz/OI2gufTkWby8S0rItaRW7xpv4PeDi8O7XfCUBx5eJfMF8NyedxOrekClAPTGOFoKDs6+0D0O7mvCW4M418x+d8ag+BW3f0mteZ30Dirtx2Q6ZxT5PT1peGrHz07LEWFjZFbvNGWnK8PpfZAyk/JVH52CVP5gQhSdU2fyodDRSSIhL+NRPbKahosgfVBcjSg6EdKoBgQCUFkP3oLg0WQspMKYZKlCLArcos3/nWWlbtnvb72kByvODVQNx/teNBs0vkmIdHDEP/othDLHXcPI+VnjKotQ0UqL0JJGQ2crv2KstOQyN6FNcxSMGxlp8UTTfwDbTwWk4+tmM3FEFJ9a56U9wqHaUj8O2i/pTvuLhUXIKGtC23aoFC31Wii+FYzK3XLmMdWdloGhQSnG+2UttmmPN2/DlJ+ChKYWmjThk3OuPswULcZEv8wmTHB6cPSCvcevwnteARS/zNl++E9zCbraG60WooW68gtOZHA+kj17P5PXMFx238LrT/PxN7V5JlX34kE1s7L+Jp8H22clRG4iqPxR6Dtl1B7P+JU52UOS2ljNzstlmGgiTeh9XpMmmabSbN0G0zlqg5UJ6afOZrPNQ03urKINAbpb9DWX+RlfEvpYx25pWTR1Ce4rdfjNl+Cdr5gimVG24bYvWTpuiw7AMl/Dn98d6npvtSLJHQ+MezxLWODvIRWRORs4DqgXlUX52NMi6Uv3I550HwhJh88hcYfNuXy1b/Je3y6T3Qp3k0vJEfhzmDx5xgfjDSAxZKHFbmITAR2BWxvKMuooG47NP8YE2rIFPZoDJKvZ/RcRg8J7YZn02ZNGv3z4Y7vlGfGWf6jGrJ9NS3d5CO0cgNwHgPrxWWxDJ/k68vyp3uiMTT+2OjaEtkbAmuzzJkLEIaKc0x3pTwgVdf2EO0Km7mCmyLlP8jL+JbSZ1ihFRHZF/hSVf/dX5GIiJwAnAAwadLoFoFYxhp9KCKOUkOG7ukkCDVzIT4fjT8BzjgkcriRCcjXHL6Voe5pI9qV/hL8UyGw0YhUxFpKk34duYg8A6zscehC4AJMWKVfVHUOMAdMQdAgbLRYsglujrfCYQQpQLhBJAiR/ZHI/iM4R0a0y2LxoF9Hrqo7e70uIhsCqwNdq/EJwBsisoWqfpNXKy2WHogEoPoWtOl4jLqhC7gQPQIJbVto8yyWUWfIoRVVfQsY3/WziHwCTLdZK5bRQIKbwfiXIf68yd8OboP4JxTaLIulINjKTkvJIhKByJ6FNsNiKTh5c+SqOiVfY1ksFotl4NjKTovFYilxrCO3WCyWEsc6covFYilxrCO3WCyWEqcgHYJEpAH4tJ/T6oAVJZXR3uvYxN7r2KSQ9zpZVXt12S6IIx8IIvIPr5ZGYxF7r2MTe69jk2K8VxtasVgslhLHOnKLxWIpcYrZkc8ptAGjiL3XsYm917FJ0d1r0cbILRaLxTIwinlFbrFYLJYBYB25xWKxlDgl4chF5GwRURGpK7QtI4WIXCsi74vIf0TkIREZV2ib8omI7C4iH4jIhyLyo0LbM1KIyEQReV5E3hWRd0Tkh4W2aaQREZ+IvCkio9xnb3QRkXEicn/mc/qeiGxdaJu6KHpHvgI1d34a2EBVNwIWAOcX2J68Iaat/WxgD2B94DARWb+wVo0YKeBsVV0f2Ao4ZQzfaxc/BN4rtBGjwE3AE6q6LrAxRXTPRe/IWUGaO6vqU6qaaQnPq5iOS2OFLYAPVfVjVU0A9wL7FtimEUFVv1bVNzL/bsV82FcrrFUjh4hMAPYCbiu0LSOJiFQBOwC3A6hqQlWXFtaqZRS1I+/Z3LnQtowyxwJ/LrQReWQ14PMeP3/BGHZuXYjIFGBT4O+FtWREuRGz0HILbcgIszrQANyZCSPdJiJlhTaqi4J3CMpXc+dSoK97VdVHMudciHk8nzuatlnyi4iUAw8AZ6hqS6HtGQlEZG9gkar+U0R2KrQ9I4wf2Aw4TVX/LiI3AT8CLiqsWYaCO/IVqblzrnvtQkSOBvYGZurYSvD/EpjY4+cJmdfGJCISwDjxuar6YKHtGUG2BfYRkT2BMFApIner6hEFtmsk+AL4QlW7nq7uxzjyoqBkCoLGenNnEdkduB7YUVUbCm1PPhERP2YDdybGgb8OHK6q7xTUsBFAzKrjd0Cjqp5RaHtGi8yK/BxV3bvQtowUIvJX4HhV/UBELgHKVPXcApsFFMGK3NLNr4AQ8HTmCeRVVT2xsCblB1VNicipwJOAD7hjLDrxDNsC3wPeEpF/ZV67QFUfL6BNlvxwGjBXRILAx8AxBbanm5JZkVssFovFm6LOWrFYLBZL/1hHbrFYLCWOdeQWi8VS4lhHbrFYLCWOdeQWi8VS4lhHbrFYLCWOdeQWi8VS4vw/kZbaLwPYjNcAAAAASUVORK5CYII=\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"tags":[],"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"id":"ZCdZTpjlJlGo"},"source":["Now let's train our linear regression by iterating over batch-by-batch over the data and repeatedly calling `train_on_batch`:"]},{"cell_type":"code","metadata":{"id":"YsHszjjaJDQZ","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1612742686207,"user_tz":180,"elapsed":3761,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"31b4c6e9-78df-40b7-c16a-5a408ed9f8d0"},"source":["# Shuffle the data.\n","indices = np.random.permutation(len(features))\n","features = features[indices]\n","labels = labels[indices]\n","\n","# Create a tf.data.Dataset object for easy batched iteration\n","dataset = tf.data.Dataset.from_tensor_slices((features, labels))\n","dataset = dataset.shuffle(buffer_size=1024).batch(256)\n","\n","for epoch in range(10):\n","  for step, (x, y) in enumerate(dataset):\n","    loss = train_on_batch(x, y)\n","  print('Epoch %d: last batch loss = %.4f' % (epoch, float(loss)))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Epoch 0: last batch loss = 0.0822\n","Epoch 1: last batch loss = 0.0499\n","Epoch 2: last batch loss = 0.0450\n","Epoch 3: last batch loss = 0.0282\n","Epoch 4: last batch loss = 0.0246\n","Epoch 5: last batch loss = 0.0457\n","Epoch 6: last batch loss = 0.0199\n","Epoch 7: last batch loss = 0.0390\n","Epoch 8: last batch loss = 0.0164\n","Epoch 9: last batch loss = 0.0199\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"zIDDhTcyJwSM"},"source":["Here's how our model performs:"]},{"cell_type":"code","metadata":{"id":"oBPYQpskJxxT","colab":{"base_uri":"https://localhost:8080/","height":282},"executionInfo":{"status":"ok","timestamp":1612742761053,"user_tz":180,"elapsed":1428,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"495a3d7f-44cc-4f6d-971a-0198e19a2197"},"source":["predictions = compute_predictions(features)\n","plt.scatter(features[:, 0], features[:, 1], c=predictions[:, 0] > 0.5)"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["<matplotlib.collections.PathCollection at 0x7fb54014a908>"]},"metadata":{"tags":[]},"execution_count":25},{"output_type":"display_data","data":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd3QUVRuHnzuzNY0ECL1X6ag0AQUsqJ8dFAuKih27ooKiYhdRBCsIotgQKwgKggqoKL2IAtKRTkhC6va53x8TAsvuhoRsKvc5J+eQ2Tt33l02v7nz3rcIKSUKhUKhqLhoZW2AQqFQKIqHEnKFQqGo4CghVygUigqOEnKFQqGo4CghVygUigqOpSwuWr16ddmoUaOyuLRCoVBUWFasWHFQSpl87PEyEfJGjRqxfPnysri0QqFQVFiEEDvCHVeuFYVCoajgKCFXKBSKCo4ScoVCoajgKCFXKBSKCo4ScoXiBEjbl85fv67j4O7UsjZFoSibqBWFoqIS8AcYc9t4FkxbhNVuxefxccalnXh0yr3Y7NayNk9xkqJW5Ipyx77tB3jv0Y8YcenLTHtlOlnp2WVtUj6fvvA1C7/8A6/bR05GLl63jz9nruD94Z+WtWmKkxhRFmVsO3XqJFUcuSIc//zxL8POfw6/N4Df58fmtBET7+TdFaOoXrdaWZtH/+TBZKZmhRx3xNr5LvNjhBBlYJXiZEEIsUJK2enY42pFrig3SCl5dfDbuHM8+H1+ALwuL5mpWUx+YmoZW2eSm+UKe9yT68UwjFK2RqEwUUKuKDdkpWWzb3tKyHEjYLDkh5VlYFEorc9oEfZ4046N0HW9lK1RKEyUkCvKDVZH5M1CZ6yjFC2JzJCxN+OMc6BbTNHWdA1HrJ173761jC1TnMyoqBVFucEZ66DzBR1ZNnsVfl8g/7g9xsalQ84n4A/w61eLWTBtEc54Jxfddi7tzmxVqjY27dCICatf5YvRM9i4YitNOjRkwNBLqd+ybolf2+/zM++jhfz0ya9Y7VYuuu1cevbrqvzyCrXZqShfZKZlMfz85/lvw240XcPv9dPj8i4M/eBuRlzyEuv/3Ig7x4MQYHPauXbY5QwccWVZmw1AZmoW09+azYp5f1GjQXWufPBiWnZuFpW5A4EAw/o+z/olm/DkegBzg/XsgWfy4Pg7onINRfkn0manEnJFuUNKyeZV29i3PYWmHRpSp2ktfv92CaMGvYk7xxM01mq38vHWt6lWO6mMrDVJP5DBnac+QnZaNl6PDyEENqeVhybdxdnX9Cz2/ItnreCF68biznYHHbc5bby74hUanFLyTwSKskdFrSgqDEIImp/WhDP7daVO01oA/DFjWYiIA1isOqt+XlvaJobw+cvfknkwE6/HB5g3I0+ulzeHTMqPwCkOy+euCRFxAAGsmf93sedXVGyUj1xRIYhLikXTNYxAcIifEILYKjFlYpOUkg1LN5OTkcviWSuC/PqHCfgD7Nq4l0Zt6hfrWkk1ErDaLPi8wTcFzaKRUD2hWHMrKj5KyBUVggsHn80P7/2Ex+UNOq5ZNE7v26FUbTm4O5Xpb81h1vi5+Lx+dIuGOzv0aQHA7w8QlxRb7GueN6gXU1/+FoLfPrqu0+3i04o9v6Jio1wrigpB43YNufuNwWamZ4KTmHgnCdXjeXnOiJAaJwd2HmTlT3+xf0doTHpxWTR9KTe1uI9po6abKfouL64sN+H2mnSLTquuzalep2r+McMwWPnTX8wcP5e/F20Ie144kutXZ+CIK7HardgcVhxxDqrWTmTUvCexO+1Re3+Kiona7FRUKHIyc1mz4B8csQ7an9UKi/XIQ6Xf52fUoDdZNGMZtryCVp0vPJXHP3sgKgWt3Lkerqp5S1hf/WE0XcMZ58Dv89O4bQOemzmMxOQqABxKyeDh3k+TsjMVI2AgNEHjdg0YNfdJnHHOiHMG/AGevHQUa39bhzvHg9VmQegaL3w/nI692xb7fSkqDpE2O5VrRVGhiE2IofulncO+9tHIL/jzu+X43D58bnPTcdmcVbw//FPuGnNT0NjsQznMeHs2i2etpGqtRPrdfxEdercp8NprFvyDphf8ENvuzFO4/smrqFo7KSSSZNydE9m9eR+Bo3zpm1dt54MRnzNk7M1BYw3DIDs9h5gEJ3OnLGDNgr/x5r2nw37yF655nc93v6cyShXREXIhRCIwCWgLSGCwlPLPaMytUBSWmePnhvjQvS4fP0z8KUjIsw/lcMepQzm0PyNfHFfM+4vbRg3ksrsvjDj/8UTcHmOn14AedOwTukoO+AP8OWt5kIgD+Dw+5n28MEjIf/xwPhMf/ZicTBe6VSfgC+D3hka+eFxeNq/cVqRY9f07Upj/+e+4czx0+d9ptOraXCUUVQKitSIfB8yRUl4phLABZRNGoDipcYUJz4MjBa00zRTi6W/+ECTi5hgPEx/9hL439ckvByCl5J9FG9i9eR9NOzSiQ6/WBV7f4/Lw1j2T+OCJz+j34MVcO+yK/FR+wzCQRng3pifXy8fPfEnDNvWQSN68ZxKeXPOGFE7A85EgtMKL8C9Tf+O1W8djBAwCvgBfjZlF76u78/Cku5SYV3CKLeRCiCrAWcBNAFJKLyF76wpFydOme0v+Wrgu5HiLzk3zRRzM5JqjRfwwHpeXh856ipd/HMG/yzbzwjVjcWW70XQN3aLTunsLHv/0fp6/5vX8OPEgJBhSkpWew8cjv2DXv3sYOnkIi6Yv4+/f15PcoBr7t4VuwPo8Pj565guccQ58Hn+h484dsXaandq4UGNzMnJ47dbxeI96YvHkelj4xR/0uaYHp59XupE/iugSjRV5YyAF+EAI0QFYAdwvpcw5epAQ4nbgdoAGDRpE4bKK8s6KeWuY9soMUvekcdo57bl62OVBERyHWfvbej5/+Vv2bT9AuzNbce3wftRsmJz/+qpf1jJn8nx8Hi99rulJ98s7h/UL3z1uMA+cOQKf24ffF0C36FjtFu59K7igVVLNxIg2b169jXu6DefAjoP5MesBf4CAP8BfC/6hSfuGfLrjXW5t82CokB+FYUjmT1vExhVbOLgrDVe2G+txNlwjPVFEYvin9wfdoApixby/sFh1vMdU4XXnePj5s9+UkFdwoiHkFuA04F4p5RIhxDhgGPDk0YOklO8B74EZtRKF6yrKMbMmzGX8wx/l1wXZvXkfv0z9jQmrXw1qEPHL578z5tZ380Vx96Z9LJj2B+8sH0WdprV4f/inTH9rdn6kyLI5qzn1nHY88+2jCCGQUvLvss3s2riXgD9A47YN2bpmO/YYO6ee05bbXxlE7SY1g2zre1Nvlv24OsRfDYCEfVsPhH1PhiGZ/uZsOp3fgfQDGcf9DIyAwd4t+/MThXye0KeAE6X56U049ex2hR4fyb8vBPnuH0XFJRpCvgvYJaVckvf7V5hCrjhJ8bq9vPfIx/kiDhDwBcjJdDH15emcc11P/vhuORa7hRlvzg5a2Qb8AVxZLqY8PY2bnr2Gb8Z9H+QGced4+PO75YzsP5r737mNJy8bxX/rdiENGbLRuWzOai6564J8Ifd5fSyY9gdv3fN+SIZoYTECBjPf+dHc0j8ekrDZnkVGYGZ1evzYY2xYbVYem3JPkaY4vW+HsO/Z7rRz3g29im+jokwptpBLKfcJIXYKIVpKKf8FzgFCHZWKSsPGFVuYNmo6uzfvo23PUxjwyGXUqF89//Wd/+4Ju3kW8AX4+ZNf+fGD+XhdXoQmwoqLYUgWz1pBUs1E/P7wQvjnzOWsX7yR7LSckLT1w3hyvYx/6EPeW/Mam1dv47G+z5GbmYvfWzxx3bJ6e7HOLxICmp/WmEuHXMjG5Zup37IO597Qi/ikuCJN44x1MOLzB3luwBiEEAQCAYSmccmQ82l/VsGbuIryT1QSgoQQHTHDD23AVuBmKWV6pPEqIajisnjWCp6/egxetw8pJRarjj3WzttLX6Z63apMe2UGP0z6mdTdacW+lsVmKThqoxBousZsz1Suq38nqXsjfiWLRLPTGrNl1TZKK5fOarcyded4qhRQU2X53DVMG/UtB3am0qF3GwY+0T9on+EwmalZ/P7NEtw5Hjpf2LFU6qgroocqY6soNlJKrmtwJwePEWlNE/Ts343UPWlsWrE1bERIWeGItdO4XUPWL94YtTkTqsfhc/uLvDl5ojjjHLy55CUatqoX9vUfJv3EOw98mO/K0i0ajjgH41eOplajGqVio6J0UGVsFcUmff8hMsJ0kDcMyYq5a9iyZke5EnEwferRFHGAzIPZJFSLj+qcBSE0QZ2mNcO+5vP6mDD0o+D9CL+BK8vNp899VVomKsoYJeSKQuOMdxLJn2CxWaIalVHeiWZBLqGJiKGJukUnvmocz/R/lRXz1oS8vm/bgbCJRkbAYPX8f6Jmo6J8o4RcUWicsQ569usWIjqOWDs9r+gSlcJUJyMNW9eL6DaRSPZvT2HJ9ysZecVopo2egZQyv2pileoJEROIqtcNjdlXVE5U0SxFENv+/o/1izdRrU4Snfp2CIkxfvC9O8jNzGXVz2vRLTruXDeGYbBmwT9mRJ7guKF5FpsFi80StuPNSYeAjn3aMP3NOWFfNvxHonrcuR7eH/4pkx//1OyidHoT2p/VmiYdGrFl9fagjWEhBLWb1cLr9mJz2Er8bSjKFrXZqQDM5r4vXDuWpd+vBAGarhOb4GTMwmdDEmoAvnhtBhMf+aTI17FYdc64rDOaprHwiz+iYXqFx+qw5ldrPBEccQ4zjj43uLyu3Wmj+elNeG3BM4XOAFWUb1QZW0WBzJowj6U/rApKqnFlu3ig5whiq8RwcHcaiTWrMGDopXTs05b3h39W5GtYbDqapuHOdtPqjBboFo2A/8QScyoTxRFxAHdePZhjW+F5XF42r97Oqp/XqhT8So4ScgUAs8bPDVnRISFt3yHS9h0CzFog4x+aQuP2DYIe+QuLmYgTYNmc1Sz/cU2hu+Mojk+kTFV3tpt1f2xUQl7JUc9bCgB8nsIl3nhcXjYs2Vzs6ykRLx0csXaq1UkqazMUJYwS8nJMIBBgxjtzuKXNgwxsdBfvPDCZjIOZJXKt3tf0OG51PkXFQ7fo9Lq6e1mboShhlGulHCGlxJ3jxuawoVt0Xr3lXX77anG+y2Pm+Hksmr6MiWvHEBMfucdjYViz4B8mj/iM/9bvpl6LOlw7/ArqNK3Jfxt2R2yAoChfHG+PoXaTmjz15cPEJqg+L5UdtSIvJyz5YSWDmt7DFVVv4rIqgxg9+G0WTlsU5Lf2e/1kHMxi7pQFxbrW8rlreOKiF1n3x0ay03PYsGQTL147lqseuZTClfVTlDU2h5Unpj4YMbRQt2i8s2JUoRtPKCo2SsjLAesWb+S5Aa+xb/sBAn4Dj8vLL5/9jhEIFVVProfV8/8u1vUmPDwlpOSrx+VlylPTkCqIpEJgsVmQUhJfLc6M3T+GRm0bEFcltvQNU5QJyrVSDvjsha9Dus1EqvpnsVqo07TWcefctnYH6/7cSLU6Vel8QcegxJ7/NuwOe07KztQiWK0oDXSLjtBE0PfBardStVYio29+O7/hxpHXLFjtVoZOHlLapirKECXk5YBdG/cUeqzFqnPJXX0jvh7w5yX2zF4JgK7rOOMdjFn4bP4NIKlWYlTKzCpKltgqMTz5xcNMfOxjdm/el1/npm6zWuzatDfk5q9ZNDqd3wGrzconz35F76u7c2b/biXaAWjH+l18/MwXrF+yiTpNajFwRH869mlbYtdThEcJeTmgRadm7N2yH6OATUaLVadq7SQe/fAeajcOXwkPzBZrS2evCvojd+W4efaq1xi/cjQAl9xxHlOe/kKFAJZzhBDMnTKfZ759hJRdaez6dw+N2zVg1S9r+fDJaSHjDb/B0tmrMfwBpDR7ps6e/Asv/vB42B6nxWXrXzu4v+cIPLkepCE5sOMg65dsZOjkIfQe0CPq11NERvnIywEDR/TH5rQXOEa3Wpi8fiwdercpcNysCT+FJPZIQ7L1rx1MGTmNQwcz+XzUdCXiFYDsQznM/3wRQzoNo16L2lww+Gxadm5GfFIcVlv4NVjAF8gvUOnO8bDuz40snrmiROx7f/ineHLcQVFOnlwv7zzwIYahNltKEyXk5YCGrerx+q/Pcuo57RBamJ0rzCiErX/9F3JcSsmKeWt44bqxPHPlaDIjxJlLQ/LZC19zdZ3bQvyqivKLNCSuLBcz352bf+ysq84Iu8EZDne2m0XTl5aIbeuXbApb1Tg7PZvMMHXrFSWHEvJyQrNTG/PKvKc47ZzwndEDvgDxVUP7NE4YOoWR/Uaz4PNF/P7NUjJTsyLeDIyAPKHUekXZ4vP6+WfRv/m/xyfF8fzM4cRXjSMm3klMghNHrB27MzQUUdM14pJKJnqlaq3EsMeFEMSo2PVSRfnIyxn9HriYvxf9G+Qe0XSNBq3qUq957aCxuzbu4bt35wYVXYpK13ZFuaP+KXUA8wnM6/bSvldrvtg7kXV/bsSV7SY3y8WYW98NOc9qs3Dh4LNLxKZrhl3B2DvfC/qu2p02zruxt6pNX8ooIS9ndLnwVAaO6M8nz36J1W4l4A9Qq3FNnp3xWMjY375eXOzKeYryj27VufzeC/l81LdMGzWD3CwXVWslctvoGzACBmNvn4Bm0fI7NAkhsDosSAlDxt1M43YNS8SucwaeSeqeND55/msEZsRU72t6MGTsTSVyPUVkVD3yUmb7PztZNH0puq7Rs3+3kFX2YbIP5bBx+RYSa1ShcbsGCBHqLhl69kjWLFDtvCozQhM89dVQdvyzk6kvfRu0+rU5rBiGDJtzYLVbuej2c7h73C0lbqPX7WX/jhSq1kokViUhlSiR6pErIS9FPn72S6aNmm7+4WkCXde55eXr6HffRfljpJS4cz3YHFaEEKTvzyC2SgyOmNColv7VbyYzLbs034KiFHHE2nlp9hO06taCftVuJjfLVaTzrQ4rn257h6Sa4X3ZiopHJCFXm52lxLa//2PaqOl4XF4CAYOAL4DX7eX9YZ9y4D+zke/iWSu4ocndXJF0IxfHXs+lCTcwqOnd9Kt2M6MGvYnXfSQ2/OCeNFw5qlVaZcbr9vHOAx+Qk5l7Qo2trTYL/y7bUgKWKcobSshLid+/WYIvQtr9HzOW8/eiDTx/9Rj270gh4Dfwe/14cr143T58Hh8/ffIr1ze5m9S96fi8Pu474/GI8ykqB0bAYNva/1g+Zw1WR/jNw0gRSofPr1pbrcZPBpSQlxKaroX1cyMEQhN88txXIYWsjiV93yEe6PEEv3+7lOxDOapQ4UmA3xfg5RvGmbH/x3x97E4bbc5ogc0ZKvKarlGzUTLNT2tSSpYqyhIl5KXEWVd2Q7eGpkn7PD5OPbttoeutpB/IYOkPq3BlKbfKyYKUea3cJGi6wOaw0uy0xjw3cxivLXyW4Z/cT6szWmCxWbA5bdgcVlp2bsrLPz4ZfvGgqHSo8MNSon7Lulx8x3l8M/b7oONCCN64exLNT2vCgR0Hj5s6Lw2JPcaGzWnF61KhhycbFquFkd8+SufzO+Yf63lFV3pe0ZWAP8B/G3YTWyWGGvWrhz3flePm/eGfMm/KQnxeH6f37cjd426mVqMapfUWFCVA1FbkQghdCLFKCDErWnNWNjJSMkN8mkbAYP2STVwwuE/YzLxjCfgD1G9ZR2XOnaT4fQGy8tLf/122mS9Gz2DO5F/IychBt+g0btsgoogDPP6/F/hh4s/kZrnwefws/X4Fd3cZRla6in6qyETTtXI/sD6K81U6dm/eF7aNmsVqNgl4df5IqtYuuFFuwG/wwYipZKmww5MTKWndoyXPXzOGh/uMZPKIqbx9/2SubXAn6/78t8BTN67YwuaV24IiYAxD4snx8OMH80vackUJEhUhF0LUAy4CJkVjvspKh96tsYSpWpebmcvTV4zmhWvHFqrYkCfXS0Cl4p+UGIbkxWvGsnjWCjy5HgK+AO4cD64sNyP7jSYQiPy92PHPLgjjM/e4vGxcsbUkzVaUMNFakY8FHgVURaYwpO5NZ/2STfS9sQ/OeAeaHvqxG36DvVv3R+wMpFAcZv3STSFNJQDcuR425Qny7s17+emTX1n589p8ca/bojbhyhXanDaadiiZNH5F6VDszU4hxMXAASnlCiFE7wLG3Q7cDtCgQYPiXrbcsfPf3Xz3zo/s35HCaee25/ybeqNbdF4e9CaLZ67A6rDi9/jofllnFk1fakYhKBQnQoT9cHe2h3u7PY4j1o7f68dqt4KAhKrxvDp/JK26NqdB63psW7MjPwdBCDMK5oISKqylKB2KnaIvhHgJuAHwAw4gAfhGSnl9pHMqW4r+kh9W8tyAMfi9PgJ+A3uMnaSaVWh3ZisWfvEH3qMKW2mahpRSNXZQlBqaJmjasRHvLH+FnMxc3r5/Mgs+X4TfF6D9Wa25/93bqN+yblmbWe6RUiJdX0POJDDSwdYZEf8wwtK41GwolVoreSvyoVLKiwsaV56F3DAMZk2Yx3dvz8Gd46HHFV0Y+ER/EqrFhx0fCAS4ps7tHEoJbuhgtVsI+AMYgaJ9vrpVx2q34M5WzR8UkdF0DYtVx3t44/I4XzObw8oHG8ZRo0GyOTxvMaFpKpWksBhZYyBnCnC45o0GIgZR7TuEpV6B50rPImTWqxDYBnp9RNyDCEfRn4JUrZVCMnrwO7z3yMfsWLeL/TtS+O6dHxnS6TFc2eELFu3auBd3GH+lz+MvsogD+ZtXCkUkrHYLjds34IXvH+f6J69EFKJdkKZrQd9TIUSJi7j0rcdIvxcj5RyMtDuQvr9K9HoliTSyIecDjog4gAHShcx5r+BzPb8h0+8C/z8gc8H/L/LQAxiuH6JmX1QTgqSUC4AF0ZyzNNmzZR+/HuMK8Xv9HErJZO5HC7lsyAUh5zjjHBgRIgUsNsuJbV4qr4viGHSLjqYLLFYLZ1/Xkw692zD5iamk7U3H7rThzi345h8T76Rei/Alk0sC6V2NTBsEeAAJgV3I1D8h6V2EvQI2Zg5sBWEFeeznHABvwT1RZdYo4NhMbDdkjQLn/6JinlqRH8XG5VvCptF7cj2snv932HNq1K9O4/YNQxJ9HLF2rrjvf9hjbCpNWlFs3l01ih9cU/ku82MatK7Ha7eOZ/3ijezfkVKgiFtsOo4YO499dG+pulFk1guY4nV4VSIBNzLz+dKzwb8D6duIlFEILNBqgwxXC0mA5TgRP/5t4Y8b+5AyOtnZKkX/KKrXqxZ2NWyxWqjTpFbYc/Zu3Y8nx5O/eSkEaBadvjf25rZR19P76u48e+VrHPjv+On3CkUkls/5iwan1GPlT38xadinx+0MlVSzCqee046aDZP5323nln4Kvm9d+OOBrUjpR4iSkx7p34ZMHwKB3SBMPzZVXkXYu5/wnEJPRtp7g2ch5lPGYeyI2DsKPlmvCYFdYSatQrQkWAn5UbTp3pJqdauyZ/O+oPBAi1Xn4jvPCxkf8Ad4qNdTpO5Nz78BSGmOv3b4FQgh2Lv1ABkHM5WIK4pFRmom1zceQlZaTqHa+x1KyWTYx/cFPQ1K70qk6xuQHoTzIrD1KrmnRS0RjJTQ4yIWCH3qjRZS+pBp14NxEJDm36XMNX3UyXMQ+om7l0TiaGTGSHDn1UsS8eA4H/x/I/UakeeOvRcyRxLsX3dC3F1R+/yVa+UohBC8+stI2vRoidVuwe60UaNBdZ6bOYzajWuGjF8x7y9yMl0hafdGwGDOB/NxZbuY/MRnavNSUWx+/fJPUnenB7V6Kwi7M9ilZ2S9gUy7GVxfgnsG8tADyIyHSm6BEXsL4DzmoANiboj6zUMaORjZkzFSr0OmDQQji9BH6wAy96tiXUcIJ1riKETNZRD3CMhscH2DzByFTOmLkfNR2PO0mCsg/lEQiYDFvAHE3YuIualY9hyNWpEfQ7XaSYxZ8CzpBzLw5Hqo2TA54hcvdU9a2MQen8fP+j83cu+0P9i7dX9Jm6yo5HT532n8tfCfQouupmtUrZ3IDxN/4pzrz8RqfAc5bxMkbjIXPL+AdwnYu0XdZhFzEzKQCrkfg9BB+sHZHxF3X1SvI40cZGp/COwhdEPxaLxg7IvORY1MyH6VYBcLkDUaaT8LYWkUcooWOxAZcy3IHBCxCBHdNbQS8ggk1ahy3DGturUIm/IMsHzuajRdD1skS6EoLEIXDHjkUv7+vfD16KRhsGfzft596EO+HP0Rb8xaRWxCmO+hdCM9PyNKQsiFhkh4BBl3lymyem2EFj4XozhI15eFEHGAGITtjKLPH0g13VGBXQhbZ3D0Bfc8Qrp8ABBAuucg4u4MO5cQmrkaLwGUa6UYNGpTn26XdApbCMsIhO9urlAUhaq1kvIWDIU/5/Dawp3jYf/OHL6aEKndm57nsy45hBaHsLYIK+LSvxUjfQjG/s4YKedh5HxedFeP+2eOL+J2sDQwRbgISO8a5MFzIfsNcE1FZoxAHrwcKV2ELyslzSePMkAJeTEZ/ul91D+lTpHOccTaadS2QfibukKRh6ZrdD6/Aza7lYcm3ondaQtbcK0gfB6N32ZGFnLhvKz4hp4A0r8LmXoleH4GmQGBHZD1kpn9WBT0aoT/Q7KBpRlYWkDcPYiqnyPE8ev959snJTLjIdMVku9CyYXAf+DfQXghtyIcoUERpYFyrRQTXddpfmoTtq/dWejVhDvHw45/dpawZYqKjMWq44xzcsNTVwHQa0B3Grapz6wJc0ndk86W1ds4uCc9P4LF5rDi8/jDfgedseES1jRIGFmsOiGGfydkPm2GGuq1IH4YWiHdNDLnPZAugh81XJD7ETLuDoSWEDxeyrB7VSLmBqR7PsERIRrodRDVvj/xjVVjDwQOhHnBC+5vwhy3QezNCGvLE7teMVEr8jB4XB7+XbaZfdvD/UeGctEd54VtgFsQUkqVwakIol6LOljt5trKardy47MD8mujANRvWYeEavH8tfAf9v93kMTkBGo0qE79U+pw4zNX06RDQ7RjE9NiJJfenH7MlayQ9CFaTP8TslNKiZH5Khw8B7y/g0wD/zpIHxQxciME3yogzA1GWIMSaKTnD4yU/yvnbgIAACAASURBVCH3tzRdMNnv5if4SO9qZM4UEFUxo0FizZhxvSEi6f1iRsdYifwH6s/7OdruOETcA8W4XvFQK/JjmDVhLhMe+RhNE/h9AZqf2piR3z5CYnLkzc/W3Vow+IVreX/4Z+gWHVe2aoysKBq6VWf/jhR8HlMgXNluJj72CVVrV+XMfl0BGDdkIj9/+mt+r9aUnak4Yu288+Mo6resS68B3Rl69kgyDpoF3AK+AH2u7cy51y6AQA5m/HYAYm5G2E4/YVtl7qeQOzH8i1kvYTivP34Wqd4I/BsJEUvpA702UnqR7rmQ8Tj5PnCZAdnvIo1MpOUUyHzyyGtoIA1IfBNhP7NIIi79O5E5E82bi94YEXc7wtoWaWlh3qCC3Cgi1GYwny4COyBMxEppENXqh4WlvFY/XD3/b0Zc8nJQrK5u1WnZqSnjFr0QMn7z6m189dpMdm3cS/terWnToyVj73iPQwcyStNsRSVACBHWLVK/ZR0mrx/HoZQMrql3R0hnKKEJ+g7qxdDJdwNm9c61v60ndU86rbo1z89/MHKnQ9Yzh88CNETimwh70SI5pJTIA91AHrvKP4pq36FZTwk9N3AAjP2gN4bANmTqQII3Km1gbQe2syD3sOslnC/agbkGDdPuUMQjkuchtKqFez/+LcjUq0C6MVfZArAjkt4CvTEy7RowUiPYcTR287p6+AzwaBGp+qFakR/FV2NmhiRcBHwBNq/ezp4t+6jT9Mh/klmD/DW8bh/SkGxZs50vX/2utE1WVBIiLaj2bTezI+dM/iVsez9pSDat3IyR+SK4Z4Ow0u70AYizBudv7kkjHbKezhPGo849dCckL0BoBfeJzR/v+8d0ZRQk4ubIY2zMRWY8CJ5FIGxmZEfc7ZD4huljNw5iuln84PsHfAUXoTLFNkJLO5mFzHgWkTQ2jP1rwfc36HXB1gMhdGTW6LwNzWNqwmQ8jUj+BUQNIPU49mhgaV7iIl4QSsiPInVP+C+oxapz6EBGvpBLKXn99vFB7bZUqKGiJEhMTuDO0x5h29r/Io5JrrUVcucCean72e8g3QshdiBYmoJvdXh3r5Sm+Mdcd1w7jNzvIHMEEK5w1DH4t4C1lXkJIxeZevGRWiOHC0/lTEQkPA/JC5AZT4D7aw6L6PExiCjkAJ4fMHK6oMWa70tKLzL9dtN1IqWZoKQlQdXPwLuMsB+OsR/pWw2BLYRfjWsgHIAAUQWR9GYh7C45lJAfRecLOrJj3a6gLuNgdq5v3P5IhbPUvemFapKsUBSXlN2ppO5NK7C2fddz0skXcQDc4F+BzFxnCpdWnZAsRMDMdjTdgObmewYIB0I4gkZJ6TVX9IUSWSDzOaTjfISwItNuDV8wSrqQmS+BrcdRIl4YnBBzPXjXgH9p5GFZo5CW+gj7mcjsCeBdecR+CQTcyEOPgIgDGe5v2Q/ZbxFRIi2nIGJvBK0G2LohRMnVjykMJ72Qb1yxhZU/rSW2Sgx9b+zNjx8uICstK3/TyR5jZ/CL1+KMdXBwdypfj/2exTOX57+uUJQokgJF3GKFbn0PRjg3z5Vi7Cd8rLUdbGcgvcuQGY/nCa6B1BpC4lg0W2tzmH9DEY32gvdPDN8O8BewFyYPwsHzKbSIiyrgvBziHkCQhUw537z5hMWFzJmIsJ8Jrq8IvQkFju/C8S4j+AZ5GDs4zkU4ryic3aXASSvkUkpeHfwOC7/8E7/Xn5+d+eiHd7Np5VaW/rCKqrUT6f/gJXTq24Hdm/dyd5dhuHM8YX2VCkVpY4+x0/e6GlSruYGQcLggDouRRr6bQDjB1hspqkDq5QTFYRvbIe0KjISXwLsYPPPz/MiFREqkZ2nkyJagsYV5stWOjM2dCq6vkTE3QNInkD4oss8+sDfvH5GqRQYo+CbiBq0aGLkc+XysoCUiYiK2JC4TTtqolT9mLOOl68eFVCaMrRLDl/snYbUFx4U/0/9VFs1YqmqnKE4YIfJkIwpfoYRqcdz83PlceL2ByH6RQvmuAXNjrhXE3GjW686ZCORGsjjv50QaMxx10ygxbGaJ2Jy3CO86EmDtZG5uur8nVNAjhBIejVYXkfA4MucDMNLAcTYi9tZCR8VEGxW1cgw/fjg/bHlZKSVrf13Paee2x+vx8cmzXzJ70s8hzZUViqISbs2UWCOBQwcif7d0q242Wc6LjrI5bXS/wMdjb69DY9FREXiFECUADDCyIecdCGw/nsXHmbMgsS6KiFvyxhdV+L2miMfcaFZZ5Ni+uhJ8y8G/FbQqedExx7xeIFZwnIfI+ynPnLRCXtCTyOHXRl7xCmsWrsPrKuxqR6EoGgWJOALO7NeVKx++lG/f+J6kpLX0vTqNBo3/RoSIkAQ0sLY3hUu6ibhKN3ZEyfporbj9mLXLwzc4LxgP+Nchkt5EZjxlptYHIU3Xi4yU0BfpBugALSliJcPyxkkr5H1v7M2qn9eGXZW3O6s129bu4K9flYgryg5nnIPBL15HrUbJPDpuo7n5JiO5QQAM8K2Fat9C2rURekyWV05ExPPw/omMvaOAjU8DiBT7frSIC9Aag6U+2HsgnFcitLgTt6sUqTRCvnfbfr4Z+z2bV22j2amN6f/gxQX2Kex+WWe6X9aZRdOX4XV7sdosCE0w4vMHsdmtbP3rvyJXmlMoTgxJq9NzqNPIy1+L40jZbUW36IxfNZrajWsiPQvzapoUJOJH5iLtngLGlobvurQJmK3UZDFuBgDYwNYOLXF0NIwqVSqFkG9etY2Hej2Fz+PD7wuwYckmfvxgPmN+fZZmHcNXd9M0jWEf38eGpZtZOe8vYqvE0Ovq7vkNJeo0q6U2NhWlgq5Lkuv4eODVXfi8gtwsnX+3PJff8Fu65xRSxAEMkJGThyqfiOcR2HKcAVZMN0pBTykecM9GBh5D6NWDXpGBveD53Yz2sfdBaCVbx72oVAohf+ve94MKVfl9Afy+AG/dO5mxvz0X8TwhBK26NqdV1+Yhr53SpRn1WtRh29odBPyV9MuvKBcEAhp/zKmCrv+HLV7ijJH0bPg10M8c4I9Si7KTGgG2nuD95TjDbGY8/VFCbmS/DdnjMbM5NUBC4rtFrlNTklQK38G6xRvDHl8f4Xg4dm3cw/olm/C6zTu2EIJXfnoKe6w9KjYqFAVhGIKA30za0XQJvtUY3g0Y6feC748ytq6io5vlcb3zjz9Ues1uQvm/roLs9zDDG11mPL3MRR4agpTlp8pppViRO2Md5GaF+scchRDhlF2pPHnpy+zauAfdYvbYvPvNwZx/Yx/ik+KC6qkoFMXFajMI+MEwjl5DSZq2cWFzHO3K80PaVYSPj1YUDhtmRIwsZEKTDWxnmmn7eUjXN4QvTSBMV4vj3OiYWkwqxYr8f7efi80Z3MbJ5rRx8e0Fx35KKRl2wfNsW/sfnlwvuZkuXNlu3rx7Ev8u2wxA9bplE/ivqHzYnQZXDdlPXKKBzWG666w2A2ecwQOjj61HIlEiXkzE4XXq8VyjZllfM+58MfJAVwzXbPMl6SZCxbFyFRVUKYT85uevpev/TsXmsBJbJQabw0rXi07jpuevKfC8Lau3c2BHCkYg+D/a6/LxzbgfABg0coCKXlGcIBKr3cBqN3DGBbj2vv0MeuQA7/+6gesf3McZ52dw1V0HmLRwA83aFTfiQhGCzKVwm7sWTLH25blOciDjMaR/M8LxP7PrUMjcfrD3iK69xaDYrhUhRH3gI6Am5qfxnpRyXHHnLQo2u5WnvhzK/h0p7Nq4h3ot6lCzYfJxz0s/kIFuCa1aJqVkz5Z9vH7HBDYs2USjtvXZuiZaSRSKk4VufTMY9tZ/uF068Yl+LHlVHxKqBrj63hQgpUztUxwmXC0WHzJ3GiJ+uNnowvMrZikDi/mT8ARCi9w1rLSJho/cDzwspVwphIgHVggh5kkp10Vh7iJRs2FyoQT8MC07Nw0pWXuYDUs2sWHJJvMX1e1ecQL0vzMFZ5zEGacqZVY8AuD5Delbb5YBjn8A/DtAi0M4L0NYmpW1gUEU22cgpdwrpVyZ9+8sYD1Qt7jzlgYJVeO57on+2GNsBQ9U4eSKE+DAzuN8rwAI89iuKB8EtoJvKXh+gKxR4PnNrNdSBoUGj0dUnb9CiEbAqcCSMK/dLoRYLoRYnpJSfh4pr370MuqfUlf5wRVRZ/u/9kL8zRc20UdRtgTA+A9c3yJT+yM9i8raoCCipl5CiDjga+ABKWVIJSAp5XtSyk5Syk7JyYV3f5Q030/8iZ0bdodseCoUhSOyUudmW8rj4k1RLAzMnp6PF1h4r7SJipALIayYIv6plPKbaMwZDdy5HvbvSMHnjVRYHn6c/IuKFVcUCme8AwT5G+Rtu3kY+NB+upybgaYF/1HrFoNm7XwItb9SOTFSw5TFLTuiEbUigPeB9VLKMcU3qfgE/AHGPzyFHyb9jJAQMAyadmxEv/sv4sz+XYOaRpSjm6qiHGOPsdPvvou4ZEhf1v25keQqr9L0lE0gA/i8Aong2cENWflrAnann+c+3k27bi4l5JWWQPiwxDIiGivyHsANwNlCiNV5P/+LwrwnzPuPf8bs93/G6/LicXvxe/38u3Qzrw5+h3u7PY4710y08Lq9OOJUCr6iYBzOAP1u28d5/X6hauJCelyYTou2m9H1ALoFHDESZ4zBiInbue6BvXy2cgPtux9C01RCT+VFIGX4J30pJdI1C+PgpRgHemAcegjpL6iQWRSsqWyt3vw+P5cn3YQnN/wfkc1pY9DTV3H1o5fz7IDXWDxrOT63Cg9ThEcIyavfbqZ5Oxd2p8SsopeX9n0MgYDZzk1T++YVGAsF9z89GhvEXI+IfxQhjvynG9nvQM6Eo8rqaiBiEdW/Q+jFC+iL1Oqt0n3lXNluAv7IzZG9Li+/TP2dg7tTWTxzhRJxRYH0vCiDtl1y80QczOSR8IsfXVciXvEpih54IfdjZMZQpHEIAGnkmJUSg2qjGyBdyOz3omrp0VSKollHE5cYS0LVONL2HYo4xu60s297CjaHNWJCkOLkRWiCdj1b0bpbbQbd+0ZZm6Mo1/jAPQvpnoe0X5BX3yXcxojf7PBUQlS69YMQgjtfvwmb0xr2dUesnUvu6ku9FrXxupWIK0Kp3bgGry14hsEvXouuq91KRWHwgGcGuL8mYts6vX6JXb3SCTlAn6t78My3j9GwTb2g40ITnHFpJ84ZeCaJyVU4/+Y+2GPUZqfCpG2XbB55Ywd3PLWEtO1TzKgEa0cq6Z+JolRxIOJuL7HZK7xrxevxkZGSSWKNhKCwwjbdW5B5MAtN1/KTfTRdCyp+de9bt1CnSU2+Hvs92YdyEIKwzZgVlZ+r793Pdffvx+aQaBr4/aORafMg/mlIHwQyraxNVJQ5dky3SWEbSgjzHBEDCU8hbKeXmGUVdqlhGAYfPPU5/avfzM2n3Ef/5MFMffnb/Gyr+Z//gSvbHZSxGfAFOPDfQVb9vBYw+3ZeNfRSPt81gW9SP1CulpOAcKUYkpJ9XP/gfhwxMn+z0mLxgm8VpF2B2eexMHVTFJUXB8Tdgaj2GVi7UahKerbeiOQ5iBqL0JwlG5FdYYV82isz+HrMLNw5Hjy5XlxZbj59/mtmTZgLwLa1O8Kurv2+ADs37Ak5rmkCoSl/aGXFarMwcvpjPD9rOBffcR7V61fLf6199yyECBeJ4jN/ZDYFN+1VVF5iADvYz0bE3oGwtkWr9hHEDAKcRBR04UTEXIPQ6yBEaKnsaFMhhVxKyRejZ4TEintyPXz2olkhoGmHRjjiHCHnWqw6DVrXC3PcQpcLT0W3lvyHrih9/L4A29fuoPP5Hbn/3dsZ/Ny1+a0AW3TMxaIW3CcJRZU8P8TdC9KDPHgRxoE+GIceQsRcjag6BWJuAvtFgANT9G3mvx2Xgr13lG2PTIX0kRuGQXZ6+B586fszOLDzILs27cXwGwgh8t0tFpuFWo1r0LFPm7DnPjjhDh486yn2bT9AwBc5Fl1R8ZBS8uGTn+PJzeDm5wZw9sCeLJr+B1tWraJT72yVSn8yYOkKxjYwDhThJC9kjyUovty9B+n5BVH1M7SE4QBIIxPcc0Fmga0HwtoiqqYfjwqb2Tmo+T3s3bI/5HidZrVI25uO3+vH7wuYTz7SzOjsPaA7d71+E3GJsRHnDQQCPH3FaJZ+v0LVYamEWO0GnyzfSGJyTTAOYhh+hPAqIT9ZcFwL7qnRmcvWHa3qh9GZq5BUuszOIa/fjD2k4bIVd7Ybd47HFHHIT8Jr0akpj3xwN3GJsXhcHqa/NZsHz3qSEZe8xNLZq/Ln0HWdgU/0D2nmrKgcWG2SjIOYtaXJRdOUiJ9UuL+M3ly+NdGbq5hUWCHvdvHpPD9rOG17nkKV5AQ69G7D8I/vJyuCy+Xv39azf0cKXo+PB898kknDPuHv3zew5PuVPDfgNT548shdulXX5lw19DJsjvBJRYqKizSgbhMVYnryEsWSHFrV6M1VTCqkj/wwHfu0pWOftvm/H0rJiFxnRcDiWSuIiXey8989QTXI3TkevnptJpcOuYBqtZMAuHHkAPoO6sXEYZ/w53fL8XtVTZaKjtAkScm+/CbICsWJYwHnoKAjUnrBswhkJti6IvRapWZNhV2RhyMxuQrV6oa/S+oWHYtV58+ZyyMm/az4cXXQ77Wb1OSpLx5m2Ef30rBNfWKrxHBK12a07NI06rYrooiAtl0lXc7JRLcY2J0BEqr6eOzNHUz+/d+ytk5RKZDgmoaUppZI3zrkgTORGQ8jM59GppyLkVV67Rkq9Io8HMM+uoeH+4wMKVAnBLhdXlbMDe/X8rp9jLt7EjUb16BDr+Coll4DutNrQPf83w3D4OaW97EnzGarouyxWDVe+3YD4MLrFmxa66RJGxcOp1T+cEWUCICxG1yzkM7Lkem3gUwPHpI7BWnrirD3KHFrKtWKHKD9WW24a8xNWKw6FpsFm9OKzWGl14DufPDEVHKzIhS0wSxx+/TlrxTYGm7xrBXc2vZBJeIlgSAkv8JqL7ofpEtfZ/6/bQ5Jm865OGOUiCsiITghKZQupOdX8K0GGaaJtnQhc6MUIXMcKp2QA/S7/yI+3vo2d48bzD1v3MLHW99m8cwVEZtNHI2UkjUL1oV9bensVTx/9ZiwmaGKE0fTNax2K32u6UGXC0/FYtXRLTrNT2/Cm4tfJLFGQtjzQoVZYnMY3PfCciJWoFMoAFO8baA3h6RPwNoBM6mnKNFqFtBrgnQTMcNThg++iDaVzrVymOp1q3HxHecBkLo3HV8RNisjbWxOGvYJHpdK1Y4mjlg7z88cRuvuLfOLnnndXgL+AM44c2U99P0hPDdgTNBnb3faaN2jJX//tgGhCQy/h+q1vYyZsZmkZLUxrSgMGsTdjbA0RlSbhvTvQKbfCYEtYcbmJaQEYUHEXAN6bZDhgiycCMfF0Tc7DJVWyI8moVocWiHrqAT8Bg3b1GfVL2up1agGtZvUzH9tz+Z9JWXiSUtSzUTa92qDOGp5bXMEr4q6XnQ6z343jA+fnMqujXup37IONz9/LR37tGX/jhQ2LfuFGoljaNY2/djpFYoISMANGQ8hsSBt3RGJY8ya4WGF3AZaDTBSQOiAjqjyMsLSxJwt4VnIfBKzPk9eY2ZLa3BeUirv5qQQcqvNSr8HL+brMbOC3CuarqFbdPxeH7pVR9N12vY8hVvaPIDFquN1+2jaoSGj5j1FXJVYajaqwX/rd5XhO6lcVKuTxIuznwgS8Uicdk47TjunXcjxmg2TqVG7BTLdG6kDm0JRAAbgBe8fyIyhiNhBSO8Sgl1zmulCqfojQm43/eGWUxDiyP6NFnMZ0tYGmfslGGkIx7lgPwchSkdiK2yKflExDIOpL37Dl6/NxJXlonq9atzx2iASqsaz+PsVxCXGIgMGn78yA+8x7hNHnINJa8ewZfV2Xhw4NigGXdMFRkApSFERmmCO93OyD+VwYMdBajepQWyVyKUTCkJKP/JAd5CR2/spFMfHhkheiHR9BdlvmSvv/N6bNtCqIRLHIWwdQs6UvnXg32Cu6K2dCrU4OREipeifFEK+d9t+3nvkY1bMXYM9xsb/bjuXgSP6Y7MHP8IPanYPe7eGj0Zp2rER41eOZv60RUwa9gkHdhwksUYV4qvGsXPD7tJ4G5WKFp2a0LhdQ+ZP/R2LzYLf6+eSIedz+ys3oBWxg7GUEpk1BnInlJC1ipMDC6LadIS1BYZ/Lxy8EDgmGkXEIpIXILQqgJkEJNPvBO9yTD+6H0QVSByHZu8cdQsrXa2VwpJxMJN7ugzjj+lLcWW7OXQgk6/HzOKl60Kb6uZkRN5h3rlhN/u2H6DP1T34dNu7/Oifxpf7JlGjQbWI5yjCY3NYqdu8NgumLcLr9pGb6cLr9jFr/Dy+Hfd9keeT2a9D7pQSsFRxcuFHijgAhG9J+EAUGQD3rCO/Zr+dJ+JuTHeMD+RBSL8BI/MlSmuhXOmFfNaEubhzPBjGkQ/U4/KydPZKdm3aGzS249mhPtjD6BYdV/aRFk+HV40HdhyMssWVm+anN+GlOSP487vlQS4qMOvJf/nazELNI33rMNLuwDjQHXImUPj2WwpFAaT+DyPrDWTgAMhwEWpuZCDlyK+uLwn/3TMgdyr4SsfzUOmFfP3iTWFbuFlsFrat/S/o2K0vDcQaoVCW1WFl3kcLGVD7VvpVu4nXbnmH9AMZqsdnIbHaLHS7pBPvLBtF7SY1In5uWWnZEeeQ0kC6Z2OkXodM7Qfe+WAcRO1yKqKGzIWcSWZlQxFGC0RMcO9NWdDfvxvp+i7qJoaj0gt5ozb1sdhCd44DfoO6zYKL2tRuUpPxK0cTlxSb3/ZN0zXsMXaSaiUy463ZpO/PICs9h3mf/Mo9XYZxet/26JbQrkLRbhsX7hqlRgFvRdM1dKte4OaOxWahdfeWPPrh3Xz87Jdc3+TuiGNbdm4W9riUEnnoXmTG8LxVjhF2nEJRfNzgWQiWUzGThA7jAEsrsB2Vcm8/m8gyGi72vGSo9EJ+6ZDzsR4j5FabhWYdG9GkfcOQ8Q1OqcvUnRO47+3b6NmvK5ffeyFDJw9h/7YDQSv7gC9AZmoWDVs3ID4pNj+VXOT1/pRG8f8DbQ5rfrPgiFUdSwghRL5In9K5GWdc2imk/rvVbuX6J6/k3ZWvcOaVXcPOk1AtnvfWvMarv4xk7pQFTHtlBoY/sgjfOebG8C94l4Ln9/Cp0ApFtBE2iH8Y4h81xdvSEuIeQFSdghBHZFPEP1JAOVsHwnlpqZhb6YW8RoNkRv8ykmanNkLTNSw2C2deeQYv/PB4xHMcMXYuvuM8nv5qKHeNuYmstOywwuzO8bBr4x4m/j2GAY9eSmxijHkPPo6IF3a17nX7MAJls/Ls0Ls10w9N4euUyTw06S68Hh8BvwHCXIUn16/Ok188xA1PXcXvXy/hj+nLws5jtVuo16I2ANNGTS+wTILVbqXF6eErS0rvb6i0e0WpIX0ISyO02OvRqs9Aqz4TLW4wQgQvZoReA1F9Ljj6Ycrp4R8HxFwJ1uhHroQjKtHqQogLgHGADkySUr4cjXmjRctOTXl3xWjcuR6zmJa1aG+7TtOaaJbQe549xkb9U+qQmFyFVl2a85VvVtCmaiQ0TSNglHJPUGG6Z2wOG4FAAL/XX+DK+N/lW3DE2Nm9eR/3d38Cd447v/Wd1W7h7Ot6cMYlnTAMg6/GzDzSkekYcjJy2bFuF43a1OdQSmZk8wR0vqAjYLpR8C40kyukN29VE3fCb12hKBpOiLkKocUXarTQ4hCJLyMDD4B7tll7xd4bYW1VwnYeodgrciGEDrwNXAi0Bq4VQrQu7rwlgSPGXmQRNwyDjSu2hqwkhTAzRs8b1AuAZXNW4Snkxmdh3CS6VY+un11CmzNaMmbhM0xP+5Cz+ncrsBqg3xdg37YDTH3xGzwub1D/Uk+ul2/HzSYnMxePy3vcYmSHr9Owdf2IrydUi+eu128yTc16EZl+P3jmmYKeMQK8i4rybhWKEyQG4u5CxEd+Yo+E0GshYm9GxN1VqiIO0XGtdAE2Sym3Sim9wOfAZVGYt1zw3tCP+PT5r0OyNxu2rs/rvz1HQlXzrp1QPQE9zKr9aIQm8n3eEccIgTPeyVUPXxJ2k7Y42GNsNOvYGKvNyo3PXI0jzhl5sDTHb1i6Kax7x2LT2bN5H468jeBIJFSPp0GregDc9fpNIX52Tde44JZz+ODfN6jVqAbSvx1yPyfYjeIC/1qwhGbUKRRRQ8QiaixAi7szyA9eEYiGtXWBnUf9vivvWBBCiNuFEMuFEMtTUlKOfblckpORw8zxc0NX45qgYZt6NGpzZIXZ98beBUaWWOwWkmpUCRGyozm9b3umbH6Tr1Pe55YXB4atLVIc+g7qnf/vei3q8M6yl2nRqUnIOE3XaNGpCVVrJVG/ZZ2wK3efx09y/WoIIbjj1UHYY0LflyPWztNfDc2PaDntnHa8PPdJOvZpS1LNKnTo3YbX5o/koffuJD4pz3Xi/ZOwYTIyF6yngAj5aikUJ4gVsAMO0OqZG5la5EVJeabUbjtSyveklJ2klJ2Sk5NL67LFYt/2lLCrYmlINq/aHnSsVqMaPDH1QWzHCLVu0anRoDoPTbiTDza+QUyCM6LL5J9F/7Luj4355VyPTkA6GpvDGtJw4XCUSSSEEJx+QfCKtl6LOry5+CXOu7EXNocVZ7wDZ5yD2k1qMuLzBwG4ZtgVIdUIbU4bPft1JTHZTFPuPaAHT381lBadmhKT4KRW4xpcM+wKpu2ZGLJ52bbHKYz++Wm+2DuJV38ZSduexzyCivi86nLHYgGtJlR5NuJ7VCiKRgCc/RHVZyKSf0ZY25e1QSdMNIR8N3C087Ne3rEKT82GyWFrkwshaNQ21N/b/bLOfHNwMmde2Q3NYkbICM3cxDv7up7ExDl5/bfnaNS2n1lecgAAIABJREFUQdjruXM8fPfOnPzf6zavHV6cBVx4y9nYnTZiEpxY7VZ69OtCs9MaR3wvMQlOYhNiQo5rmsajH9zDxLVjeHDCnbzw/eN8sGEc1euapQdO6dKcJ798mBoNq2Ox6tgcVvoO6sXQyUOC5ul8wam8vfRlZhz6iI+3vM0tL15HTHyo60ZKA+lZjMz9Eun7O9RQ+9mED1z3g2sWZE+M+B4ViqJhgOsbwEAIgTRyMTJfxNjfBWP/aRiHHkUGKkbmdrGLZgmzTuNG4BxMAV8GXCel/CfSOWVR/fBEGTdkIvM+WhCUTm6PsTH2t+dpduoR4UzZlcqXr37Hb98sJm3PIQzDCBp/+T0XcuvL1wOwZc12Hug5Imx2Y+P2DXhv9WsAbP9nJ/d0HR7k2rHYLLQ4vQnjFr2AK9vF7k37qFa3Kkk1qnBdgztJ2ZUa9n1c/dhl3PrS9Sf8OUgpycnIxR5jy39iKPIcRhoydSAYe8nfPbV1QCRNNHf63T+CzEaKapA5HFANIhQljQ0R/xDE3IxMuxp864DDf+sW0JIRyT8ihKOgSUqNSEWzir2bJqX0CyHuAX7EDD+cXJCIVzTueXMwVarHM/3N2eRk5NKobX3ueeOWIBHfu3U/Qzo/hjvbHTYMz5PrZcbbcxj84nVomobP6w+7gWhzWOl99ZGssUZt6jPym0cYc9u7ZBzMQhoGp/ftyKMfmpmRzjhnkB21mtQIK+SaRePa4f2K9TkIIYhLPLEys4eRGY9DYAdBAu1dhTw0HLw/5w3yYz4oKhFXlAYaCIeZLezfyBERB/CDzPh/e2ceHmdV/fHPeWefbE2TFJBSFpGlZW+BshVsZa8sWlAou4gsgijKT0AEUQQEEZQiFBAVyqIVZCkiiwhikcWibIJilU1omyZtlskyM+/5/XEnaaZ5JzNJZjJL7ud5+jzNu9x73nbm5L7nnvM90PUIREf3/Sk040LGNl+4ruspsfr9+dfx9L1Ls+aQL4ktYvG1D3HX9++jd72UvnBViA03n8SPl17e3+KsD1Wl+YMWojXhITW7lz35Kt8+/Mr0t4dIkDnz9+GrC0/P8SkLg2oPumJnvB302JUyWyzpBJGmP0L3o2j7VYBHKm1kPk7dJWNtmCfjVsY2n2TSyX75D69ldeIiwkfvrGLR935DTyzdiTs+h89+dS4LXrhykBPvu7dpckPWxgu7zNmer916BvUb1BEI+QlGghx4yif58g1fyP5whUYTZHbW1olbCo0P770XF+28BfxTwKubj0TA711tXEqMi1ZvhaauoYY1K9YOeU0wHODJO57xav2Oukq8Jz4oO2QkzP783ux39J60rW4nWhslGBpZPHukqNthdFEkBMFd+0uaxalC/dtCYv0NTrsat4yWADiTwf1PhvNBpPERVGph1d4MCp903YOG5oDTCMke1r01irk3UvplMXZFngeOOu8wwlWhIa9xXaWuqdazCbTjcwhFhr5/ODiOw4SmujF34m7sPnTlnujarxulwpV7ob3L+s9L3RUmvbBfUS6CXUtYRk8c3HfxXnELRA5H/FOQ3me9pWm1G7ofQSbeDaF9MJ9JHwR2QRruzblUv5jYb1EeOOCk/Xj/X//jvuuW4DgO3esVEAXCAXY9cCc+ddwsfnbhXYPu9wV8zD527xHN3dXRxaO3P8Wyx19h0qZNHHbmgWyaqqQcSzTxb2i7FOhOW2Br66kwaSkiYSSwNTQ9jsbuM53KfVtBxzVjbqulEvGSvYiAbwOjUJgD4mtE6m/GFKgrIvlbXBUau9mZRzrWdPLf19/j9T+/yd1X3I+bdEkkkux+yC6c//OziFRH+PNvX+CK+dfj+BwUcBNJzvjRScz90gHDnq+9tYMzp/8frSvX0BPrxfE5BEJ+vnXP15g5d3r2AfKI2341dN7OoM1MqULqrkTCBw66R90WdOU+wODGHxbL6HCACDQ+iOM3NR/qtqEr92ZQRx+JIPU/S28YUaKM6+bLxSART/Dh8hXUNdZS25D+ataxppPnlywjEU+w2yG7UD+pbkRz3HbhIn7zo4eJ96Q7z7qmWu7930J8vrFrRuGuvRi67h18QiJIzbeR6Ge971t1sFmdWyx5xweRw3Dqruo/4nYtgbXfTP2UBPwQPQan9oKiWDhcCpZHbvHGH/CzydbeuiDVE6qYM3+fUc/x7P0vDHLiYHqSvv/W/zKqDRYCCc1Gux8a3PhBXQjtmfnGuiuh5ajCGmcZpySh+2kYsE5yIoeiwRmp4rM+udmtimdinrCOvIypqvVWL3QTSWId3Tz9q6VMmFTH9rO2zZg6mTdC+0Jghims0Bhm4ykMVacgvo0y3+c0FNYuy/jGY6NSfBtA1QlFMKZwWEdexhxx9iFcf8bCtFJ/cYSqCVWct+8lBIJ+FKWmvppr/nApG22xQf91a5vbeO7Bl0jEk+x+6C40TR6dQxVxoP5m6HkM7VpiQirRo5DgbkPf2OvdWchiGT0RiGZoHVhh2Bh5GaOq3Hju7Sy55QnjtF0lWhuhY01nWnWnOMKUbTbm1td+BMAzi5/jqhNvwHEEVVDX5eTLj2XeV+eOrf3xf6LtV0Lvs2M6r6USCZviHe0yhT3aC5EjkNrLyk5bfCjsZmcF0/zBat584W0mblTPbRcs4pWn3xh0TSga5Kd//QF1jbUcO+V0erp6084HI0EWvHBlmsZ6IXFjv4W2b2OKM4rTl9RSijgM+/PgbAJVJyPRYyCxHNwPwL8N4tuwIBYWE1uiX8E0btzA3kfuztSZWxFr925Q7Ph8xNq7WfrgS5566Ml4gj/cPbqVsSabUXdN9uvcGLRdgkkDs07cMhCvz8NQLQ/9EJqFU3UcIj4k8AkktF9FOvGhsI68wth33h6DmlsA9MR6uPO7v+ad19/F6y3MTaqn9nouaPw13FUHoav2Q1fuhbt6Ppr8KPMN8WUZmkdYLF4M5aYS0PPkmFlSqlhHXmEc/uWD2GjzSYMkA9yky18e+isP3vgYid7BVXDBSIB9Pjtz2PNpcjXacjwkl2PCJHGIL0NbjkV18DyqSTT+hkn9slhyIkuz8jIooS80ZePIVZX/vPYury99i97u3uw3jFMi1REWvHglZ153MptO22RQhyHzb6cEwwF8fgcRIRQNcfAps9l2908Mez7tWpxSNhxIEpLNaM9T6ddqL9pyAnQswOqNW/JDBCKZUwlVFe1+HLf1dNyWL6JdD3suMMqdskg//HD5Ci6aewWr3ms2pe2qnHvTacw+ZvRFNZVIKBLi4C/M4dHbn/JsYBGKhjjz+pP58O0VxHvj7P2ZmSNy4gAk38VTw5luWHMubt2PcCL7AxiNlfirDCqRBkwj3CQ2Zm7JnYARxIpmLijTtouge4nJZgE0/gJ0/w4m3NDfFLwSKHlH7rou3/jUd1j5bjM6QPP72i/exKZTN+HjO25WPONKnPoNvEv/3aTLVtM/zoEnfnLUc0hwOtq9ZHBFJwC9sPY8NPiY2XzqfhBvJw7UXY3Qg669JPM1FgsAIYh8Bqk+Y8hNTY2/afq8Dvw8aRf0/tkUrgV3LbypY0TJh1beWPoWbc3taU4cIN6T4MEbf18kq8qDI885hFA0PVbu+Bw22mIDNs/QAHrYhA81Os5k2rx0zYoIGDKcEv8HhI+AwLT82GWpMByQOgjNQSbehlP3neyZKb1L8YyvawzteaYgVhaLknfka1a1eabLuUmXlg9bi2BR+bDjvtP44lXzCUWCRGsjhKMhps5s4tpHNsBtORF37XfQRCYx/twQCSENiyGwU4Yr4qjbbv4a2C7zQIk3UF0L8ZdHZY+lnKiCwM4Q2NtINfinQu01UHs5OBtjdOvDIFHwb4k0PY5T/9Ps1cJ9SB0mZLc+QcSZkL/HKAFKPrQybc+tiXukxYWiIXY/tPRlJ4vN4WcdzAEn7se/lv2H+qYuNp54JujT0NsDvIh23Qf1C5HQ7iOeQ5wJUHsZuvqzDA6LhJHQLPNXZ7PMgyRXQPNnsTHy8USneRMTX0pGduf+MxqZB/G/mYbI/s0hsOvwY9rhA6D9ux4NqBwIf3rU1pcSJb8ir99gAkd9Pb0DTzASZMPNmtj/hFlFtKx8iFRH2GHWVDbe6B7QNtZtTiaALrTtIs/c8uEggU9A5DOYrj99B6MQng2BndH4q9BxdeYBkh+A+96obLCUI92gnWjrqbht16DdT6KaRESQ4M5I9HNIcLcRbUyKU4PU32JW5lIFUg1SjdT/GPFNKsCzFI+yKdH/y8N/5YEbfkf7mk5mzduDT5++v2ejYgto4l3o+aPpmxneH3EmAuCu2B3UKxwVRCY903/diOdVhd4/mVW+Jk2vw9BsQNDmOZB8P8Odfmw6ogUwv/x9myET70KcaF6GVI1D/GVTURzYEcdXn5dxi0HZ65HPnDt9zLvelAuaeBeIg29ztPNG6Lg5dcaBtsvRuh+aFECnGpJejlyB0be1EhEIzVoXSum37x1INg9xZxjoGPX8lmIQBGkA/TA/w2kMEv9GO29Bar5i2q71vmyEsAI7ISOpCHZXox0L+pU23cA007XK//H82FwClI0jtwxGE8vR1i+bla44QBi0g/Qu4ZgUwNCzED0B2q8FBuqxBIy4vlNVQEuzvBb7t4DEKwWc31I4ekFbcr9cNkq9FQ6VYtoDXQ+gge3RtedhPj+pxUb9zUhwx5ynU02gqz8H7kr6M1jir6CrPw9NTyFOde62lzAlHyO3eGOqJOen2qR1m5WMtjDIiYPRNel5GokeB5FDgJCJFxKGwHZI3ffzZJOig6o8Ad8mMFSqWOIN7EexnPEqCAOzWk+FR6QKfFtC7ffILYzmomvOBe00ixPtBG1BW082IZKcTXs6tS80MA1Rjcxt98O5j1Pi2G9PudLzTEqvJIc9DlUggYgPp+4KpOlxpO6HSMNinIZ7EWd4PUNVFbfzDtyVe+J+tDXuyoNwW09HV+yIrpiG23wY2rsujVBEkAk/GWJEF6rOxL4gVhpxiBwHvk0BP8hEWHsO2R15OHWPVym9Qs8fcjch+R6oV3PvLhPyqxDsN6dccVdBzpoRSdOKLYX4Nhx6heyBum1o7E7ofgq0PbVxmVr9u8uhZ/m6ixNvoi0nQeP9iH8LM2dgazQ02/tLGNgdp+YcXLcFuu4all2WUkYhdhv9DjnxQpbrxTSH8E+DwJYQf85jyATo2txN8G+7rtFE2lRRpIKKz0a1IheRq0XkTRF5RUTuF5HKyrIvZQI7ZzjhB4KYuKIfCEHtxaPKSFG3HV19BHT8FBJ/H6B0OBS9aOdtaUek9lI8N1WT7+N23gGJf43YRkupMhyBKgdqr0Em3omE9l0XlklDILhH7kMGdwP/J0j/3AXAaTJ55hXCaEMrjwPbqeoOwD+BC0ZvkiUXJLANhPYjLW+bEPi3gom/hKrTofrLSOMSnOjRo5pLY3dCchWZY6FeJKH7j2jsLtRtM+O4HXi+VrvvQfuVELf9O8c3QfBNNtlPwb1Ti5WBdQkRiBzZ/5aXCyKC1P8Cqo43UhJSD5F5SMOvERms21+u5C2PXESOBOap6vxs19pWb/lBNYnGFkPX3akehZ9GoifmLf+2D3f10abKbkREQPzIxEVoxw3Q81hebbNUGFKPTLgaCc0yG+fdD6FdDwBBJHq00VqpINXC4VLwnp0i8hBwr6remeH8acBpAFOmTJn+zjuVs9FQ6bitZ0LPE6MbxPk4uO8zvFW9pWwI7AWJ11IbizETFpE6cNdPNYxAaH/oeZTM4bkI0vgQ4s+TsFsFMeKenSLyhIi85vHn8AHXXIR5Z16UaRxVXaiqM1R1RlNT00ifw1IEpOokTNFO2lGM4qGAszmEDzNZCZlw3y2YfZYiI1EkeiTS9Eek9ltI9dnIhOug8Q9QdaoJiUgUCEP0GGTCD5CmJyA4B+8agzja9asxfojyJmvWiqp+aqjzInISMBeYo8Wo97cUHAnuhtacD+1Xm5x0TYB/c6T+5kFSou6qOSblyxOb7VryOJNNRpTnm5Ng/g/9A86HwLcxhA8yMefovLSrqTkHrT4NkivB14RIKubt2xAiB6Lx5zy07BOQzFOl6DhhVOmHInIQcD6wr6pnZwFLheBUHYdGPmOKd5wJiH9L7wsjn4GOm0h3BA74tzYpkLGf9XdrMQRB+ipSrfJhUXE2Q5p+D/G/o10PQfdvU2l78ZQW+IFQ9QXofd6kiWo3hA9Fqk4ecuNQJAxeYZLALt4ptBJFQnvn77nGAaOKkYvI25i8ntWpQ39R1dOz3Wc3OysX1R605RRIvG6cgIRAwsjEe8A3xaQkdi4EXWPEkWovRP07wqqZ5FTcZCkQQWh8AtFm41wD0wCF5Efg1BeslN1d+y3ofmjAL/cQ+KcgDfdXVFZJviiIaJaqZliWWcYrIiGYeKdppRV/FXwbpTINzJdSqk+F6lNRdRFJhVqSK1ECZM9NtxSGIFSdBS3zUk1AkkACpBbCc5Dqc4DCOHKp/S4Ed0Vji0yIJXyIybyyTnxYlI2MraVyUFVIvA0opEI0umpWSthofaLg3w7cj8D9H+Uld9u3J1DqIaMApois0+OcD6QWaXwE8TWMsV2W9Rlx1orFkk80/jq66pNoyzy05Wh01b4Q/zvUXILJjOnLYhATK218AGrOSDWdGMqJe7X0KjLOthTerpTq5SAEfFtB6ACTNTIkLumKmANJmsYPsV+OykpLYbFaK5YxQ91OtOUEo9XSfzCGtp6CND0FE+9AOxdC8h0IzECqvwjORtB8KEPHzwPg3wYSrxb6EYaH+/rYzBOYZlqiaTcmxe9zSM1XEQkYRcrYIui81cjH+rZISSEMDGNlK6PvNRuclpLFOnLL2NHzmHeWgiahewkSnY8EF6Sf6n2RrLFzCa2XCVMp9OlwD4UL2oVMesFk/kjNur0HUiXqVcdB1XEAaMeNaMcbw7TDAZ8tzillbGjFMnYkm/F2yt1oclWGezIcH4h2YGK8JRheGRW57F9JSp/Ehzh1aU7cE6cB71DMUASRqpOHeY9lLLGO3JIV1W606z7ctivQ2OKswv6mwYSHEwrOwNPZShQJ7uo9WHCn3IxM/ovx+XEOIVWn5n55+BBT1JWVVOzdmYRMuB4JbDtSAy1jwHj85FuGgSZXoKv2R9sug9jtaNv30OZPocn/Db7WbcFtPRddsZ1pMNF6Jppcse6CwE4Qmkm6YmMY/NtDcE/P+cX3MQgOWVycIp7qejRePtKOUfKruwrJ9Zcdqc7yE38OzscYcmUe2B1pehRpegYJf3LU1loKy3j51FtGiLZ9D9zmAWXUMXBb0LWXpl+nCXT10Sl1wziQgJ6n0NVHmQa69HUKWgA1Fxjn7Z8GNd9AJt42pKKd1C+AYA7ORNeazbzxQHA/ZNJSnMjBw75VAjuYzeX6WzAhKQ8SbyG+j2UP1VhKAvu/ZBmanqcYnNXgQu+f0sMnPX8EdzXpKYJJ0y+x+/f9R0T8OFWfx2n8DU7j/ThVxyMSRBP/xm09HXfFdNxVs3E7F/WPbzSlF5h0uqGQeqTmPIYfA+4jW5qe56TknjMQAmcKWZtRZyWSKosfQUf5FCIy9Eret9GIx7aMPdaRW7KQ6SOy3vHE8lT623poDE38e8gZNPE+uvoo80ujr41c+w/Q9itNd6Lel9DEPyCw9dCmaqtp9Fx3jRF/GhYCE24yXd6HhemHmh0Hqr+CNC2BwO7DnGMgAlUnI6HRjNGHj4z/v5Gj8jC+Zayw6YeWoYkcDF0PY8IlffghvH96OMS/ZUr8ar3qQIlmFthKoZ23eDSS7oLYL9HYXSCB1LjZsjiSaMcCnPrr0eB0U2yUa9m/fzpOeA9UrkJbv0R/eGhYOHhXcUag+kyc6tSmZOQINP5ChmuHohYa7sDJ18Zj73MpNcv1Tzgml99SNtgVuWVIpOZC8G8GUoVRKqwC3yZI7bfTLwzNMn0Q09YGPqOal603YvxlvJ1mEuhJpRfmkoqnRp0RwJmYSrXLkcTLqCoSmmmqSaPHpFb1wwlfRKDqa0ZSgFSXJqmD6nOQqtMGXDYXfNvkMLYD/hkQOR4mLEQ2eCF/ThzAXZPpRErK1lIu2BW5ZUjEqYOGh6B3qdFH8W8Bwb0GxWdF/NBwD9p2eSomrkYsq/bi7AJI/i1MZeKoNUmkX7tFRKDuMrT1bMyq3MV83DOtsgcU0fg3Q2ovRmv+D11zNvT8mdxW9jHoXow0LkEklNrkDQzayBUJQuO9aOx+6H4EpAbcD1MVlwPkfyWMTLgC8W+a6z/A8AjOSHX0WZ8oErKZKuWEFc2yFB2Nv4Gu/jzpLcFGQhhpWIQEtjeaLrFfm3i7dpjQTXA6JFZA75Okb+D6IXQATv11nqO6zYdB4s3cTJAoUvsdJHJ49msHoG4Mbf8edD0IJMD/CaT2MiS487DGGS5u21UQu4t1With8G+JNNxjFQhLkILI2Fos+UACU6F+Adp2idG/xgHf5FScNpsOSB8RpP5GJLA9bucdpptR30pcoqaJQfREaJlP+srfB85kpO7bnqOqG0utlHNEY2jv34btyMWJInXfR2svAxKmGcMYIDXnp2Rk7zL7EOFDkehR1omXGdaRW8YUTbwL8ddMeltgp/6wg4T2gcYnTbqiRMBtRZsPM5koOeGCb1PUbYX2H5AWotAYxJdB6ymp2O/At1AfVJ2AOBn6jUqqL2nOhGEUoRARP2P5tRQRCM9GwrPHbE5L/rGbnZYxQdXFXXM+2nwo2nYR2noy2nxImsaKiKT0QoKIbwOoX0jOm40SNHHm3udBPByhxkyYZVAcvhe67sk8rIQguIeHHQ5GbmC9r5AEkMgRudk8CrTnOdyW43FX7meqaRNvF3xOS+liHbllSNTtQOP/RN2O0Y0Tuxu6H8VkoXSmHOt/0bXnZbxHAtsYB53TBL3g/3hKe9trBS0ZjgOJ5WjPs5ntqLsCfB9LZe6EUqGaHaDxYQjsinHoRkpXJi5CnAm52TxC3K6HTYpk7/Om2UbPo+jqeWj8nwWd11K62NCKxRNVF21PbYSJHzSBRj+H1FwwsorC2J0M3sxMQu8y1G3xDG2IhNDoKR4NmyFN4lUiEDkacSainqtnMKmT1aCrPc4l0NYzYeLtSHD6YDt8k6DxMeh9FhLvQWAbCEw3YYmGO/rboxXagYP5f6H9ctL/LY2Urbb/EJl4c8FtsJQedkVu8UQ7b4HY3fSvoOmB2K/QzptGOmCGE+JdEdp3tvocqP7quhx1/7ZQ+8OUil89+DaD6m+afHdMap/U32r6TUr1uvz3mnOR+h+TuQy/G+24PrMd4kNC+yJVxyHBGWkpheLUjIkTB8BtAc+3I03l41vGI3ZFbvGm82cMXkF3Q+ftUH3W8McL75/6xbBeHrfTYLoAZcA0RjgJqk5KPxH9dOZ7gjvCpKUm/1tjEJzZ329S638OrfMH2wGQRUqgJHBqMp/zNY2dHZaSwq7ILd5oW8bjI6k9kOqzUqvqvhWxH4ggdVcNqXw4UkSCSPiTSOTQtKbBEpxqSv69yCIlUAqIhCByBIOFwSJI1RnFMMlSAtgVucUb/9bryt3Tjm81IscrzkRoXIJ23Wc26XxTkOgxiH9sW4hljruHkepzx9SWkSK1F6MkjAZO335F1dlIZG5xDbMUDVvZafFEe19CW07B5GMrZnMxhNTfkiflveJhGhL/AjpvBrfVVFHWXIiE9ii2acNC3XaTF+/b2KzULRWPrey0DAsJzjDaKR0LTHm6f2uk+iwkMK3Ypo2ajHH3UaDuWuh9yWywBmeMSivce/xWtOsBSPzHlO2HD0acmqFj5pZxg3XkloxIYKpp6GAZErfz59D+w1TsXU2eef3tSCBLI4wc0fibaMv8lMBVN9r9AHT8BBoWI059XuawlDd2s9NiGQXa+zK0X0u/3K52grvKVK5qrjoxWeZY+w3TcKMvi0hjkPwIbf9xXsa3lD/WkVvKFk38F7f9Wty1l6I9T5timbG2IXYPabou605A/K+jH99dY7ovDSIOPY+OenxLZZCX0IqInAdcAzSpanM+xrRYhsLtegjWXoTJB0+g3b815fL1N+U9Pj0kugbvpheSoXBnuPgzjA9GGsBiycOKXEQ2AQ4A3h29ORZLdtTthLXfwoQaUoU9GoP4iyk9l7FDQgfiWS2qcaN/PtrxnerUOOt/VUMQmTfq8S2VQT5CKz8Czie3XlwWy+iJv7guf3ogGkO7Hx5bWyJzIbAV65y5AGGo+brprpQHpO7qAaJdYTNXcGek+kt5Gd9S/owqtCIihwMfqOrfsxWJiMhpwGkAU6aMbRGIpdIYQhFxjBoy9E8nQZi4CLqXoN2PgjMBiRxrZALyNYdvQ2h83Ih2JT8A/zQI7FCQilhLeZLVkYvIE8CGHqcuAi7EhFWyoqoLgYVgCoKGYaPFkk5wV7wVDiNIEcINIkGIHIlEjizgHD4I7Vuw8S3lTVZHrqqf8jouItsDmwN9q/HJwDIR2U1VP8qrlRbLAEQCUH8z2noqoKAu4EL0OCS0V7HNs1jGnBGHVlT1VWBS388i8l9ghs1asYwFEtwFJv0Zup8y+dvBPRH/5GKbZbEUBVvZaSlbRCIQOaTYZlgsRSdvjlxVN8vXWBaLxWLJHVvZabFYLGWOdeQWi8VS5lhHbrFYLGWOdeQWi8VS5hSlQ5CIrALeyXJZIzBeUhnts1Ym9lkrk2I+66aqOqjLdlEceS6IyEteLY0qEfuslYl91sqkFJ/VhlYsFoulzLGO3GKxWMqcUnbkC4ttwBhin7Uysc9amZTcs5ZsjNxisVgsuVHKK3KLxWKx5IB15BaLxVLmlIUjF5HzRERFpLHYthQKEblaRN4UkVdE5H4RmVBsm/KJiBwkIm+JyNsi8s1i21MoRGQTEXlKRN4QkddF5CvFtqnQiIhPRF4WkTHusze2iMgEEVmc+p7+Q0T2KLZNfZS8Ix9HzZ0fB7ZT1R2AfwJMEX5FAAACPUlEQVQXFNmevCGmrf0C4GBgKnCMiEwtrlUFIwGcp6pTgZnAWRX8rH18BfhHsY0YA64HHlXVbYAdKaFnLnlHzjhp7qyqj6lqqiU8f8F0XKoUdgPeVtXlqtoL3AMcXmSbCoKqfqiqy1J/b8d82TcurlWFQ0QmA4cCtxbblkIiInXALOA2AFXtVdU1xbVqHSXtyAc2dy62LWPMKcDvim1EHtkYeG/Az+9Twc6tDxHZDNgZeL64lhSU6zALLbfYhhSYzYFVwO2pMNKtIlJVbKP6KHqHoHw1dy4HhnpWVX0gdc1FmNfzRWNpmyW/iEg18BvgXFVtK7Y9hUBE5gIrVfWvIrJfse0pMH5gF+BsVX1eRK4HvglcXFyzDEV35OOpuXOmZ+1DRE4C5gJztLIS/D8ANhnw8+TUsYpERAIYJ75IVe8rtj0FZC/gMBE5BAgDtSJyp6oeV2S7CsH7wPuq2vd2tRjjyEuCsikIqvTmziJyEHAtsK+qriq2PflERPyYDdw5GAf+InCsqr5eVMMKgJhVxy+AFlU9t9j2jBWpFfnXVXVusW0pFCLyJ+BUVX1LRC4FqlT1G0U2CyiBFbmlnxuAEPB46g3kL6p6enFNyg+qmhCRLwO/B3zAzyrRiafYCzgeeFVE/pY6dqGqPlJEmyz54WxgkYgEgeXAyUW2p5+yWZFbLBaLxZuSzlqxWCwWS3asI7dYLJYyxzpyi8ViKXOsI7dYLJYyxzpyi8ViKXOsI7dYLJYyxzpyi8ViKXP+H2tolxSGs29qAAAAAElFTkSuQmCC\n","text/plain":["<Figure size 432x288 with 1 Axes>"]},"metadata":{"tags":[],"needs_background":"light"}}]},{"cell_type":"markdown","metadata":{"id":"lBcqiop7mH7x"},"source":["## Making it fast with `tf.function`"]},{"cell_type":"markdown","metadata":{"id":"kjZ8kuruNdj6"},"source":["But how fast is our current code running?"]},{"cell_type":"code","metadata":{"id":"NXoe7S5RmStB","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1612742773251,"user_tz":180,"elapsed":4033,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"b82e998e-d8a8-4043-a7c3-eb9ed0b7833c"},"source":["import time\n","\n","t0 = time.time()\n","for epoch in range(20):\n","  for step, (x, y) in enumerate(dataset):\n","    loss = train_on_batch(x, y)\n","t_end = time.time() - t0\n","print('Time per epoch: %.3f s' % (t_end / 20,))\n"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Time per epoch: 0.164 s\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"kHsvPqyRN_E3"},"source":["Let's compile the training function into a static graph. Literally all we need to do is add the `tf.function` decorator on it:"]},{"cell_type":"code","metadata":{"id":"oEYFkThcOGcg"},"source":["@tf.function\n","def train_on_batch(x, y):\n","  with tf.GradientTape() as tape:\n","    predictions = compute_predictions(x)\n","    loss = compute_loss(y, predictions)\n","    dloss_dw, dloss_db = tape.gradient(loss, [w, b])\n","  w.assign_sub(learning_rate * dloss_dw)\n","  b.assign_sub(learning_rate * dloss_db)\n","  return loss"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ocOuskvoOKsx"},"source":["Let's try this again:"]},{"cell_type":"code","metadata":{"id":"KT2w6DVmONB5","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1612742798476,"user_tz":180,"elapsed":2407,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"e16dea65-d4aa-41d4-b579-7ed4eeffb1a8"},"source":["t0 = time.time()\n","for epoch in range(20):\n","  for step, (x, y) in enumerate(dataset):\n","    loss = train_on_batch(x, y)\n","t_end = time.time() - t0\n","print('Time per epoch: %.3f s' % (t_end / 20,))"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Time per epoch: 0.080 s\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"YYPWZaSqOfEL"},"source":["40% reduction, neat. In this case we used a trivially simple model; in general the bigger the model the greater the speedup you can get by leveraging static graphs.\n","\n","Remember: eager execution is great for debugging and printing results line-by-line, but when it's time to scale, static graphs are a researcher's best friends."]},{"cell_type":"markdown","metadata":{"id":"B3I3v_FqjFty"},"source":["# Part 2: The Keras API"]},{"cell_type":"markdown","metadata":{"id":"FjLI719fPfJi"},"source":["Keras is a Python API for deep learning. It has something for everyone:\n","\n","- If you're an engineer, Keras provides you with reusable blocks such as layers, metrics, training loops, to support common use cases. It provides a high-level user experience that's accessible and productive.\n","\n","- If you're a researcher, you may prefer not to use these built-in blocks such as layers and training loops, and instead create your own. Of course, Keras allows you to do this. In this case, Keras provides you with templates for the blocks you write, it provides you with structure, with an API standard for things like Layers and Metrics. This structure makes your code easy to share with others and easy to integrate in production workflows.\n","\n","- The same is true for library developers: TensorFlow is a large ecosystem. It has many different libraries. In order for different libraries to be able to talk to each other and share components, they need to follow an API standard. That's what Keras provides.\n","\n","Crucially, Keras brings high-level UX and low-level flexibility together fluently: you no longer have on one hand, a high-level API that's easy to use but inflexible, and on the other hand a low-level API that's flexible but only approachable by experts. Instead, you have a spectrum of workflows, from the  very high-level to the very low-level. Workflows that are all compatible because they're built on top of the same concepts and objects.\n","\n","![Spectrum of Keras workflows](https://keras-dev.s3.amazonaws.com/tutorials-img/spectrum-of-workflows.png)\n","\n","\n"]},{"cell_type":"markdown","metadata":{"id":"o9DSVjdHPkOw"},"source":["## The base `Layer` class\n","\n","The first class you need to know is [`Layer`](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer). Pretty much everything in Keras derives from it.\n","\n","A Layer encapsulates a state (weights) and some computation (defined in the `call` method)."]},{"cell_type":"code","metadata":{"id":"Io3dUQzaPnPc","executionInfo":{"status":"ok","timestamp":1613057465631,"user_tz":180,"elapsed":6134,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}}},"source":["from tensorflow.keras.layers import Layer\n","\n","class Linear(Layer):\n","  \"\"\"y = x.w + b\"\"\"\n","\n","  def __init__(self, units=32, input_dim=32):\n","      super(Linear, self).__init__()\n","      w_init = tf.random_normal_initializer()\n","      self.w = tf.Variable(initial_value=w_init(shape=(input_dim, units), dtype='float32'),trainable=True)\n","      b_init = tf.zeros_initializer()\n","      self.b = tf.Variable(initial_value=b_init(shape=(units,), dtype='float32'),trainable=True)\n","\n","  def call(self, inputs):\n","      return tf.matmul(inputs, self.w) + self.b\n","\n","# Instantiate our layer.\n","linear_layer = Linear(4, 2)"],"execution_count":4,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Vo3etyK8BO4a"},"source":["A layer instance works like a function. Let's call it on some data:"]},{"cell_type":"code","metadata":{"id":"hBUCLfHVBQLF"},"source":["# (instances, features)\n","# tf.ones (2,2) represents, in this example, 2 instances and 2 features\n","\n","y = linear_layer(tf.ones((2, 2)))\n","assert y.shape == (2, 4)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"MXqpznsxBaCC"},"source":["The `Layer` class takes care of tracking the weights assigned to it as attributes:"]},{"cell_type":"code","metadata":{"id":"j_FaUtEYBeJw"},"source":["# Weights are automatically tracked under the `weights` property.\n","assert linear_layer.weights == [linear_layer.w, linear_layer.b]"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"WOpnrPmMsv0k","executionInfo":{"status":"ok","timestamp":1612973282640,"user_tz":180,"elapsed":1211,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"00a95299-3619-48cb-8178-0faa0d5f3c5a"},"source":["linear_layer.weights"],"execution_count":null,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[<tf.Variable 'Variable:0' shape=(2, 4) dtype=float32, numpy=\n"," array([[ 0.03018152,  0.03737427,  0.01635162,  0.06385078],\n","        [ 0.01258114,  0.04676739, -0.0638971 ,  0.06425511]],\n","       dtype=float32)>,\n"," <tf.Variable 'Variable:0' shape=(4,) dtype=float32, numpy=array([0., 0., 0., 0.], dtype=float32)>]"]},"metadata":{"tags":[]},"execution_count":7}]},{"cell_type":"markdown","metadata":{"id":"b6PZ6QXUHdxA"},"source":["Note that's also a shortcut method for creating weights: `add_weight`. Instead of doing\n","\n","```python\n","w_init = tf.random_normal_initializer()\n","self.w = tf.Variable(initial_value=w_init(shape=shape, dtype='float32'))\n","```\n","\n","You would typically do:\n","\n","```python\n","self.w = self.add_weight(shape=shape, initializer='random_normal')\n","```"]},{"cell_type":"markdown","metadata":{"id":"lphpMGIiHRUP"},"source":["It’s good practice to create weights in a separate `build` method, called lazily with the shape of the first inputs seen by your layer. Here, this pattern prevents us from having to specify input_dim in the constructor:"]},{"cell_type":"code","metadata":{"id":"BpPjScZKHXhS","executionInfo":{"status":"ok","timestamp":1613057472359,"user_tz":180,"elapsed":2259,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}}},"source":["class Linear(Layer):\n","  \"\"\"y = x.w + b\"\"\"\n","\n","  def __init__(self, units=32):\n","      super(Linear, self).__init__()\n","      self.units = units\n","\n","  def build(self, input_shape):\n","      self.w = self.add_weight(shape=(input_shape[-1], self.units),\n","                               initializer='random_normal',\n","                               trainable=True)\n","      self.b = self.add_weight(shape=(self.units,),\n","                               initializer='random_normal',\n","                               trainable=True)\n","\n","  def call(self, inputs):\n","      return tf.matmul(inputs, self.w) + self.b\n","\n","\n","# Instantiate our lazy layer.\n","linear_layer = Linear(4)\n","\n","# This will also call `build(input_shape)` and create the weights.\n","# two instances with three features each ones\n","y = linear_layer(tf.ones((2, 3)))\n","assert len(linear_layer.weights) == 2"],"execution_count":5,"outputs":[]},{"cell_type":"code","metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"kmoBcuTyJbE7","executionInfo":{"status":"ok","timestamp":1613057472689,"user_tz":180,"elapsed":496,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"3d1381e4-b34f-4c71-f738-64ed01e362d9"},"source":["linear_layer.weights"],"execution_count":6,"outputs":[{"output_type":"execute_result","data":{"text/plain":["[<tf.Variable 'linear_1/Variable:0' shape=(3, 4) dtype=float32, numpy=\n"," array([[ 0.08239684, -0.03070358, -0.02475566,  0.09632542],\n","        [-0.09029395, -0.02019074,  0.03192595, -0.07288326],\n","        [ 0.01096379, -0.04184381, -0.01880849,  0.00200797]],\n","       dtype=float32)>,\n"," <tf.Variable 'linear_1/Variable:0' shape=(4,) dtype=float32, numpy=array([ 0.0131527 ,  0.04130157, -0.03308423, -0.10247675], dtype=float32)>]"]},"metadata":{"tags":[]},"execution_count":6}]},{"cell_type":"markdown","metadata":{"id":"86khdsF3Pnr0"},"source":["## Trainable and non-trainable weights"]},{"cell_type":"markdown","metadata":{"id":"32zvCEKLICr5"},"source":["Weights created by layers can be either trainable or non-trainable. They're exposed in `trainable_weights` and `non_trainable_weights`. Here's a layer with a non-trainable weight:"]},{"cell_type":"code","metadata":{"id":"LZ8s28NnX20u","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1613057476027,"user_tz":180,"elapsed":809,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"55368ff3-97b0-4485-f7f4-09daf3625ea4"},"source":["from tensorflow.keras.layers import Layer\n","\n","class ComputeSum(Layer):\n","  \"\"\"Returns the sum of the inputs.\"\"\"\n","\n","  def __init__(self, input_dim):\n","      super(ComputeSum, self).__init__()\n","      # Create a non-trainable weight.\n","      self.total = tf.Variable(initial_value=tf.zeros((input_dim,)),\n","                               trainable=False)\n","\n","  def call(self, inputs):\n","      self.total.assign_add(tf.reduce_sum(inputs, axis=0))\n","      return self.total  \n","\n","my_sum = ComputeSum(2)\n","x = tf.ones((2, 2))\n","\n","y = my_sum(x)\n","print(y.numpy())  # [2. 2.]\n","\n","y = my_sum(x)\n","print(y.numpy())  # [4. 4.]\n","\n","assert my_sum.weights == [my_sum.total]\n","assert my_sum.non_trainable_weights == [my_sum.total]\n","assert my_sum.trainable_weights == []"],"execution_count":7,"outputs":[{"output_type":"stream","text":["[2. 2.]\n","[4. 4.]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"6oBkX6ZfYO8j"},"source":["## Recursively composing layers"]},{"cell_type":"markdown","metadata":{"id":"YeLDL9MJI2dK"},"source":["Layers can be recursively nested to create bigger computation blocks. Each layer will track the weights of its sublayers (both trainable and non-trainable)."]},{"cell_type":"code","metadata":{"id":"N5HBH-dtYQuk","executionInfo":{"status":"ok","timestamp":1613057786026,"user_tz":180,"elapsed":781,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}}},"source":["# Let's reuse the Linear class\n","# with a `build` method that we defined above.\n","\n","class MLP(Layer):\n","    \"\"\"Simple stack of Linear layers.\"\"\"\n","\n","    def __init__(self):\n","        super(MLP, self).__init__()\n","        self.linear_1 = Linear(32)\n","        self.linear_2 = Linear(32)\n","        self.linear_3 = Linear(10)\n","\n","    def call(self, inputs):\n","        x = self.linear_1(inputs)\n","        x = tf.nn.relu(x)\n","        x = self.linear_2(x)\n","        x = tf.nn.relu(x)\n","        return self.linear_3(x)\n","\n","mlp = MLP()\n","\n","# The first call to the `mlp` object will create the weights.\n","# 3 instances using 64 features\n","y = mlp(tf.ones(shape=(3, 64)))\n","\n","# Weights are recursively tracked.\n","assert len(mlp.weights) == 6"],"execution_count":9,"outputs":[]},{"cell_type":"code","metadata":{"id":"5pfvma4-Khcl"},"source":["mlp.weights"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ECtx6wgIH3jv"},"source":["```python\n","z1 = (3 x 64) x (64 x 32) + (32,) = 3 x 32\n","z2 = (3 x 32) x (32 x 32) + (32,) = 3 x 32\n","Z3 = (3 x 32) x (32 x 10) + (10,) = 3 x 10\n","W1 = 64 x 32\n","b0 = (32,) \n","W2 = 32 x 32\n","b1 = (32,)\n","W3 = 32 x 10\n","b2 = (10,0)\n","```"]},{"cell_type":"markdown","metadata":{"id":"WavMVtXGQk-z"},"source":["## Built-in layers\n","\n","Keras provides you with a [wide range of built-in layers](https://www.tensorflow.org/api_docs/python/tf/keras/layers/), so that you don't have to implement your own layers all the time.\n","\n","- Convolution layers\n","- Transposed convolutions\n","- Separateable convolutions\n","- Average and max pooling\n","- Global average and max pooling\n","- LSTM, GRU (with built-in cuDNN acceleration)\n","- BatchNormalization\n","- Dropout\n","- Attention\n","- ConvLSTM2D\n","- etc.\n","\n","\n"]},{"cell_type":"markdown","metadata":{"id":"pdrw7OppQ6At"},"source":["Keras follows the principles of exposing good default configurations, so that layers will work fine out of the box for most use cases if you leave keyword arguments to their default value. For instance, the `LSTM` layer uses an orthogonal recurrent matrix initializer by default, and initializes the forget gate bias to one by default."]},{"cell_type":"markdown","metadata":{"id":"_oq88tadFz8Z"},"source":["## The `training` argument in `call`\n","\n"]},{"cell_type":"markdown","metadata":{"id":"A2NkTT0AQV8j"},"source":["Some layers, in particular the `BatchNormalization` layer and the `Dropout` layer, have different behaviors during training and inference. For such layers, it is standard practice to expose a `training` (boolean) argument in the `call` method.\n","\n","By exposing this argument in `call`, you enable the built-in training and evaluation loops (e.g. `fit`) to correctly use the layer in training and inference."]},{"cell_type":"code","metadata":{"id":"ysXzHB5KJiLt","executionInfo":{"status":"ok","timestamp":1613057962379,"user_tz":180,"elapsed":732,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}}},"source":["from tensorflow.keras.layers import Layer\n","\n","class Dropout(Layer):\n","  \n","  def __init__(self, rate):\n","    super(Dropout, self).__init__()\n","    self.rate = rate\n","\n","  def call(self, inputs, training=None):\n","    if training:\n","      return tf.nn.dropout(inputs, rate=self.rate)\n","    return inputs\n","\n","class MLPWithDropout(Layer):\n","\n","  def __init__(self):\n","      super(MLPWithDropout, self).__init__()\n","      self.linear_1 = Linear(32)\n","      self.dropout = Dropout(0.5)\n","      self.linear_3 = Linear(10)\n","\n","  def call(self, inputs, training=None):\n","      x = self.linear_1(inputs)\n","      x = tf.nn.relu(x)\n","      x = self.dropout(x, training=training)\n","      return self.linear_3(x)\n","    \n","mlp = MLPWithDropout()\n","y_train = mlp(tf.ones((2, 2)), training=True)\n","y_test = mlp(tf.ones((2, 2)), training=False)"],"execution_count":11,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"SyC7KfV-YcYS"},"source":["## A more Functional way of defining models"]},{"cell_type":"markdown","metadata":{"id":"CxbEQANKQB6F"},"source":["To build deep learning models, you don't have to use object-oriented programming all the time. Layers can also be composed functionally, like this (we call it the \"Functional API\"):"]},{"cell_type":"code","metadata":{"id":"jiL-0N7sYc6X","executionInfo":{"status":"ok","timestamp":1613058579495,"user_tz":180,"elapsed":785,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}}},"source":["# We use an `Input` object to describe the shape and dtype of the inputs.\n","# This is the deep learning equivalent of *declaring a type*.\n","# The shape argument is per-sample; it does not include the batch size.\n","# The functional API focused on defining per-sample transformations.\n","# The model we create will automatically batch the per-sample transformations,\n","# so that it can be called on batches of data.\n","inputs = tf.keras.Input(shape=(16,))\n","\n","# We call layers on these \"type\" objects\n","# and they return updated types (new shapes/dtypes).\n","x = Linear(32)(inputs) # We are reusing the Linear layer we defined earlier.\n","x = Dropout(0.5)(x) # We are reusing the Dropout layer we defined earlier.\n","outputs = Linear(10)(x)\n","\n","# A functional `Model` can be defined by specifying inputs and outputs.\n","# A model is itself a layer like any other.\n","model = tf.keras.Model(inputs, outputs)\n","\n","# A functional model already has weights, before being called on any data.\n","# That's because we defined its input shape in advance (in `Input`).\n","assert len(model.weights) == 4\n","\n","# Let's call our model on some data.\n","y = model(tf.ones((2, 16)))\n","assert y.shape == (2, 10)"],"execution_count":12,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"vK5HqnT3Xgcz"},"source":["The Functional API tends to be more concise than subclassing, and provides a few other advantages (generally the same advantages that functional, typed languages provide over untyped OO development). However, it can only be used to define DAGs of layers -- recursive networks should be defined as `Layer` subclasses instead.\n","\n","Key differences between models defined via subclassing and Functional models are explained in [this blog post](https://medium.com/tensorflow/what-are-symbolic-and-imperative-apis-in-tensorflow-2-0-dfccecb01021).\n","\n","Learn more about the Functional API [here](https://www.tensorflow.org/alpha/guide/keras/functional).\n","\n","In your research workflows, you may often find yourself mix-and-matching OO models and Functional models."]},{"cell_type":"markdown","metadata":{"id":"6p0KngmPTScu"},"source":["For models that are simple stacks of layers with a single input and a single output, you can also use the `Sequential` class which turns a list of layers into a `Model`:"]},{"cell_type":"code","metadata":{"id":"QNhTY6frTaP2","executionInfo":{"status":"ok","timestamp":1613058619655,"user_tz":180,"elapsed":763,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}}},"source":["from tensorflow.keras import Sequential\n","\n","model = Sequential([Linear(32), Dropout(0.5), Linear(10)])\n","\n","y = model(tf.ones((2, 16)))\n","assert y.shape == (2, 10)"],"execution_count":13,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"cydf3i_FFXlh"},"source":["## Loss classes\n","\n","Keras features a wide range of built-in loss classes, like `BinaryCrossentropy`, `CategoricalCrossentropy`, `KLDivergence`, etc. They work like this:"]},{"cell_type":"code","metadata":{"id":"019Nm1eWFaUO","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1613058627244,"user_tz":180,"elapsed":697,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"74ac6bb5-589f-4a28-8246-312ce6dd6c32"},"source":["bce = tf.keras.losses.BinaryCrossentropy()\n","y_true = [0., 0., 1., 1.]  # Targets\n","y_pred = [1., 1., 1., 0.]  # Predictions\n","loss = bce(y_true, y_pred)\n","print('Loss:', loss.numpy())"],"execution_count":14,"outputs":[{"output_type":"stream","text":["Loss: 11.522857\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"RNLZsnswFbE_"},"source":["## Metric classes\n","\n","Keras also features a wide range of built-in metric classes, such as `BinaryAccuracy`, `AUC`, `FalsePositives`, etc.\n","\n","Unlike losses, metrics are stateful. You update their state using the `update_state` method, and you query the scalar metric result using `result`:"]},{"cell_type":"code","metadata":{"id":"4dUZkMWATKMC","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1613058686839,"user_tz":180,"elapsed":657,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"8b1abfa4-67c5-4181-db47-a5392501dbda"},"source":["m = tf.keras.metrics.AUC()\n","m.update_state([0, 1, 1, 1], [0, 1, 0, 0])\n","print('Intermediate result:', m.result().numpy())\n","\n","m.update_state([1, 1, 1, 1], [0, 1, 1, 0])\n","print('Final result:', m.result().numpy())"],"execution_count":15,"outputs":[{"output_type":"stream","text":["Intermediate result: 0.6666667\n","Final result: 0.71428573\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"doUSrciie2Px"},"source":["The internal state can be cleared with `metric.reset_states`."]},{"cell_type":"markdown","metadata":{"id":"uwx7DjFBZ-C-"},"source":["You can easily roll out your own metrics by subclassing the `Metric` class:\n","\n","- Create the state variables in `__init__`\n","- Update the variables given `y_true` and `y_pred` in `update_state`\n","- Return the metric result in `result`\n","- Clear the state in `reset_states`\n","\n","Here's a quick implementation of a `BinaryTruePositives` metric as a demonstration:"]},{"cell_type":"code","metadata":{"id":"vVByLrJyaBx_","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1613058693921,"user_tz":180,"elapsed":708,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"b5492ec9-d2d4-4823-a2ff-9e34b5fb8d43"},"source":["class BinaryTruePositives(tf.keras.metrics.Metric):\n","\n","  def __init__(self, name='binary_true_positives', **kwargs):\n","    super(BinaryTruePositives, self).__init__(name=name, **kwargs)\n","    self.true_positives = self.add_weight(name='tp', initializer='zeros')\n","\n","  def update_state(self, y_true, y_pred, sample_weight=None):\n","    y_true = tf.cast(y_true, tf.bool)\n","    y_pred = tf.cast(y_pred, tf.bool)\n","\n","    values = tf.logical_and(tf.equal(y_true, True), tf.equal(y_pred, True))\n","    values = tf.cast(values, self.dtype)\n","    if sample_weight is not None:\n","      sample_weight = tf.cast(sample_weight, self.dtype)\n","      values = tf.multiply(values, sample_weight)\n","    self.true_positives.assign_add(tf.reduce_sum(values))\n","\n","  def result(self):\n","    return self.true_positives\n","\n","  def reset_states(self):\n","    self.true_positive.assign(0)\n","\n","m = BinaryTruePositives()\n","m.update_state([0, 1, 1, 1], [0, 1, 0, 0])\n","print('Intermediate result:', m.result().numpy())\n","\n","m.update_state([1, 1, 1, 1], [0, 1, 1, 0])\n","print('Final result:', m.result().numpy())"],"execution_count":16,"outputs":[{"output_type":"stream","text":["Intermediate result: 1.0\n","Final result: 3.0\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"i0PdvHdAdQl0"},"source":["## Optimizer classes & a quick end-to-end training loop\n","\n","You don't normally have to define by hand how to update your variables during gradient descent, like we did in our initial linear regression example. You would usually use one of the built-in Keras optimizer, like `SGD`, `RMSprop`, or `Adam`.\n","\n","Here's a simple MNSIT example that brings together loss classes, metric classes, and optimizers."]},{"cell_type":"code","metadata":{"id":"4jNl1ykEdkj8"},"source":["from tensorflow.keras import layers\n","\n","# Prepare a dataset.\n","(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n","x_train = x_train[:].reshape(60000, 784).astype('float32') / 255\n","dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n","dataset = dataset.shuffle(buffer_size=1024).batch(64)\n","\n","# Instantiate a simple classification model\n","model = tf.keras.Sequential([\n","  layers.Dense(256, activation=tf.nn.relu),\n","  layers.Dense(256, activation=tf.nn.relu),\n","  layers.Dense(10)\n","])\n","\n","# Instantiate a logistic loss function that expects integer targets.\n","loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n","\n","# Instantiate an accuracy metric.\n","accuracy = tf.keras.metrics.SparseCategoricalAccuracy()\n","\n","# Instantiate an optimizer.\n","optimizer = tf.keras.optimizers.Adam()\n","\n","# Iterate over the batches of the dataset.\n","for step, (x, y) in enumerate(dataset):\n","  # Open a GradientTape.\n","  with tf.GradientTape() as tape:\n","\n","    # Forward pass.\n","    logits = model(x)\n","\n","    # Loss value for this batch.\n","    loss_value = loss(y, logits)\n","     \n","  # Get gradients of loss wrt the weights.\n","  gradients = tape.gradient(loss_value, model.trainable_weights)\n","  \n","  # Update the weights of our linear layer.\n","  optimizer.apply_gradients(zip(gradients, model.trainable_weights))\n","\n","  # Update the running accuracy.\n","  accuracy.update_state(y, logits)\n","  \n","  # Logging.\n","  if step % 100 == 0:\n","    print('Step:', step)\n","    print('Loss from last step: %.3f' % loss_value)\n","    print('Total running accuracy so far: %.3f' % accuracy.result())"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"MIJYBrXoekXD"},"source":["We can reuse our `SparseCategoricalAccuracy` metric instance to implement a testing loop:"]},{"cell_type":"code","metadata":{"id":"sl6FKvqbeqX9","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1613058739360,"user_tz":180,"elapsed":714,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"075cee7e-7b46-433d-934a-0daeb3d8166a"},"source":["x_test = x_test[:].reshape(10000, 784).astype('float32') / 255\n","test_dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test))\n","test_dataset = test_dataset.batch(128)\n","\n","accuracy.reset_states()  # This clears the internal state of the metric\n","\n","for step, (x, y) in enumerate(test_dataset):\n","  logits = model(x)\n","  accuracy.update_state(y, logits)\n","\n","print('Final test accuracy: %.3f' % accuracy.result())"],"execution_count":18,"outputs":[{"output_type":"stream","text":["Final test accuracy: 0.965\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"CEP7jzC8YVWy"},"source":["## The `add_loss` method\n","\n","Sometimes you need to compute loss values on the fly during a foward pass (especially regularization losses). Keras allows you to compute loss values at any time, and to recursively keep track of them via the `add_loss` method.\n","\n","Here's an example of a layer that adds a sparsity regularization loss based on the L2 norm of the inputs:"]},{"cell_type":"code","metadata":{"id":"LbBVP--jYgHg","executionInfo":{"status":"ok","timestamp":1613058745300,"user_tz":180,"elapsed":768,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}}},"source":["from tensorflow.keras.layers import Layer\n","\n","class ActivityRegularization(Layer):\n","  \"\"\"Layer that creates an activity sparsity regularization loss.\"\"\"\n","  \n","  def __init__(self, rate=1e-2):\n","    super(ActivityRegularization, self).__init__()\n","    self.rate = rate\n","  \n","  def call(self, inputs):\n","    # We use `add_loss` to create a regularization loss\n","    # that depends on the inputs.\n","    self.add_loss(self.rate * tf.reduce_sum(tf.square(inputs)))\n","    return inputs"],"execution_count":19,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"m4qoQk7abK5v"},"source":["Loss values added via `add_loss` can be retrieved in the `.losses` list property of any `Layer` or `Model`:"]},{"cell_type":"code","metadata":{"id":"VlJc_4pbbQ2N","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1613058748814,"user_tz":180,"elapsed":714,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"7ff5001c-e17d-4997-f97e-abb0ac6a1a10"},"source":["from tensorflow.keras import layers\n","\n","class SparseMLP(Layer):\n","  \"\"\"Stack of Linear layers with a sparsity regularization loss.\"\"\"\n","\n","  def __init__(self, output_dim):\n","      super(SparseMLP, self).__init__()\n","      self.dense_1 = layers.Dense(32, activation=tf.nn.relu)\n","      self.regularization = ActivityRegularization(1e-2)\n","      self.dense_2 = layers.Dense(output_dim)\n","\n","  def call(self, inputs):\n","      x = self.dense_1(inputs)\n","      x = self.regularization(x)\n","      return self.dense_2(x)\n","    \n","\n","mlp = SparseMLP(1)\n","y = mlp(tf.ones((10, 10)))\n","\n","print(mlp.losses)  # List containing one float32 scalar"],"execution_count":20,"outputs":[{"output_type":"stream","text":["[<tf.Tensor: shape=(), dtype=float32, numpy=0.826754>]\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"jkI3GA2TbWvY"},"source":["These losses are cleared by the top-level layer at the start of each forward pass -- they don't accumulate. So `layer.losses` always contain only the losses created during the last forward pass. You would typically use these losses by summing them before computing your gradients when writing a training loop."]},{"cell_type":"code","metadata":{"id":"2m0xNYGEbZe2","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1613058758074,"user_tz":180,"elapsed":6046,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"3ffcdfb8-3792-404a-acd6-21a7503120b8"},"source":["# Losses correspond to the *last* forward pass.\n","mlp = SparseMLP(1)\n","mlp(tf.ones((10, 10)))\n","assert len(mlp.losses) == 1\n","mlp(tf.ones((10, 10)))\n","assert len(mlp.losses) == 1  # No accumulation.\n","\n","# Let's demonstrate how to use these losses in a training loop.\n","\n","# Prepare a dataset.\n","(x_train, y_train), _ = tf.keras.datasets.mnist.load_data()\n","dataset = tf.data.Dataset.from_tensor_slices(\n","    (x_train.reshape(60000, 784).astype('float32') / 255, y_train))\n","dataset = dataset.shuffle(buffer_size=1024).batch(64)\n","\n","# A new MLP.\n","mlp = SparseMLP(10)\n","\n","# Loss and optimizer.\n","loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n","optimizer = tf.keras.optimizers.SGD(learning_rate=0.1)\n","\n","for step, (x, y) in enumerate(dataset):\n","  with tf.GradientTape() as tape:\n","    # Forward pass.\n","    logits = mlp(x)\n","\n","    # External loss value for this batch.\n","    loss = loss_fn(y, logits)\n","    \n","    # Add the losses created during the forward pass.\n","    loss += sum(mlp.losses)\n","     \n","    # Get gradients of loss wrt the weights.\n","    gradients = tape.gradient(loss, mlp.trainable_weights)\n","  \n","  # Update the weights of our linear layer.\n","  optimizer.apply_gradients(zip(gradients, mlp.trainable_weights))\n","  \n","  # Logging.\n","  if step % 100 == 0:\n","    print('Loss at step %d: %.3f' % (step, loss))"],"execution_count":21,"outputs":[{"output_type":"stream","text":["Loss at step 0: 3.732\n","Loss at step 100: 2.304\n","Loss at step 200: 2.221\n","Loss at step 300: 2.220\n","Loss at step 400: 2.197\n","Loss at step 500: 2.106\n","Loss at step 600: 1.922\n","Loss at step 700: 1.901\n","Loss at step 800: 1.860\n","Loss at step 900: 1.699\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"7V0yRrfYFuVT"},"source":["## Using built-in training loops"]},{"cell_type":"markdown","metadata":{"id":"BNwAjgEXPpnP"},"source":["It would be a bit silly if you had to write your own low-level training loops every time for simple use cases. Keras provides you with a built-in training loop on the `Model` class. If you want to use it, either subclass from the `Model` class, or create a `Functional` or `Sequential` model.\n","\n","To demonstrate it, let's reuse the MNIST setup from above:"]},{"cell_type":"code","metadata":{"id":"omdNf2x4jovv","executionInfo":{"status":"ok","timestamp":1613058769230,"user_tz":180,"elapsed":1031,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}}},"source":["# Prepare a dataset.\n","(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n","x_train = x_train.reshape(60000, 784).astype('float32') / 255\n","dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n","dataset = dataset.shuffle(buffer_size=1024).batch(64)\n","\n","# Instantiate a simple classification model\n","model = tf.keras.Sequential([\n","  layers.Dense(256, activation=tf.nn.relu),\n","  layers.Dense(256, activation=tf.nn.relu),\n","  layers.Dense(10)\n","])\n","\n","# Instantiate a logistic loss function that expects integer targets.\n","loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n","\n","# Instantiate an accuracy metric.\n","accuracy = tf.keras.metrics.SparseCategoricalAccuracy()\n","\n","# Instantiate an optimizer.\n","optimizer = tf.keras.optimizers.Adam()"],"execution_count":22,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"MYU5PkaLiriO"},"source":["First, call `compile` to configure the optimizer, loss, and metrics to monitor."]},{"cell_type":"code","metadata":{"id":"Eo6GhvzjJjdb","executionInfo":{"status":"ok","timestamp":1613058770183,"user_tz":180,"elapsed":660,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}}},"source":["model.compile(optimizer=optimizer, loss=loss, metrics=[accuracy])"],"execution_count":23,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"98zDjMPej06U"},"source":["Then we call `fit` on our model to pass it the data:"]},{"cell_type":"code","metadata":{"id":"-OdztL4nj4ed","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1613058778734,"user_tz":180,"elapsed":7899,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"08bf6795-0d5d-4223-a6fa-1fbac1950435"},"source":["model.fit(dataset, epochs=3)"],"execution_count":24,"outputs":[{"output_type":"stream","text":["Epoch 1/3\n","938/938 [==============================] - 3s 2ms/step - loss: 0.3954 - sparse_categorical_accuracy: 0.8865\n","Epoch 2/3\n","938/938 [==============================] - 2s 2ms/step - loss: 0.0965 - sparse_categorical_accuracy: 0.9707\n","Epoch 3/3\n","938/938 [==============================] - 2s 2ms/step - loss: 0.0627 - sparse_categorical_accuracy: 0.9810\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<tensorflow.python.keras.callbacks.History at 0x7fed0009da58>"]},"metadata":{"tags":[]},"execution_count":24}]},{"cell_type":"markdown","metadata":{"id":"bZc2ss8qln1p"},"source":["Done!\n","\n","**Note:** When you use `fit`, by default execution uses static graphs, so you don't need to add any `tf.function` decorators to your model or your layers.\n","\n","Now let's test it:"]},{"cell_type":"code","metadata":{"id":"d0-XQbxOlqB3","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1613058787158,"user_tz":180,"elapsed":749,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"356a60e4-22ac-4341-f189-1570e5c3f44b"},"source":["x_test = x_test[:].reshape(10000, 784).astype('float32') / 255\n","test_dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test))\n","test_dataset = test_dataset.batch(128)\n","\n","loss, acc = model.evaluate(test_dataset)\n","print('loss: %.3f - acc: %.3f' % (loss, acc))"],"execution_count":25,"outputs":[{"output_type":"stream","text":["79/79 [==============================] - 0s 2ms/step - loss: 0.0955 - sparse_categorical_accuracy: 0.9720\n","loss: 0.096 - acc: 0.972\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"DkmbEn55nV7y"},"source":["Note that you can also monitor your loss and metrics on some validation data during `fit`.\n","\n","Also, you can call `fit` directly on Numpy arrays, so no need for the dataset conversion:"]},{"cell_type":"code","metadata":{"id":"7q8UuNivngVe","colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"status":"ok","timestamp":1613058798289,"user_tz":180,"elapsed":7124,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"183ef2ad-9d29-48d9-878c-9b0fda192fc9"},"source":["(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()\n","x_train = x_train.reshape(60000, 784).astype('float32') / 255\n","\n","num_val_samples = 10000\n","x_val = x_train[-num_val_samples:]\n","y_val = y_train[-num_val_samples:]\n","x_train = x_train[:-num_val_samples]\n","y_train = y_train[:-num_val_samples]\n","\n","# Instantiate a simple classification model\n","model = tf.keras.Sequential([\n","  layers.Dense(256, activation=tf.nn.relu),\n","  layers.Dense(256, activation=tf.nn.relu),\n","  layers.Dense(10)\n","])\n","\n","# Instantiate a logistic loss function that expects integer targets.\n","loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n","\n","# Instantiate an accuracy metric.\n","accuracy = tf.keras.metrics.SparseCategoricalAccuracy()\n","\n","# Instantiate an optimizer.\n","optimizer = tf.keras.optimizers.Adam()\n","\n","model.compile(optimizer=optimizer,\n","              loss=loss,\n","              metrics=[accuracy])\n","model.fit(x_train, y_train,\n","          validation_data=(x_val, y_val),\n","          epochs=3,\n","          batch_size=64)"],"execution_count":26,"outputs":[{"output_type":"stream","text":["Epoch 1/3\n","782/782 [==============================] - 2s 3ms/step - loss: 0.4463 - sparse_categorical_accuracy: 0.8704 - val_loss: 0.1176 - val_sparse_categorical_accuracy: 0.9662\n","Epoch 2/3\n","782/782 [==============================] - 2s 2ms/step - loss: 0.0970 - sparse_categorical_accuracy: 0.9708 - val_loss: 0.1050 - val_sparse_categorical_accuracy: 0.9682\n","Epoch 3/3\n","782/782 [==============================] - 2s 2ms/step - loss: 0.0590 - sparse_categorical_accuracy: 0.9818 - val_loss: 0.0811 - val_sparse_categorical_accuracy: 0.9750\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<tensorflow.python.keras.callbacks.History at 0x7fecaa3127b8>"]},"metadata":{"tags":[]},"execution_count":26}]},{"cell_type":"markdown","metadata":{"id":"88ExjKfCo7aP"},"source":["## Callbacks\n","\n","One of the neat features of `fit` (besides built-in support for sample weighting and class weighting) is that you can easily customize what happens during training and evaluation by using [callbacks](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/).\n","\n","A callback is an object that is called at different points during training (e.g. at the end of every batch or at the end of every epoch) and takes actions, such as saving a model, mutating variables on the model, loading a checkpoint, stopping training, etc.\n","\n","There's a bunch of built-in callback available, like `ModelCheckpoint` to save your models after each epoch during training, or `EarlyStopping`, which interrupts training when your validation metrics start stalling.\n","\n","And you can easily [write your own callbacks](https://www.tensorflow.org/guide/keras/custom_callback)."]},{"cell_type":"code","metadata":{"id":"oAylVdYJqcZ3","colab":{"base_uri":"https://localhost:8080/","height":181},"executionInfo":{"status":"ok","timestamp":1597001599989,"user_tz":180,"elapsed":16350,"user":{"displayName":"Ivanovitch Silva","photoUrl":"https://lh3.googleusercontent.com/a-/AOh14Git9r91cROvzBPiAlvwQtPMEFxLz44uDidMPM-PrQ=s64","userId":"06428777505436195303"}},"outputId":"8a703239-4a73-41e3-f9b3-7ae3c3ee91db"},"source":["# Instantiate a simple classification model\n","model = tf.keras.Sequential([\n","  layers.Dense(256, activation=tf.nn.relu),\n","  layers.Dense(256, activation=tf.nn.relu),\n","  layers.Dense(10)\n","])\n","\n","# Instantiate a logistic loss function that expects integer targets.\n","loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n","\n","# Instantiate an accuracy metric.\n","accuracy = tf.keras.metrics.SparseCategoricalAccuracy()\n","\n","# Instantiate an optimizer.\n","optimizer = tf.keras.optimizers.Adam()\n","\n","model.compile(optimizer=optimizer,\n","              loss=loss,\n","              metrics=[accuracy])\n","\n","# Instantiate some callbacks\n","callbacks = [tf.keras.callbacks.EarlyStopping(),\n","             tf.keras.callbacks.ModelCheckpoint(filepath='my_model.keras',\n","                                                save_best_only=True)]\n","\n","model.fit(x_train, y_train,\n","          validation_data=(x_val, y_val),\n","          epochs=30,\n","          batch_size=64,\n","          callbacks=callbacks)"],"execution_count":null,"outputs":[{"output_type":"stream","text":["Epoch 1/30\n","782/782 [==============================] - 4s 5ms/step - loss: 0.2485 - sparse_categorical_accuracy: 0.9263 - val_loss: 0.1376 - val_sparse_categorical_accuracy: 0.9586\n","Epoch 2/30\n","782/782 [==============================] - 4s 5ms/step - loss: 0.0935 - sparse_categorical_accuracy: 0.9722 - val_loss: 0.1372 - val_sparse_categorical_accuracy: 0.9579\n","Epoch 3/30\n","782/782 [==============================] - 4s 5ms/step - loss: 0.0627 - sparse_categorical_accuracy: 0.9801 - val_loss: 0.0861 - val_sparse_categorical_accuracy: 0.9747\n","Epoch 4/30\n","782/782 [==============================] - 4s 5ms/step - loss: 0.0432 - sparse_categorical_accuracy: 0.9860 - val_loss: 0.0922 - val_sparse_categorical_accuracy: 0.9748\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["<tensorflow.python.keras.callbacks.History at 0x7f5377a12828>"]},"metadata":{"tags":[]},"execution_count":44}]},{"cell_type":"markdown","metadata":{"id":"fxINLLGitX_n"},"source":["# Parting words\n","\n","I hope this guide has given you a good overview of what's possible with TensorFlow 2.x and Keras!\n","\n","Remember that TensorFlow and Keras don't represent a single workflow. It's a spectrum of workflows, each with its own trade-off between usability and flexibility. For instance, you've noticed that it's much easier to use `fit` than to write a custom training loop, but `fit` doesn't give you the same level of granular control for research use cases.\n","\n","So use the right tool for the job!\n","\n","A core principle of Keras is \"progressive disclosure of complexity\": it's easy to get started, and you can gradually dive into workflows where you write more and more logic from scratch, providing you with complete control.\n","\n","This applies to both model definition, and model training.\n","\n","![Model definition: spectrum of workflows](https://keras-dev.s3.amazonaws.com/tutorials-img/model-building-spectrum.png)\n","\n","![Model training: spectrum of workflows](https://keras-dev.s3.amazonaws.com/tutorials-img/model-training-spectrum.png)"]},{"cell_type":"markdown","metadata":{"id":"MfO_uy61upRm"},"source":["\n","## What to learn next (<font color='red'>suggestions</font>)\n","\n","Next, there are many more topics you may be interested in:\n","\n","- [Saving and serialization](https://www.tensorflow.org/guide/keras/save_and_serialize)\n","- [Distributed training on multiple GPUS](https://www.tensorflow.org/guide/distributed_training)\n","- [Exporting models to TFLite for deployment on Android or embedded systems](https://www.tensorflow.org/lite/convert/python_api#converting_a_keras_model_)\n","- [Exporting models to TensorFlow.js for deployment in the browser](https://www.tensorflow.org/js/tutorials/conversion/import_keras)"]}]}