sentiment-analysis-network/Sentiment_Classification_Projects.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment Classification & How To \"Frame Problems\" for a Neural Network\n",
    "\n",
    "by Andrew Trask\n",
    "\n",
    "- **Twitter**: @iamtrask\n",
    "- **Blog**: http://iamtrask.github.io"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What You Should Already Know\n",
    "\n",
    "- neural networks, forward and back-propagation\n",
    "- stochastic gradient descent\n",
    "- mean squared error\n",
    "- and train/test splits\n",
    "\n",
    "### Where to Get Help if You Need it\n",
    "- Re-watch previous Udacity Lectures\n",
    "- Leverage the recommended Course Reading Material - [Grokking Deep Learning](https://www.manning.com/books/grokking-deep-learning) (Check inside your classroom for a discount code)\n",
    "- Shoot me a tweet @iamtrask\n",
    "\n",
    "\n",
    "### Tutorial Outline:\n",
    "\n",
    "- Intro: The Importance of \"Framing a Problem\" (this lesson)\n",
    "\n",
    "- [Curate a Dataset](#lesson_1)\n",
    "- [Developing a \"Predictive Theory\"](#lesson_2)\n",
    "- [**PROJECT 1**: Quick Theory Validation](#project_1)\n",
    "\n",
    "\n",
    "- [Transforming Text to Numbers](#lesson_3)\n",
    "- [**PROJECT 2**: Creating the Input/Output Data](#project_2)\n",
    "\n",
    "\n",
    "- Putting it all together in a Neural Network (video only - nothing in notebook)\n",
    "- [**PROJECT 3**: Building our Neural Network](#project_3)\n",
    "\n",
    "\n",
    "- [Understanding Neural Noise](#lesson_4)\n",
    "- [**PROJECT 4**: Making Learning Faster by Reducing Noise](#project_4)\n",
    "\n",
    "\n",
    "- [Analyzing Inefficiencies in our Network](#lesson_5)\n",
    "- [**PROJECT 5**: Making our Network Train and Run Faster](#project_5)\n",
    "\n",
    "\n",
    "- [Further Noise Reduction](#lesson_6)\n",
    "- [**PROJECT 6**: Reducing Noise by Strategically Reducing the Vocabulary](#project_6)\n",
    "\n",
    "\n",
    "- [Analysis: What's going on in the weights?](#lesson_7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "56bb3cba-260c-4ebe-9ed6-b995b4c72aa3"
    }
   },
   "source": [
    "# Lesson: Curate a Dataset<a id='lesson_1'></a>\n",
    "The cells from here until Project 1 include code Andrew shows in the videos leading up to mini project 1. We've included them so you can run the code along with the videos without having to type in everything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "eba2b193-0419-431e-8db9-60f34dd3fe83"
    }
   },
   "outputs": [],
   "source": [
    "def pretty_print_review_and_label(i):\n",
    "    print(labels[i] + \"\\t:\\t\" + reviews[i][:80] + \"...\")\n",
    "\n",
    "g = open('reviews.txt','r') # What we know!\n",
    "reviews = list(map(lambda x:x[:-1],g.readlines()))\n",
    "g.close()\n",
    "\n",
    "g = open('labels.txt','r') # What we WANT to know!\n",
    "labels = list(map(lambda x:x[:-1].upper(),g.readlines()))\n",
    "g.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** The data in `reviews.txt` we're using has already been preprocessed a bit and contains only lower case characters. If we were working from raw data, where we didn't know it was all lower case, we would want to add a step here to convert it. That's so we treat different variations of the same word, like `The`, `the`, and `THE`, all the same way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "len(reviews)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "bb95574b-21a0-4213-ae50-34363cf4f87f"
    }
   },
   "outputs": [],
   "source": [
    "reviews[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "e0408810-c424-4ed4-afb9-1735e9ddbd0a"
    }
   },
   "outputs": [],
   "source": [
    "labels[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson: Develop a Predictive Theory<a id='lesson_2'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "e67a709f-234f-4493-bae6-4fb192141ee0"
    }
   },
   "outputs": [],
   "source": [
    "print(\"labels.txt \\t : \\t reviews.txt\\n\")\n",
    "pretty_print_review_and_label(2137)\n",
    "pretty_print_review_and_label(12816)\n",
    "pretty_print_review_and_label(6267)\n",
    "pretty_print_review_and_label(21934)\n",
    "pretty_print_review_and_label(5297)\n",
    "pretty_print_review_and_label(4998)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project 1: Quick Theory Validation<a id='project_1'></a>\n",
    "\n",
    "There are multiple ways to implement these projects, but in order to get your code closer to what Andrew shows in his solutions, we've provided some hints and starter code throughout this notebook.\n",
    "\n",
    "You'll find the [Counter](https://docs.python.org/2/library/collections.html#collections.Counter) class to be useful in this exercise, as well as the [numpy](https://docs.scipy.org/doc/numpy/reference/) library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll create three `Counter` objects, one for words from postive reviews, one for words from negative reviews, and one for all the words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create three Counter objects to store positive, negative and total counts\n",
    "positive_counts = Counter()\n",
    "negative_counts = Counter()\n",
    "total_counts = Counter()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO:** Examine all the reviews. For each word in a positive review, increase the count for that word in both your positive counter and the total words counter; likewise, for each word in a negative review, increase the count for that word in both your negative counter and the total words counter.\n",
    "\n",
    "**Note:** Throughout these projects, you should use `split(' ')` to divide a piece of text (such as a review) into individual words. If you use `split()` instead, you'll get slightly different results than what the videos and solutions show."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO: Loop over all the words in all the reviews and increment the counts in the appropriate counter objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following two cells to list the words used in positive reviews and negative reviews, respectively, ordered from most to least commonly used. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Examine the counts of the most common words in positive reviews\n",
    "positive_counts.most_common()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Examine the counts of the most common words in negative reviews\n",
    "negative_counts.most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, common words like \"the\" appear very often in both positive and negative reviews. Instead of finding the most common words in positive or negative reviews, what you really want are the words found in positive reviews more often than in negative reviews, and vice versa. To accomplish this, you'll need to calculate the **ratios** of word usage between positive and negative reviews.\n",
    "\n",
    "**TODO:** Check all the words you've seen and calculate the ratio of postive to negative uses and store that ratio in `pos_neg_ratios`. \n",
    ">Hint: the positive-to-negative ratio for a given word can be calculated with `positive_counts[word] / float(negative_counts[word]+1)`. Notice the `+1` in the denominator – that ensures we don't divide by zero for words that are only seen in positive reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create Counter object to store positive/negative ratios\n",
    "pos_neg_ratios = Counter()\n",
    "\n",
    "# TODO: Calculate the ratios of positive and negative uses of the most common words\n",
    "#       Consider words to be \"common\" if they've been used at least 100 times"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examine the ratios you've calculated for a few words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"Pos-to-neg ratio for 'the' = {}\".format(pos_neg_ratios[\"the\"]))\n",
    "print(\"Pos-to-neg ratio for 'amazing' = {}\".format(pos_neg_ratios[\"amazing\"]))\n",
    "print(\"Pos-to-neg ratio for 'terrible' = {}\".format(pos_neg_ratios[\"terrible\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking closely at the values you just calculated, we see the following:\n",
    "\n",
    "* Words that you would expect to see more often in positive reviews – like \"amazing\" – have a ratio greater than 1. The more skewed a word is toward postive, the farther from 1 its positive-to-negative ratio  will be.\n",
    "* Words that you would expect to see more often in negative reviews – like \"terrible\" – have positive values that are less than 1. The more skewed a word is toward negative, the closer to zero its positive-to-negative ratio will be.\n",
    "* Neutral words, which don't really convey any sentiment because you would expect to see them in all sorts of reviews – like \"the\" – have values very close to 1. A perfectly neutral word – one that was used in exactly the same number of positive reviews as negative reviews – would be almost exactly 1. The `+1` we suggested you add to the denominator slightly biases words toward negative, but it won't matter because it will be a tiny bias and later we'll be ignoring words that are too close to neutral anyway.\n",
    "\n",
    "Ok, the ratios tell us which words are used more often in postive or negative reviews, but the specific values we've calculated are a bit difficult to work with. A very positive word like \"amazing\" has a value above 4, whereas a very negative word like \"terrible\" has a value around 0.18. Those values aren't easy to compare for a couple of reasons:\n",
    "\n",
    "* Right now, 1 is considered neutral, but the absolute value of the positive-to-negative ratios of very positive words is larger than the absolute value of the ratios for the very negative words. So there is no way to directly compare two numbers and see if one word conveys the same magnitude of positive sentiment as another word conveys negative sentiment. So we should center all the values around netural so the absolute value fro neutral of the postive-to-negative ratio for a word would indicate how much sentiment (positive or negative) that word conveys.\n",
    "* When comparing absolute values it's easier to do that around zero than one. \n",
    "\n",
    "To fix these issues, we'll convert all of our ratios to new values using logarithms.\n",
    "\n",
    "**TODO:** Go through all the ratios you calculated and convert them to logarithms. (i.e. use `np.log(ratio)`)\n",
    "\n",
    "In the end, extremely positive and extremely negative words will have positive-to-negative ratios with similar magnitudes but opposite signs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO: Convert ratios to logs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examine the new ratios you've calculated for the same words from before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"Pos-to-neg ratio for 'the' = {}\".format(pos_neg_ratios[\"the\"]))\n",
    "print(\"Pos-to-neg ratio for 'amazing' = {}\".format(pos_neg_ratios[\"amazing\"]))\n",
    "print(\"Pos-to-neg ratio for 'terrible' = {}\".format(pos_neg_ratios[\"terrible\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If everything worked, now you should see neutral words with values close to zero. In this case, \"the\" is near zero but slightly positive, so it was probably used in more positive reviews than negative reviews. But look at \"amazing\"'s ratio - it's above `1`, showing it is clearly a word with positive sentiment. And \"terrible\" has a similar score, but in the opposite direction, so it's below `-1`. It's now clear that both of these words are associated with specific, opposing sentiments.\n",
    "\n",
    "Now run the following cells to see more ratios. \n",
    "\n",
    "The first cell displays all the words, ordered by how associated they are with postive reviews. (Your notebook will most likely truncate the output so you won't actually see *all* the words in the list.)\n",
    "\n",
    "The second cell displays the 30 words most associated with negative reviews by reversing the order of the first list and then looking at the first 30 words. (If you want the second cell to display all the words, ordered by how associated they are with negative reviews, you could just write `reversed(pos_neg_ratios.most_common())`.)\n",
    "\n",
    "You should continue to see values similar to the earlier ones we checked – neutral words will be close to `0`, words will get more positive as their ratios approach and go above `1`, and words will get more negative as their ratios approach and go below `-1`. That's why we decided to use the logs instead of the raw ratios."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# words most frequently seen in a review with a \"POSITIVE\" label\n",
    "pos_neg_ratios.most_common()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# words most frequently seen in a review with a \"NEGATIVE\" label\n",
    "list(reversed(pos_neg_ratios.most_common()))[0:30]\n",
    "\n",
    "# Note: Above is the code Andrew uses in his solution video, \n",
    "#       so we've included it here to avoid confusion.\n",
    "#       If you explore the documentation for the Counter class, \n",
    "#       you will see you could also find the 30 least common\n",
    "#       words like this: pos_neg_ratios.most_common()[:-31:-1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End of Project 1. \n",
    "## Watch the next video to see Andrew's solution, then continue on to the next lesson.\n",
    "\n",
    "# Transforming Text into Numbers<a id='lesson_3'></a>\n",
    "The cells here include code Andrew shows in the next video. We've included it so you can run the code along with the video without having to type in everything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "\n",
    "review = \"This was a horrible, terrible movie.\"\n",
    "\n",
    "Image(filename='sentiment_network.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "review = \"The movie was excellent\"\n",
    "\n",
    "Image(filename='sentiment_network_pos.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project 2: Creating the Input/Output Data<a id='project_2'></a>\n",
    "\n",
    "**TODO:** Create a [set](https://docs.python.org/3/tutorial/datastructures.html#sets) named `vocab` that contains every word in the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO: Create set named \"vocab\" containing all of the words from all of the reviews\n",
    "vocab = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to check your vocabulary size. If everything worked correctly, it should print **74074**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vocab_size = len(vocab)\n",
    "print(vocab_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the following image. It represents the layers of the neural network you'll be building throughout this notebook. `layer_0` is the input layer, `layer_1` is a hidden layer, and `layer_2` is the output layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhYAAAFOCAYAAADaclTUAAAABGdBTUEAALGPC/xhBQAAACBjSFJN\nAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAB1WlUWHRYTUw6Y29tLmFkb2Jl\nLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1Q\nIENvcmUgNS40LjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5\nOTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91\ndD0iIgogICAgICAgICAgICB4bWxuczp0aWZmPSJodHRwOi8vbnMuYWRvYmUuY29tL3RpZmYvMS4w\nLyI+CiAgICAgICAgIDx0aWZmOkNvbXByZXNzaW9uPjE8L3RpZmY6Q29tcHJlc3Npb24+CiAgICAg\nICAgIDx0aWZmOk9yaWVudGF0aW9uPjE8L3RpZmY6T3JpZW50YXRpb24+CiAgICAgICAgIDx0aWZm\nOlBob3RvbWV0cmljSW50ZXJwcmV0YXRpb24+MjwvdGlmZjpQaG90b21ldHJpY0ludGVycHJldGF0\naW9uPgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4K\nAtiABQAAQABJREFUeAHtnXnQVsWV/5tJqlJJJmocmZqaqIWEYSKoGI0RNJqouDDEAYNBGaPGII46\noENcmNEQRv1JiUuIS8QN1zAoikoR44aYkbglSkCRxJRjCDjzx5Axku2PqUq9v/vp5Dzpt997n+fu\n23NO1fPerZfT377v7W+fPt09bCAQo6IIKAKKgCKgCCgCikAOCPxZDmloEoqAIqAIKAKKgCKgCFgE\nlFjoi6AIKAKKgCKgCCgCuSGgxCI3KDUhRUARUAQUAUVAEVBioe+AIqAIKAKKgCKgCOSGgBKL3KDU\nhBQBRUARUAQUAUVAiYW+A4qAIqAIKAKKgCKQGwJKLHKDUhNSBBQBRUARUAQUASUW+g4oAoqAIqAI\nKAKKQG4IKLHIDUpNSBFQBBQBRUARUASUWOg7oAgoAoqAIqAIKAK5IaDEIjcoNSFFQBFQBBQBRUAR\nUGKh74AioAgoAoqAIqAI5IaAEovcoNSEFIHeCGzevNkMGzbM3H777b0D91GISy+91Bx77LF9VGIt\nqiLQXgSUWLS3brVkikAjEHjnnXfMfffd1whdVUlFQBHojYASi94YaQhFQBEoCIFrrrnGTJkyxWzb\ntq2gHDRZRUARKBuB95edoeanCCgC/Y0Aw0Fjx47tbxC09IpAixFQYtHiytWiKQJ1RGDMmDFmYGBg\nkGrqXzEIDr1QBBqNgA6FNLr6VPk2IYBD54wZM6xzJw6e/A488MAhjp6/+tWv7P1ejTFpEZ/wrqxY\nsWJIPjhPYknw5cknn7R6cCQe6anzqY+SXisCioCLgBILFw09VwQqQoCG/ayzzjIjR460vXl69PwW\nLlxobrnlFksERLWddtrJnHzyyeapp54KJQOEwyHy/vvvt+EIL3LuueeaCy+80Bx55JGdfPBv+OUv\nf2mHJyAQYbJq1SqzaNEiM2/ePBtv1qxZYcH0niKgCCgCRodC9CVQBCpGABIAgbj66qvNRRddNEgb\nrBI7duwwJ510kjn//PPN+PHj7fPJkyebiy++2Dz22GOGoQVfHn/88U44eYZF5Dvf+Y554YUXzO67\n7y637fnNN99sPvrRjxoIw6ZNm4xLRggYFq+TgJ4oAoqAIuAgoBYLBww9VQSqQICGHjnssMNCs99z\nzz3tfQiGCGTimGOOsVYJuecesXLw3CUdV1xxhZkzZ84gUuHGOeecc+zsjAceeMC9bc9PPfXUyHhD\nAusNRUAR6GsElFj0dfVr4euAwPTp0+3wglgjXJ2wZlx//fXurc75zJkzzfr1640/fPHSSy/Z+zwX\nwT+CIY8o8kI4rBgHHHCAWbt2rUTrHMeNG9c51xNFQBFQBLohoMSiGzr6TBEoEQGGKljXQRw3OR5y\nyCHW7yJMDQjJHnvsYfB/cGX16tX2Ps9FxNoxYcKEQem7eXEOUXn33XclWue48847d871RBFQBBSB\nbggoseiGjj5TBEpAAKsEsy0YvkAgAeK8uXXrVnPKKadEasEQBf4P7swPVrHkfpi88cYbnbQlD//o\nW0DC0tF7ioAioAhEIaDEIgoZva8IlIQADpvbt283zz77rHXe9B0nu6lx/PHH2yGOJ554wgaDFDDk\nwX1XxOKAY6aKIqAIKAJFIqDEokh0NW1FIAYCzz//vDn00EOHzMSQqN3IAH4Z+EU88sgjNjjDIlz7\n/hoMqSAbN260x7A/WE4YDmFKqooioAgoAmkRUGKRFjmNpwjkhMDw4cPNT3/609DUGOJg/YhucvbZ\nZ9vZISxwxbAI177gmMmsD6a14twZJkuWLLG3Z8+eHfZY7ykCioAiEAsBJRaxYNJAikBxCLDoFE6T\nvqUAZ84jjjjCHHfccTZzccD0NZk0aZK9JX4VrHkRJldddZWdgopTJ2mLYKlglU5Ix2233TZoiqqE\n0aMioAgoAnERUGIRFykNpwgUhAANPT4SrH7pztL40Y9+ZBv7K6+80hICCEPYMt5ijYCcfP7zn48c\nUsF3Ax+M+fPnm4ceeqiTFzNLENav0BU1C6pkTVYR6CMEhgUe4YN3A+qjwmtRFYG2IIAFgiXBIShh\n5KMt5dRyKAKKQP0RUItF/etINVQEeiKwZcsW67SppKInVBpAEVAECkZAiUXBAGvyikDRCODgydoV\nYU6bReVNnu6wTZxzd62NovTSdBUBRaB6BJRYJKgDnOtk34YE0VoXFLO7bJ8tDQorRmrDUU1V/8u/\n/IthZkmZ/hH4a/gLa/W6TrI+RzVIaq6KgCKQBwJKLPJAsY/SYPYAm1nRO5aGhNUc16xZY2cwMMNA\npRwEILkQu//8z/8csqx3Wg3efPNNs//++6eNrvEUAUVAEdBt0/UdiI8Alor777/fzh5w96FgB80H\nH3zQEgs2vtIloeNjmiUky33nLVidRo0alXeymp4ioAj0EQJqseijys5aVNmK2yUVkiZmbqwYTz31\nVOQCTBJWj4qAIqAIKALtRUCJRXvrNteSyVbcJ554YmS6slDT66+/HhlGHygCioAioAi0GwElFjnV\nL8ME+B+IMyNHHBy57wrXPOs2XEAjThg/LmZqf1ttphf64SQ/nvEjHo6npJnW+VTIwr777ivJDzmy\nUBOLLa1du3bIM73RDATUAbcZ9aRaKgJ1RkCJRQ61c+mll9rFiUaOHNlxaMSxkSWSGT6AcIiweiKN\nL5tFRcnq1attGHdpZvaB2GeffezSz+xeKY6TWBD8PPx0v/jFL9pbLAmddlz+vffes2n08uzfe++9\nI/e98PXS6/ohgCPu4YcfXj/FVCNFQBFoDAJKLDJWFbMgIBBXX321YellV7AWsA8EDo+y8RMNM8su\ns1lUVO+QNQncpZkJxz4Q/JYvX26wDIgwxRCSwg6ZWDN8+fGPf2w+9alPmZtvvjlyqWc/Tti1EAs3\n77Bwu+66q90CPOyZ3lMEFAFFQBFoPwJKLDLW8QsvvGBTOOyww0JTkqEHdwOpKVOmGKwO7M3gy4oV\nK+wzwogQbvv27ZakyD33SGMP6bjxxhvd2/acfNjVMquwj0Uc+ehHPxonmIZRBBQBRUARaCkCSiwy\nViwzJBiWGD9+/JCUsGZcf/31Q+5jyTjggAPsRlD+w0ceeWTI0swMdRx66KFdLQ7jxo2zhEQsI5Iu\n+fSyMkjYbse4hCEuAemWlz5TBBQBRUARaC4CSixyqjscKH3HykMOOcTgdxEmMjUT3wkRiAjDJv7S\nzFgruI/zZdRP/DFcywjp7rbbbpJ8puMuu+xi40cN30ji7777rl0FUq712CwEXnvtNevf0yytVVtF\nQBGoEwJKLDLWBmSA2R9YFRAadnGsxFHylFNOCc1BpmY+9thjneePP/64PZdnnQfBCcMZkm63I9aQ\nIkSIBeXtJvh0jB49ulsQfVZjBPClycPCVeMiqmqKgCJQMAJKLDICfNFFF1n/h2effdZw3mvWhGTH\nx/vkk0+2lgi599BDD9l7/oedfSBefvllCVbJUaaZdptVAunApyPKSlOJ4pqpIqAIKAKKQKkIKLHI\nCDezMbr5P2zatCkyhxNOOMFOH8UvgiERVq3kni/HHXecDdfNWsCUV4ZJ3KEVP50s1/iQ4K/xve99\nLzIZsbgcf/zxkWH0gSKgCCgCikC7EVBikbF+sSb89Kc/DU0Ff4RFixaFPuMmjp+sacG6FQyJcB62\nXLYMp2ARCRMIB1NUGS5h346iBN+PJUuWdKbOuvlQVoaDjjnmmFBHVjesntcXAXxkVBQBRUARyIKA\nEoss6AVxWadi/fr1dmVLNymcOY844giDtQHxnSolLNNEIQU4Z86ZM0duDzpCFm677TYbhsW2XMsF\n+eAkCsG56qqrBsXL+4I1Mxi+gfwwLVYEK4kswrV06VK5rccGIsCQ3kEHHdRAzVVlRUARqAsCSiwy\n1gSN7BNPPGGYZunO2PjRj35kF85i0Sx68czaCHOsxBqBXwLkZPLkyZHa0KizKiL+C1g2JC+2MIeQ\n0CDE9e+IzCTGAxbomj9/vrXEiA5jx441EydOtDr4/iExktQgioAioAgoAi1CYFgww2CgReVpZFFY\nRIulsLvtH9LIgqnSjUMAsqifhMZVmyqsCNQKAbVYVFwdMpOi266hFauo2SsCioAioAgoArERUGIR\nG6piAjK0wGwLhjrKEn8hLxnSiDqG7UFSlq6aT3kIvPnmm4YVXFUUAUVAEciCgBKLLOhljIvTI/t7\n+CttZky2Z3Rml3RbZMt/FjUbpWdGGqBRCDCzZ9SoUY3SWZVVBBSB+iGgxKKCOmEmB9YBnB5xhCzT\nWlFBcTVLRUARUAQUgT5C4P19VNbaFBUioWSiNtWhiigCioAioAjkiIBaLHIEU5NSBJqMQK8N5ppc\nNtVdEVAEykNAiUV5WGtOikCtEWCdlMMPP7zWOqpyioAiUH8ElFjUv45UQ0VAEVAEFAFFoDEIKLFo\nTFWpooqAIqAIKAKKQP0RUGJR/zrqquGvf/1r88gjj3QNow8VAUVAEVAEFIGyEFBiURbSBeXzkY98\nxMyePdv893//d0E5aLL9gsBrr71m96Hpl/JqORUBRaAYBJRYFINrqamy46nuKloq5K3M7L333jO6\niVwrq1YLpQiUioASi1LhLiYz1sS45ZZbiklcU1UEFAFFQBFQBBIgoMQiAVh1Dfq3f/u3Zvjw4eaZ\nZ56pq4qqlyKgCCgCikCfIKDEoiUVPXfuXMOGZiqKQFoE3n333bRRNZ4ioAgoAh0EhgUbTg10rvSk\nsQjgvPmxj33MsHoiDp0qikBSBNi/Rj8HSVHT8IqAIuAjoBYLH5GGXv/1X/+1mTlzpnn44YcbWgJV\nWxFQBBQBRaANCCixaEMt/rEMzA5ZvHhxi0qkRVEEFAFFQBFoGgJKLJpWY130Peqoo8z27dvNm2++\n2SWUPlIE/oQAQ2g33HBDZx0UXXDtT9jomSKgCKRDQH0s0uFW21hXXHGF2bFjh7n22mtrq6MqVi8E\n8M1xF1ibNm2aeeihh+qlpGqjCCgCjUFALRaNqap4ik6fPl1nh8SDSkP9EQGG0Fw55ZRT3Es9VwQU\nAUUgEQJKLBLBVf/ArGnBT/cPqX9d1UXDSZMmdVQZN26cOeGEEzrXeqIIKAKKQFIElFgkRawB4efM\nmWOWLVvWAE1VxToggG+OyFe+8hU51aMioAgoAqkQUB+LVLDVOxIOeDvttJP5r//6L8M0VBVFoBcC\nZ555pt1vRtdB6YWUPlcEFIFeCLzv3wLpFUifNwuBD3zgA+Y3v/mNJRYHH3xws5RXbStB4Pe//73d\ngOz444+vJH/NVBFQBNqDgFos2lOXg0rCviEXXHCB2bBhw6D7eqEIhCEgs0LUwhWGjt5TBBSBJAio\nj0UStBoUVsbNf/jDHzZIa1W1KgQgFEoqqkJf81UE2oWAEot21eeg0uCI98ADDwy6pxeKgCKgCCgC\nikCRCOhQSJHoVpy2bkxWcQVo9oqAIqAI9CECSixaXuknnniiYcEjXZug5RUds3gMjW3evNm8/vrr\nZsuWLeatt94yGzduHBKb1Td32WUXs99++5mxY8eaT3/607pr7hCU9IYioAiEIaDEIgyVFt1joawb\nb7zRrF27tkWl0qIkQQBH3scff9yuyDp8+HAzceJEs++++5oxY8bYacksqOYKli6mKr/zzjtm06ZN\n5tlnn7U/yAazRr7whS8oyXAB03NFQBEYhIASi0FwtO+CNS0+8YlPGHqq6pzXvvqNKhH1ftddd5k7\n77zTBpk7d645+uijU78DpLdmzRrz2GOP2fUumHH01a9+NXV6UXrrfUVAEWg+Auq82fw67FqCj3zk\nI4a9IJYuXdo1nD5sDwLsVgqZfO6558ztt99upxyffvrpmUgA7xHDaXfccYe1ZoAWm5ddeOGFgzYw\naw+KWhJFQBFIi4ASi7TINSjerFmzzMqVKxuksaqaBgGsUvvvv7959NFH7Y8dSg866KA0SXWNg+WL\n3XMZLkHI45577ukaRx8qAopA/yCgxKIP6lrG0BlrV2knAlgppk6dahjywJ+mCELhIycEAyKzePFi\ng6MwQyYqioAi0N8IKLHok/qnwVm+fHmflLZ/iklDzj4fNO4QCoY8yhZIzLp168yIESPsEMybb75Z\ntgqanyKgCNQIAXXerFFlFKmKrmlRJLrVpA2pmDJlitl1112toyZ+EFULQyKXXHKJJTliKataJ81f\nEVAEykVALRbl4l1ZbpitZ86caR5++OHKdNCM80NASMUBBxxg8KWoA6mgdFhMbrrpJnPkkUcatVzk\nV9+akiLQJASUWDSptjLqyuwQdbLLCGINorukAifKugmzR5Rc1K1WVB9FoDwEdCikPKxrkRNTBBmL\nVzN1LaojlRJM8Vy/fr2tx1QJlBQJEotTJ/4XdbGolFR0zUYR6GsE1GLRZ9V/9tln27UN+qzYrSku\njTVOuKtWrap9mRgW+dSnPmXOOOOM2uuqCioCikB+CKjFIj8sG5ES496Mf8saBDh16oqcjag6uxAV\nMzCYAVLGdNI8UGHY5rDDDrPTYKuYsZJHGTQNRUARSIaAWiyS4dX40AyB8GMPkSuuuMI6/jW+UH1S\ngPPOO89gcWoKqaBaGAJh9U9mikAy+lXY+G3YsGF9by381a9+Za655hqz5557WjzAhB/3eKbSDgSU\nWLSjHmOVAusEqzPus88+du2Dr3/967HiaaDqEaDeXnzxRfPP//zP1SuTUAOIEI7Dl112WcKYGrxN\nCLCpHd+e+++/367cOjAwYPg98cQTdh8ankHAVJqPwPubXwQtQVwE6DGyOiMEQ6VZCCxatMgsXLiw\nsU6QbFiG47BuXNas9y5PbZnujuAftPvuu3eSPvbYY82ECRPMEUccYU499VTz6quvdp7pSTMRUItF\nM+stldYMgej26amgqzSSWCua7KOAHw87on7jG9+oFEvNvBoEXnrpJfPUU09Z4uCSCtFmp512ssN8\nzHYirEqzEVBi0ez6S6w95EIXyUoMW6URbr31VjNv3rxKdcgjc6wVzGjpZ1+LPHBsYhpbt261ao8b\nN66n+q+//nrPMBqg3ggosah3/RSiHQsY3X333YWkrYnmiwCNMFves8FX0wWrBcR2zZo1TS9Krvrj\ne4DzojgyyvHcc88d4nOAIyzPn3zyyUgd6PEThrCuiOOkpM+RYQg/HHFIX/JZsWKFOfDAA0PTdNPv\ndo7/xNVXX20OOeSQbsHss5133rlnGA1QbwSUWNS7fgrTDrM6pmmVeiNAI8zYdFumBPPePfbYY/UG\nvUTtIBU0ttQzTozi0MgRGTt27KChgZNOOsnsscceXdcxWb16tQ1DWBGcImncGWrYtm1bJx8I6y23\n3GKdayWse8QfAv8eLGboNGvWLPdx7PMxY8aYiy66aJBvhR+ZpemROOTDj6vXNUMgeFlU+hiBn/zk\nJ31c+voXPSAVA4F1qf6KxtQwWD+FFjNm6PYEe+ONN2y5b7vttkGFCqbhDgREYWDHjh2D7stFsBfM\nwMknnyyX9njOOed0jUN6hBEhbdIhrzAJiIZNL7AodB4HJMfqS1o8L1rQl/fC1bvoPDX94hBQi0XN\niF7Z6mCaVqkvAo8//rgZP358fRVMqBmWF7z/cUhVMdZKceihhxqcF8Nk9OjR5t133x30iB1tsTo8\n8MADg+5zwbAFzwgjQrjt27dH+ungTMlsjBtvvFGidI7cD3O27ATIeIIlhanIS5YsMcccc4y56qqr\nMqao0euAgBKLOtSC6qAIhCAg04LbRv7YkZU1OVSMnVqJQ2uY4Pvw/PPPD3mEX4Tsaus/ZOE7nhFG\nhKGObuSFcDhVQkj8GRlxnC0ln6THSy+91A71sK5FYJUxDz74YCTBSpq2hq8WASUW1eKvuSsCkQiw\n7Drz+9sm++67r5FZAm0rW5ry0GvHeZOeu+tYydTwvffeOzRJVmBl+qa7oBT+GjTSPHMFawX33bT9\nc/HHCIZN3KimCEdKdIb4sC4L/iJYVCBXUVabQQrpRSMQUGLRiGpSJfsRAXr1I0aMaF3RceTbsmVL\n68qVpkAQChw0f/aznxlmawWj3p0fje3HP/7x0GQnTZpk77uOsAybIfLMXvzxT+C70EnXzcM/dy0d\nbvy8zhmqobyQIpZ5h2BOnz49r+Q1nZogoMSiJhWhaigCYQiwp4JKOxFg2OHiiy+20zBvvvnmRA0s\nfg8MH2CJEGFWBfd8n4jhw4ebl19+WYJVdoRUYBlhqCZwZjVXXnllZbpoxsUioMSiWHw1dUVAEfAQ\n+MQnPmFWrlzp3e2/S1kIit1fo6QbIcDCIStVMryAFYB7vhx33HE2HEMlUYK/A8Mj7tBKVNg098n7\nwgsvtA6azz77rMFqpdJeBJRYtLdutWSKQC0RYMdTFdPxX4jyN8F5E+IQJQwh4KPAuhUMiXAeNqxw\nyimn2CRYRyJMaPTvu+8+w3BJUQ0+sz5wDl28eLH6UoRVQsvuKbFoWYVqcRQBRaAZCEACmGLJAlTu\nSppYDVh1k9kcDG384he/iCwQ00EhBQyJzJkzJzQcZCFYP8OGwUHUtVxAXliQiuGSIqd6vvLKK3YI\npCjiElpwvVkZAkosKoNeM1YE+hOBN998szUriWatQQgF5IHhCpmpAVnYa6+97FTUL3/5y9ZqwTOX\nfEi+WCOwBGDZmDx5stwecmTFTPwaRo4caS0bktcVV1xhCQnDE0XOyvjxj3/cKYfkHXUMK+eQAumN\nWiMwjLW3aq2hKqcI9CEC7BHyzW9+05rLzzvvvFYhwOJY9NJlCedWFa6CwuDgy7RUbZArAF+zDEXg\n/aF39aYi0AUBGgbMtTifMW3wrbfeMhs3bhwUgxUWWYNhl112Mfvtt589P+iggwaF0YvBCEAmgiXW\n7U16j+zt4E4nHBy6uVdshqWSDwIMa2CxmD9/fj4JaiqKQA4IKLHIAcR+SOKZZ56xi9gwV57x2IkT\nJxoWOmL6GI2gvzokq0aywBMfvk2bNtnlhDG3sqEWJlviqxOfMYIT7xA4uuSLBvjtt99u3euFSf7w\nww9vXbmqKBBrXTB9M+3mYEl15p1MumgWi24VOcyStAwavngEdCikeIwbmwM96LvuusuarSETc+fO\nNUcffXTq8XHSYxfHZcuW2SWdcST76le/mjq9pgLrkomPfexjXcvPODQf8zaRMHbUxDcgbGpkU+u0\nCr2xGuKbgbWiLGJRRTk1z+YhoM6bzauzwjWGANxwww2G9Qaee+458+ijj5oNGzYYtrzOsn03jSON\nCWPrsgkVDSvz28mzzYLDImWWcmOZ4NcLTzbs+sEPftAqaFhR9OCDD25VmcosDDM5IJysYKmkokzk\nNa+4CKjFIi5SfRKOho/ez6hRo+zwhWuaLwICeu9f//rXDUMs7B0AeWmLCImgPJC0NFYHvPaRtoyh\ny/sFUVVpPgL8//7d3/2d7Xg0vzRagrwQUGKRF5ItSIdGjLnzVTTw0uB86lOfsovopGmE61AFLpnI\ng5Rh6cCPpS0NMe8Y6zJcf/31dagu1SEjAtQnPhTXXnttxpQ0epsQUGLRptpMWRaGIaZMmWJjf/vb\n3+5pnk+ZTc9o6IEfBw6LrNTnO4T2TKCCAOgsMznIPg8y4Rdj//33N9ddd5056qij/EeNu2boi5Ui\nP/zhDzeifhsHcMkK824yNFPEe19yUTS7HBFQH4scwWxiUkIq8CxftWpVZaQC7LBS3HHHHQa/giOP\nPNLQW6+jYP7FMsGPcz6q8itC36985St2Rk4RaZeZ5iOPPGLJBO8aviWudadMPTSvfBCQ+lNSkQ+e\nbUpFLRZtqs2EZXFJRd1MmTRCs2fPNmvXrq1FzxYCwfRZpNdMjoTV0DM49cR0PfLv5ezZM7EKA9C7\nXbBgwaDZIDRO2jBVWCkZssbpmqmnbfH/yQCFRvUQUGLhAdIvl3UmFVIHVZMLLCaymFPZZEIw4Aip\nYaXK3/72t9ai4z5ryjl1edlll4X6iii5aEot/klPvh84JFN3TSa7fyqRnuWJgBKLPNFsUFpnnnmm\nee+992q/rDLOYSysxTBNGQ6dYt6lKtPO5MjzNYDcUG5+6MPU36b18GmE2Brct1a4OIF7HfB2ddLz\naAQgijfeeKO1KEaH0if9ioASiz6sedaouPPOO826detKaayzQsyCSiwNjv9FEeKSiTo12n5je889\n99gZM02pN6kryCFLvvfaGwQSRe+3DAIpuukxHQJ0TFhBVxc5S4df22MpsWh7DXvl4+NNz5CZDE2Y\ndYH6Yna99957c5kZQXpFz+TwYE98CakIIzmQrHHjxjVmXJtyTJ06NbbJXMlF4lel9AgMzTE02LYV\nYUsHssUZKrFoceWGFY2GiX0amrZjJnuVnHbaaZYQpOnRumQibG+TMKyquCd6hpEK9JFZKDfddFPt\ne4tpdVVyUcWbFz9PLGdYzYqyIMbXREPWFQElFnWtmQL0Ege6ppnSBYqkpIiGTWZy1JlMSPnQF2LR\ny5IkJKsuM2ZEf/dIOVgbhamlaWYcQS4gkOoY6KJaj3Omgl966aW5WA/rUSLVIm8ElFjkjWiN0wub\n7ldjdYeoRmPDR41hjCirBWHqMJNjiPI9bkAqkDgNKWG/973vmYsuuqg203Hd4gmpGDlyZKZebRJM\n3Pz1vDgE5H9QCHtxOWnKTUZAt01vcu0l0B1rBdJkZyt68uyIyo6r7lCOSybwH+nV408AWylB0T9J\n75yP+j/8wz+YD37wg5Zo1clyIaQC4BYvXpwJP0gW5IJfHMKVKTONHAuBFStW2P/BWIE1UP8iMFCQ\nPPDAAwMBqvZ38sknF5RLccm6+lMOV6RcHIOdGt1HPc8D03AHl9tuu61n+LwCTJs2beDhhx/OK7nK\n0gl2+hwIGpkBjvILLBSV6ZM1Y8qQRH/Cu0Kd8h7WoW4DS9JAsGrqwAUXXOCqmPk8IFIDpK1SPQL8\n72ldVF8PdddAl/QOvsptF3p8K1euNBMnTmxFUdlnguEOHBz5RQ2L1L2wMvMjrv7UI974rmCBCsiG\nXaW0yu3nsYgxTMUMkDQ+FW6Z/HOsFfyw7KhUhwC+PcOHD2+cRbA6xPo3ZyUWfVD3Tz/9tJk5c2Zj\nG2C3iiASOI6xzXpTheECIRVJysAQCA2sL2BCelu2bLELUXFelkB2WNOA5ddZvMsdospTBxkqUnKR\nJ6rJ0uJ/jk0CVRSBXggosYhAaPr06Yx/dH4RwRpxm90kWcymLTJ+/PjGbspFQ8wPMpBEehERCAcL\nUPHhx2rADJoiCQbkiIXWKAeLl+FQm7RMScpPWCUXSRHLLzz1zQ67Rx99dH6JakqtRaBUYsH2usOG\nDev86Hm+8847keC+9NJLtnfqxtlzzz3tNr3i+e9HdsMSn9+xxx5r8yR/JE4YnJTccH4+7vWTTz7Z\nyYM45Me9NBJWZhwW0SetMAxy8MEHp42eOZ7gSDnyEHHObFrvFUKBiP5xsSCePwQSFff000+3jTxr\nlQjBwISdl4A5Qy44yT733HN2VgpDH3GHc7LqIeSiSNKUVcc2xl+zZo0J/LRCLWZtLK+WKSMCRTmB\nuM6POG/yC1Qd8ttjjz0Gtm3bNkSNq6++ekhYNz7x3njjjSHx3DB+GuIsGSeMqz/hXXHjX3LJJZF6\nSn5u3G7Om4R30/bPySup4GgVrNSYNFqu4aUcxxxzTG7pBkM7tXBYjFsg6gEnxDTiO2zGTQOn0Lvv\nvtvWf2DRsE6VQQORWA/yv/zyywelk7YscXWPEy4tLnHS1jCDEWiL8/fgUulVUQiUMt30/vvvD9qW\ncAlIhZ2Pv3z58k4ALAsXX3xx5zrshHjHHXec2bRpk91SOixMrzSIEydMWNpyb+HChXI65HjWWWeZ\nfffd12C67yVYOAjfTchrxIgRZtasWd2CDXq2efNmM2rUqEH3qrp46qmncst6v/32M7wDTRB612k3\n2Oo1BNKt/PTusWDww9LAO7ZkyRLryMuy4LwXvE9YAX3BGvHuu+/aDeCCmR6GH6bwo446yg9a2bX4\nlhQ9BFNZAWuSMRYzrJ5M81ZRBOIgUNpQSNBbNYGFwfos7Nixw3At4hIPhjjYtEiElfuCKZ0dX4eg\n1y6PbMNy6623dq7DTggfsDL7i2qQ44QJS1vuBZaGTh6BpUNu2yP7W8QRt1wuVjSegbWnkwTYRA0D\ndQI5J8THLF4XYagnDxk7dqw1xeeRVpFpCDFIM1SQZAikVxkYfsGxEj8M/h94T+fNmxdKKkjrnHPO\nsVu1E5Z1MubPn18rUiHlFXKBD4BKMQjwzgRTiEsb7iqmFJpqqQgUZQrxhxKCBm5QVqz/EBS08wvI\nhn3uxwtbJ8IdVmFIxBU3TcKFSZwwvh5uOm78gBC4j+y5P6QhZeNh2FAIQzpumj5WxKecEgbd4sr1\n118/wK9KEb05Mhzi4pFWL8zgmGfrKgxDZDXVZ41fV2yK0IuhJjBXyR8BhlIZQlNRBOIiUIrFAqvD\n7rvvHrQrfxL/WnrhbK8sEjSmocMIX/7ylyWItVpg7g8T5tX3kjhhuqVx/PHHD3l86KGHDrrXzUGV\ngAzniGCt8LFhn4tTTz1Vgpif//znnfNeJ5i06d1nFRwvxQkz6dHNm+EQzOoMd/XCxY3XpHMsDfyy\nmOjF0tGkclepKxYZMFfLRb61IA7SdRoCy7eEmloRCJRCLEaPHh1b97fffrsT1m+g5cHOO+8sp/Yo\npGTQzeDCD+c/5zpOmLB4cs8nAdyHCLgSpZ+ECXrwcmpoeMMabteX47333uuEj3Pi6xMnTpFh1q9f\nb/1JmrwWRRQ+NG5I0pkfbnqkEXcWiBuv38+VXOT/BtABYDaIiiKQBIFSiEUShTSsItBUBKR3F7aI\nVZIyRS2ElSSNfg0r5EIIXr/ikFe5capnTR8VRSAJArUjFuyIKPL888/L6aCj28PnQZU9chxSffEt\nFGFWDTeOazXBUTMYx+r6u/LKK93oXc/x+o8aKuoaUR8mQoChCwhFVlKhQyCJYA8NLNYiJReh8MS+\nyfonYCl4xo6oAfsegVKmmyZBmWlwIsxoYBaBP10zmJsvQQx+GGPGjOlcl32CDwMLYrniEiL060Us\n9tlnn0504kJM8iJLTCX0iVgnswQnzAo4//zzE8ToHbQXLr1TqEeIvMhAGUMg5PHyyy9b3yTeXUSm\nlXIOMZowYQKnhv9F3h/+/5rWuFAOysovK9mzYPThH6wVTFVWUQSSIlA7YsHaFDTGkArkn/7pn8y3\nvvWtDrlgtU53eqrr1Ji08HmE99eWYIVMdz2KOPpBjHBwxfeAcn/xi1+0W04LYSJNVjsUTIJZIYnM\nk3kQC9ElD8zySAMrDNaYKgVHwTyXsmYIJIvDZxQWDNHwDrEWwfbt2y1xYAryKaec0iG9ki8NMXog\nTPNet26dfReJN2nSJLs0PBufNUGEXFD+phGjqvHl3V66dKkJFkarWhXNv4kIxJ0+kjScO10zbNpn\n0Eh2pk8GuA1afdOfrsnzsF9AQIZMXXTDRU3LjBPG1Z/wrrjxe51TTlfCppvy3M8vKl3iJ5G6T8tM\nUhY3LCtBVjmNlpUnmeKYlxQxtZSt1JkqGDSwduXMLHlQXlbxZFt00gN77jVBAgtgrnXVhDJn1ZG6\nZnVbFUUgDQK187EIGlS7sqS7YBT3fMGq8cQTT+Q2ZOCnH/c6WDY8MigLZ8U19+MgRfhuglVj1apV\n3YIMecbsAnqqbROmJYvJvuyy0atH8uoFk16es0DYwnz//fc3l112mVmwYIG1QDCUJVaJNHjR+8cs\nzmJZ7GK6detWqzMbkdV9iqfsLyLOtWnK329x7rnnnlZtXNhv9Vd1eWtJLAAFB0WIg08wIBQ0wKz9\nUAfzPOs7BNYGO5QhlclaFOgetdKnhPOPhMcZ1CcrEArK/Oqrr8YmKpI2DQJj5W36qNKQQZZYJrts\nERzBNS9h6CGP9NCNXU2FUGzYsMEUMWwBQWHjMfTGT4N6yHOjs7xwddNRcuGi0f0cosu7VMS70z1n\nfdoWBIZh5mhLYbQc4Qjgn4ETHks6t0FoxCCe9J7LFJw00+75EaVnXo6f9DAh4SzTfcYZZ5S6/DL1\ncdppp1kfjMWLF5eadxSuUffz9ouJyqfJ99k2AL8syKOKIpAGgdpaLNIURuOEI4DTHebrNgiN2LJl\ny0r3VhcCkGbPjyjc8xoCgThCKqhjyGOeOkbp7t5nVUacWFm4bcqUKbW2joENFhfqUyUcAayBJ510\nUvhDvasIxEBAiUUMkJoehA8/pk0x4ze5PJ/+9KftMNPw4cNt40ADUWQjQQ9XSEXeuGUdAkE3lqRn\nNlGes1PSlJMGm82qGLZDp7q/a0ouwmtZ/pey+OOEp6x3+wkBHQrpk9pui3kTk//q1attI+ZWnXwQ\n5V4eQxZYFGi883LSFN04ZiUr6IV1AMGht2wrhc044g/Oo7Nnz7ZDVUVgF5FtqttZ6yFVpjWOhPWL\nBftw9lVRBNIioMQiLXINiydm96y95KqLzWyH6667rucW3vSY3RVQmXWRxEESvJAkceJik0faOGmy\nsFXdSIVg0DRykQcRlbI39QhZBQfIVhHvfVNxUb2TI6DEIjlmjY1BbwRpqlMW1gqcA5ntkFRozCFV\nIqxsGtWbhpRgASjq45q1l4z16dlnn60tqRCMm6In+lLn1HedLD+CY1lHyOCNN95YulN0WeXTfMpD\nQIlFeVhXnpNYLRiPj2pUK1cyQgHpTeGgmMf4L+mBgysy7l5k7zUrqZAZGE3pVWJZ2WWXXcwdd9zh\nQl3L834nF2eeeWajVlat5UukSlkElFj02YvAgkY0zmVP1cwKMx89pMgGCouIuzZKHgTGLXfWIRAh\nV/fee2/PoSA33yrPm6Zz0daqKuuiW97S6WD4sJ+tNt0w0mfxEVBiER+r1oTEa3/q1KmNWdei6F66\nWC98IoFVwJWsloys1ooyyJVb3rzOpf6wEDWh0cpKAPPCrcx0INXsC1MkcS+zPJpXtQgosagW/0py\np1cGuWhCz7doXWlEIBZxhobQJa1DaFZSQXzIYFMaZ//FZkiEFWCbMtug38gF3wM2eGRquooikBUB\nJRZZEWxo/CZ47dPgM6Uy2PiqkAYpa+NB/DgOoVnz4RWjYWZH0qaungoGzMxp0qykPOqtCZ8HIe/u\nu9wEvVXH+iKgxKK+dVO4Zpg/WbERf4s4PfbCFXIygFQcffTR5jOf+Uwhs1j4mOY980OGVJxidFZ5\n9IdZ3DC9zpturZDyNXGNBMhFXIuWlLNpx7ascdM03NusrxKLNtdujLLxsV++fHmtyIVYKvbZZx9z\n6qmn5jILxIWChjqrv4SbXrdz3yE0Tb7UURv2emlqz5j3EYJRN/Ld7b1L8gxLUh07F0nKoGHrhYAS\ni3rVRyXayLBIHXwuaHzYp2DixIkdS0WeRIC0slgPklRQmCmd8iXx06BRY82NJg0hdMOIsfw5c+Y0\nbufMtpILHGsvuOCCVGvDdKtnfdbfCLzv3wLpbwi09HvvvbcZOXKk3eb9d7/7nfnsZz9bCSj07tkl\n82tf+5q5+OKLOzrQo9qyZYv5v//7v9SzCmgYXnvttdJIBcrjaImFwpXddtvN+hpQJn7oRThICL/f\n/OY3hjAi3/3ud83//M//2CWy5V7Tj2vWrDF///d/36hifOADHzD8eIeot7bIzTffbP2YWNFWRRHI\nCwG1WOSFZAvSoTd9zjnn2JIsWrSotEaYBhWnxLfeesvcfvvtkfkSjoY4qUk6bbwsVZrWMiJEQ/K+\n/vrrra/J6aefLrcafaQusBg12VEwbd3WreJ419pkDasbvv2sj+5u2s+175WdBpuxVqY18mNsn4ag\nKOHDhuMYPUCmIjKPvtswBUsu8+PDHldE/6RkJG76YeHIM22vFodSMJDfpk2bzIc//OHa7xYahkPY\nPepPdqYNe96Ee9RNknewbmXi/w7BcjRt2rTClq6vW7lVn/IQUGJRHtaNyQnrAeZ5hAaShZkYi81L\nsIxAWugt7dixw/ZeWd8gzuJJ0vDyYZcPZJRe5IPQmJUpeflDQFA2btxop5qWSYyKxgr/mc2bNxed\nTaHpN5lczJ071/4/L1u2zJxyyimF4qSJ9ycCSiz6s957lpoGnM3KcDTcb7/9rIMX47AQAkhGr0bd\nzwAigHWCNHDgY2vm5557zixYsCBVw8+HnYZXLBJh+YmFw39W5DXlRLc8BIJCj7JtwgyX119/vfHF\nEnKR9H+h6oK/99571jl65cqVdviRZf6j/o+q1lXzbyYC72+m2qp1WQhAMLBg8MMCsGLFCrNkyRL7\nYWL4YtSoUXYYA6LgC8SBrb3ZiZNFrvj5W55naYjpxfNBRC+3R58lTb8MSa7RJe0QSFg+9OpHjBgR\n9qjR9yZMmGBJZaML8UflIRe8f5DYOBa3upUZJ2lmhZRt1asbDqpPvggoscgXz1anRuPtLsnMBxWL\nxosvvhhabhxBGe7o1oOXXl+3MKGJ//EmH0R6jJAJZmAwhJM2rW75xHmGhSHPvBkmonevUm8E+L9o\nKrmA7GOZVFEE8kRAiUWeaPZZWmIlyNqY0sunt5+210RPkTSefPJJc+yxx1ZSC1VZSSopbMZMIYCY\n4dskQi54F9O+x2XjAalYtWpV2dlqfn2AgPpY9EEl172IYnVIO1Yt48Psp8F52nTS4pT3EEhaPZoS\nr4lDBnGwFaIt72OcOFWF4X+OIc221kVVuGq+f0BAiYW+CbVAgI+yzERJohAmaER6iaTDh73Mj3te\ns0CSlFvD1hMBeQ/LfP/SIPHoo48O8ktKk4bGUQSiEFBiEYWM3i8dAUzkQhTiZM7wAx9y+ZhLHOk5\nJklL4iY96hBIUsRMpmGv5LmVH0Pex7qSiyuvvDJXX6DyEdYc646AEou611Af6YdZll+cD7I06FGm\nXCEchCtK0DPPWSBF6Vm3dLHwMDOkzSLkogxymxRHId5J42l4RSAuAuq8GRcpDddBgAb15ZdfNtu2\nbetMG5RppQSSaahyzsyG8ePHxzK98kEWS0QnQ+cE/4m4Mz8gHTiWkh7WkCgS4iSf6DTvWSB+5nvs\nsYd57LHH/NuNv3Y3YWt8YboUgHeZ9xVykUdjzv/dj3/8Y/PGG2/YPUtYj8L9vyM/IWz8D/J/N2bM\nGLVOdKkjfVQMArpXSDG4ti5VPo6sYYE3//bt2+0H7PDDDzdjx461U0opsMwOISyNBz/5CL7yyis2\n3owZM8ykSZPMUUcd1RUjsUi4gfiw8qFO85FGJ4iF9CTddNOch+mXJp1ucchj3rx5dpn1buGa9owF\nmRDWRukH4Z3l3U373sr/HauwsmAa/3eQzt13393C5//fcZMp4Fu3brXLdvP/yv/c5MmT7fozeRPs\nfqhDLWMyBJRYJMOrr0LzQWQ/gcsuu8yWm4/a9OnTU30gSYDG/aWXXjKLFy+2JING84wzzgi1JPgf\nYz7MSBZikIWY2Mz/+CcPXdz0os7BgHVAIGhpG6aotKu8zxLxNIyzZs3KVJ9VliFp3tRlXEsbaT/y\nyCPmxhtvtP8zccl4lE68O08//bRh92D+B88++2wzc+bMvsE+Che9XyACAyqKQAgCDz/88EDQiA8E\nZGKA87zlBz/4gU2bPIIdPAeCxnNIFsGH2N7nGAw7DHme5gb5kHcWyRo/Tt7kwe/AAw8cePDBB+NE\naUwY6lzqVMpZBqZ1AKhXOQMiPxAMY9hfEf934B6stDkQNCn2GPZ/VwecVIdmI2Carb5qnzcCfHiC\nhXPsh63XRzCPvMkD8sLHlI+qL3fffXco6fDDJb0m3zQf1aIwIV33J+W5/PLLB/i1RXi/qOswccuf\nF5EMy6fqe2HvEOXl/wDSVQSh8MtMfoHVwubH/5iKIpAnAkos8kSz4WnxgaEngwWhbBELCY2oNPh8\ngDmnMSpCSDdJA0bYJOG76UzebkMaFVZ6sFHPm3af+qXH3EvAOQ4+vdKp63OXXIS9+2XpjR4QPUiG\n/N+Vlbfm014E1MeiwGGmpiTN+C9+FPhTPPDAA6l9KLKWl7HgL33pS+b3v/+9CRog89nPftYmWaRP\nA2lT/jiOdVkcNsknaCw7ECWZpcKUVhY0Eie9TiINPGF329tvvz1xWcBeBDyCnr1cNvZIme677z7r\nEF1l/fL+s5U6DtZV/v83tiJV8SEIKLEYAkl/3eCjMmXKFFto9g2o2mOcBvhrX/ua3fdj7dq1nQY/\nS6Peq0bBILAgdG3skuYvaUreWRpDtptnQ7KmbxaFQyIEdsOGDQJLqqNL0nBujUMKU2VUcKQLL7zQ\nfP/73zf33nuvGT16dMG59U6e2TqLFi2ys5CaimnvUmqIMhBQYlEGyjXNQ0jFAQccUJtGC50gN3x0\nly9fPugjl7RxTwo76YdZEmjIkF69ZOKL5NngkT/EBItHLx0k/zoe2cvllFNOMSeccEJu6vkELqz+\ncsssx4R4v9evX283AaMMdalXyN/s2bMH/d/lWGxNqk8QUGLRJxXtF7OOpMLXUcgF1gTIBjrTyBbZ\nmwpb7yKK0LhEAt2LHKoAC6SpVguwmjp1qrUMFWkVo/4CXwGLVZ7kziaY0x+XVBSJRVp1lVykRU7j\nCQJKLASJPjvW/eMm1SF6MiyC0HDQuyvygwx5gcRAYFxS4TZa6FIkkSB9V9CJ/NzhIfd53c/xrViw\nYEGu1opeZaYOIaUidbBmMNxw5513mnXr1hX6DkuZ0x5Z84L1ZuquZ9ryabxiEVBiUSy+tUydj8Yl\nl1xSeO8xr8IfeeSRJpgCa+bPn2+TdBv7vPLw06FRwqHuL/7iL8zw4cPt46obJholdBKS5etc1+u6\n6O0SwyqsGWK1aQo5ZCEzlg1/6KGH6vpqqV41RUCJRU0rpii1pOdbpRd60rKF6VwEufB7uCyFDKmo\nmlC4eEGyGFJoynLYNObgh+WgyCEsF6M4535dF13H5EceCxcuNKeffnocFSsPg86HHXaYnTHSFJ0r\nB00VsAgoseizFwEHumDeeqf335Ti+6ZZyAaS1ekNgiLi9mJd4lLG8Ivo0OuILpALzNQsr15naVLD\n5FozsszgiaoPZvaw10fTev9iZeGY9X8tChu93z4ElFi0r04jSyQfCXGGjAxY0weQIjZgkt662/jH\nVdltQIgT5icRRlqIh19HHT6uN910k/l//+//mf/4j/+olRXArQNIBdOY6zTjyNWv2zn17645EvaO\ndIvvPyO9Js/qabrjsF8fel0CAu1d+0tL5iPACntlLBfs55vXNasEBg37oBUC3RUMw/IJSNSgFRzj\nrC4YlSarQZJelYJulIFVUsGian2isGB1TZaGj4N3VBp1uQ/m8uMdSCpgUcVqtkn1jApPmYOmKPOq\ns8GOqzYd0rrtttuisuur+8GCZB1MwKVsoR7Il1+wE3Vu2ZdfktxUzyehoEfVAdZ/2bs9yyf38lJp\ny9LQ7KfgfqRpuNzGlY+gNALSCCdBmTjdhPx6hekWP+2zsHxpsOpGLtCT5aHbQir8+vLfL/+5fy2N\nMrg0WXjX+GUR+Z5y7CeRhpsjRMKVqokFuki9HHPMMa5qmc6VWDSEWHR7OeO8AXzs27DZEI26u4kV\nH2x2//ze975nG3w+5GmFuHHjpyEtWfRyyZObjlguwjZwc8OVcU5dQCjaSirCMOQ9kF/Yu5NHgxyW\nb9n3KBvfoLQEye0Z+41r2WUpO79u3+46EAtXh7zqRolFHxCLrB+Fsv8Re+X3N3/zNwPf+ta37Add\nGlw+7lklaRrkHdaYZNXDjR8nD9nEyrXkuGmUcQ52WE+y9mrL0LWoPHgXhGTIe1k3i1KWsmMtTDOU\nGixHP7DHHntYYsKx36QbsagDFn79cJ1V/iwotErLEXj66adNYLGo9YI8casAh01mQ7z99tvW8VKm\nMOJgx7O0ksYRVPLGsbMIQaegYerpMMoS2ayNwBRiZowUpU9YGXHSZMYDU2BxKm3qyqBhZUt6j7ri\nPeTH+c0332zcmUZJ06tbeJZjX7ZsWWK1gl6w2bZtm403Z86cQfFXrFhhhg0bZn+XXnqpfXbNNdd0\n7vHs3HPPNZs3bx4Uz71gU7sDDzxwUBzuvfPOO26wQeekR7qS95577mnQhThyj6OfBte+foTjnq/j\njBkzbFpuxieddJK9R16IW37SQQKr0CAd0NMXP8yTTz45KMhLL71kwNMtC/pIvm5g3tFTTz3V3qKe\nnnjiCfdxuvM0zASzlozLBLlaJsq9QKlByQWLMHX8F2Cq/nM3jbBxN5iTa0LrlpebcVz9iOPqQDxX\nuj0jHM5IbhnR7+STTx4yjiZpuumBBfEZ15JygZGvA+nJc/8Y12zFMEianoboXbcjvUF3OET0o8eY\nxoKQNp7ki3k4qbVD4oYds6SH1YJeMpaDNFiE6RN2Dx0lL96vIvMKy78J99ginl9bhDrnG8Qxibjf\nPb55rrhmeL6l7vfQ/975cfmGdgvP9zTMIZF0/LTl+uqrrx70zG2zSEvCRR3d73ecb7dbftIUOeec\nczp5hVl5yEd04LlrZXCfSRj3CM6+BGSik14evhZ/KomfU8h10ookPIWWQrkF8gH1Xxoq0Y0rabhH\nP05S/Sii+9K7L0WvZ2kqz8/LLYt7zgspEufllLBRR9LO+8MP1tQhdYqOYXXFPZ4RhrDEyUtoPMPK\nBOlI+uHLixSQTtK8fTwok5jR/Wdxr0kDYkG9c8yanpsvaQuhwDSeF3ZuHm05x9ckb3yq/r+jTEn8\nefzG2K9bvx1wv4P+ud/gdSMVEpdvkNvoch72rZLw/tH9Zrnfbz+cey35xfl2++UXfPz7PkFyiQfn\nIi5BcHXyz/22Dp3dMH5+kn7cYyJikaYi/QZYKsqtJBcYFI9b+aThShr9XD18sKOepa08Nz23EsPO\nyQOJ83K6GPjnUb17P1zca+rPfanDdO92j7jyDsTNMyxct/HeJB/zJGHD9PDvgXcY4fHDhV1niRuW\nHnrQY4aEYeHhPE150QsnUSwT1C3HNOmE6djme2CVl9Tl/453KIkvj/v994kB2PgNKGGkUaMd8L9/\n0mj78dxvd7dnrj7UjxvPt1bwXL5VvpXDjec/k2+31D3pyA/dXPF1lWd+Q+/mRxiXHLn5uW2MiyXl\ncLH0CRdpunH9/HieRGK/+T4AbsbdnqGMq7D0XgVoCiiVJ4qTtjzn6OblV76A2k2Hbs9c3dx8fL3d\nZ26cJJXnxvPLRTncMlNOV9xnlCeu0LugEc5D0NF9oV2dkpyThtRbWr34uEV94LAa0Bj2EhretCSg\nW9qkGSd/Nw3CZ7V2uOn557wHNAoQDOqKnicEQXD0j4TlvYGU8CMsw2lF6ujr3ORriBcY5yF1+r/j\nHeBdiCtuJ8TvQJKG/2322wLf4iHP3XRdS7jo5bYhfHdF+F7LtyosHvfkOUfJz02P75cv7rfd/z67\n6fnP/PK76bpldLFzMUEXIVvufVd3SZNw7vfb18UlHmHYSDpxjrGdN4PpfIGuf5AgUzNr1iy5tM50\nAfCd61tuuaVzzkngwd+5Zq38RYsWda7ZWGr33XfvXHPiLnvr53XRRRfZ1fwkwmuvvWZPs+gnacU5\n4qCzfv36TtClS5eaMWPG2GvKceutt5qg8ux18FJGOsL45Tr22GNN8PJ00mXznzwkeNnsapVZ08Jp\niTqnTFmFNEjLd4xKki4YP/fcc6FRZOdTHAu7SUAAejpGdosf9SxoiG26cZ1JxUlT9I5KN8v9o446\nyi7jvmHDBuscxv8g+0BEyc4772zmzZtnHWLB6Y477rA7kxapY5QuTbwfEDCz6667Zla9bv93fOOS\nfJtefvnlDgZ77bVX5zzsJGich7QF8m31w7vpshqvL5MmTerc4ntNffB76qmnOvfD4oXdIwLfq6BB\ntb+tW7faNHCQxEkU50q3TehkkPHkc5/7XCeF73znO53z559/vnP++c9/3joIc2PTpk2d+wGBGoKl\n66RJwJ///Oed8JyMHDmyc/3KK690ztOcvD9upDQVSUGQ8ePH2+WZDG8AACv4SURBVN00IRWIVAIv\nkktQ7MPgj1v5QQ9LbneOr776audcTrLoJ2nEOcatPCmrX3mSR1i5PvnJT8rj2h2XL18+hFRQfwHL\nNbvssovZd999Q3V+/fXX7Yco6DEPqlfIBWlCFNOIT0b9NPwtz/3naWaB+Gl0u6YBlpkqURtcQXwC\nS4UN1y2tvJ+JbuinUgwCeRH6uv3fsTT5ypUrY4PGRn4ifCe6yejRo7s9HvRM2hBuHnfccYOehV1A\nKnwZO3asf6vTKRzyILhBGkEv35x11llhj3O/55aL7yWdWojWj370o05eLJsvElgk5NR+a2WWSeem\nd9KNIP7iF7/wQie7jE0s0lSkEAtUojd+3333DWqcXEuGqO33YpkGFEey6hcnD8LkVXlxyxVXr6hw\n9OqZJpZVIAau8A8WZxMsSCUCgYDhT5gwoZMMaaYlFp1EupxIw+43oLJ3Q5eouT0i77B9RtABYuHr\nllvGmlArEKjb/x3WuCSSh4UzSX5FhYVUBENbnU6x5IPlecSIEQYrvdsGyfMsR9pPOm7333+/TQZL\nBR2qJUuW2Gustu73NEteecf9s7wTjEqPivFfMnqzKsUj0Kt3H0cD14rEP1McUuGnK5Yrue+mKffy\nPtLDohF3paghEDcP99xf70LWmZD7blg9VwRcBNz/kSb930kZsGoWIW66gfNkZ5hChiv8Y9g3EKuS\nL34bJc/pSAlxIG/CkceVV14ZanWXeFmPrEsjgqWCsoq4wyDcY/hSBELiY+Bfo3tREptYZK1IFtHx\nhXu+hcK1chBexrP8uP51Vv389KKu61R5UToWfR+GnlayxE2TJz0sLAPib1H0EEiUjuJ3wfbvch4V\nVu8rAmEIZPnfyRI3TJe493bbbbdO0G6m906gmCcHH3xwJ2TcDirkQvzfiBzmoxV2j7AsQCdy9tln\nD/JfoNMspEPC5HVkQS0RLBWufu4wCGH22WcfCWqwbqBXWkkyLBWWR2xikaYiJUNW+xJzDhWLYwkC\n6xOzjoSFWLiVH+ajwDCCrCiG8wySRT/JO84xz8qLk1/WMIxr+ivCZU3T/SdLmlaWuEnzkvBYBvBl\nKHMIRPKWo/hTnH766VYXITryXI+KQC8EsvzvZInbS69uzz/+8Y93Hv/sZz/rnGc9cR0b8XmQdoB0\n6ay6q2qyKqeIrDDJNX5wbjzOxTdOwocdmVwgjTbDu1/84hfDgoXec4fSQwN4N2U4RG6LfmHDIPhf\nSAebthW93G8/7bDbdvqrcLKasYikI9eJj4F5JJYwNSVIvPNzp18GhRi0tkGgVCdNf4oL8fx5v1y7\nEpj8OvmQpzs10Z9uyhQbJK1+6CrlcstEmlHP3PvudFPRI6j0TproJeLG88tMGPIXXcDAFbnP0dfT\nDeefyzRC/37Sa3cqEjpQD9RtXCGsWz7SIM20wnS+JNNogw+B3awsbX5Z4oVNP2V6KffLFnAAO94L\nmVIKjv6PhbUIE4zxV6Jn2bjknZ/gmzXduv3f8d4GFrfYxXL/5/lW+uJ+t6O+B+63j7ZGxP2eumH8\nc/cbTHz/ebdryc9vd7rFcfND17CwEsYtP+HCxMVQ0nKnn7px/PQkvH8EO1/cdsttc/1wca7DSxIR\nM01FuiQBxaUxcv9h/ELGrXwf3DT6uXH8BjvqWdrKc9PLQizkJZGXM6K67O28PnBhLzd68LGgLnke\n9uMZYURn9+jj3a0c/jMWbKLxiys0pjTkZTfm3QhEWfqQD3ixrgL4cxTiAC5hP8Lz7kA4iJNlga24\nddSmcGCahPhGlb1u/3dJy+V/y+X7L+V1v6VJiQVpR31b5DsT9o1x85RwcvQJhBALjm7DK+E50sbR\nFsk90nAlTEf5dvu6uPHkHMwkbTl2a/ij3hmJSzsk5YrKw68nCRf3mIhYJK1IrAlSGI5uJXd7hvI+\n4G46nFNZfuGT6kc+bmPv6tfrWZrKc/NKSiy6vZzoGiVJPwRR6YB1mA5+vcS9Dqu/qLzD7idZAdBt\nwMGjLCEvLATdBN0gH0UI1gYhBhAJrnvpE6UHZZEFtiAZEI+0aUXl0ab71Ck4ZZW6/d8lJfSU3/1u\n+A2i+51PSiwEW77Fbh58g2jsw76xEodn5CffK77N6MJ9ucfRbWNos1wCIXFIM4qQ8MyPR7rkhbjl\n536UuOVzO+hR4cnT1wl9/TZO4lMvUu6oepCwcY7RJekSO25FumBQKF98a4bPogDHDUPBu4Ej6cfV\nj/CkJ4D6oHd7RtykleemF/bSk7/oQrld6fZyuuH8cxquJKZLP757jQ5unYquSY+kQVpZhAaThjKO\n+GTCv46TRpIwMtwQN07S8L3SpXw0akURACEsvFdYNVTCEeD/Ig/yVaf/Owgq5CKJuI2n/11Lkk4Z\nYd1vMA14v4hLmIT0ZCl7KmKRJUONWz4CNDB59or553NJUlxiQRyfvKVFI+5HO4xEuBaMtPlHxcti\ngUDXLA0RebPcMg1+0o9/VHm63UdfCB7vVxjO3eL2w7Mk5DcOHnX4v0tT1/T6ZRghTm87DhZpw7gd\nI75HbmeWzp7oyfeFsP0g1I98w8EkDxlGIkGiKi1G4MILL7QrYzIjIU/B+/qFF16wi4Yxx/qXv/zl\noOQ/+tGPGlYTZYruIYccMmiK1qCACS+eeeYZO3+8l6e7rF8RNLRDcmAtCe7nuUR12EJYQzLucSNt\nGmBy2mmnmRkzZpgFCxbkWq4eKhum0AY9UcM0PJboV/kDAjfccIOdLn/ttdfmCklV/3f8P7GgW0Bg\nE5eHGReyYmVAkApd+6Gbcq4e3cLxLOi5p1qvp1e6dXvuYpJbmfNgJ5pGvRFg46C8NkSquqSY4efM\nmWPH+3vp0qsX3et5r/Td51iEslgb3LSSWj3wfcBKEXdoyM0rr3N05h3jlxcOeelWRTpg8OCDDw4M\nHz68NXik8a9wsRdrQV69YjftJOeub0XQyHd66+553YdskpS3W1jXmpSnhUaHQrqh3pJnfORoePj4\nN10oy1/+5V/ajzbEIIocRN13y09aeQwRkRdp5SmkF6cMjHnTmOdRjjz0R5+8h97y0KuMNKgD6oyf\n1AdYVEn48ix31rLgKyKNd15DomnLhx+B61cgeuHwGOb/ljafusejHig7Q0DusFBWvXUoJEC1H4Th\nECRvs2zZ2D3yyCPmxhtvHLQSnruLKAvKyPBG2BCIr2+34RI/rH8ti14Vud8HZYvaxIw6ZcW/VatW\ndcrs61jFNXqxeRZDVW1etpx3xx0WCKsnholWr149aMfmKuoka568h1OnTh1U3qxpavz2IqDEor11\nO6hkMj4a9KRq1QgNUjLGxf777299CE444YTQ0DT2wdBPZyt79grpRTDSLPMNnuRVRsMZ5ndRV1Ih\nlSLkounvm5RHji6JjfNu8Y5AOIjX6z2UPOp4PPHEEw1bip933nl1VE91qhkCSixqViFFqsPHniVd\nm/pxwFpx2WWXmQ0bNkTC5JOEOL1KEvPjRWYQPAhr6LuFz+OZS2RwCrzzzjvNunXrak0S605+4tQL\ndR0MS3WCprFOUV/s8cBS0E0U/jewVrSNJDaxLpqisxKLptRUDnrSONHLwnzbtN4TPb/DDjvMXHfd\ndeaoo44KRYPyId3KFtVQkD7xe1kg+MiGmbxDFcr5JjpijfnXf/1X8+yzz/bUNefsUyXH7oyBD0hj\nZouAMQ2oSB51TZqk8+ijj9pZFZJ2U45qrWhKTdVHTyUW9amLUjRhR9mNGzc2rvcUR+8kVgcBmzgi\n//u//2sOPfTQUCuANDhpeqySftajNFA33XSTiRoKyppH3vEha2B27733RhLCvPNMmp77DuCj04tc\nJk2f8PhaLF68uPZWJr9sonc3K6EfR68VASUWffYO0DjR8587d67Je12LoqDkw48plmOUNYJnWRt9\nsAnzz6Bx5FkRDU4SzBhaYOvpO+64I0m0ysPKEFZdhm6oT9fpMut7Exdgev7BzIrGWG+w7mFxaqql\nJW69aLj8EVBikT+mtU+RDwamWUy+VTeWvcCK00unoUCiSEevPNzn5Ed64MKRBcA+9KEPmWA9gsqG\nQNBPPvLdyJVbjrqdV21OBzeROE6XEjbPI+8TJKYJFif+D6ZMmWIJfVN9svKsO00rGQJKLJLh1ZrQ\n9CJnz55d6ymB8nELFtTpOk02D2uFW7FCVOjVumPsUf4ZbtyizqtumLOWizoq0wGwyrrqhpWskMoU\nVN7rusqZZ55prWNNdTitK679opcSi36p6ZBy1tlrX0jFrrvu2tUfJG9SAUzkzZBIr6Eitxdc1Ng8\n+oi1oule+UWSI+osb6dLsC9C/v3f/90ORTJTpI4Wwzp/F4qoD00zfwSUWOSPaaNSlI/IkiVLavOR\nE1IBkN0WfxLLQh5DIFJppEn+fPCTkBa/YcvT3E4dsd9K0/fhEKuF698guKc5lkXs0ugWFUfer82b\nN9fSYijfg27/d1Fl0/uKgCCgxEKQ6OMjH5O6rJTIh/dLX/qSGTlypPWil1U0w6onScMfFt+/h2WA\n/ISoQBbQJ02vknhuA+oOqfj5drtGB+JSVtGrW/i6P2OBs25Thrvp72NaltNlN52SPEN/ROpRhiPr\n4HPBe4ZPBaKkwsKgfzIg8L5/CyRDfI3aAgSCzWdsQ85skb322svw8a9CVqxYYcfhp0+fbhufD3zg\nA5FqFEEq+ODvtttunTzJnymoHLvp0ongnEBQsFrIb8uWLeZnP/uZJSq/+c1vBuXjRBty+v3vf99I\n73bIwwbe+OAHP2ief/55wzvXS2jsXnvtNYsZjTLDTZAswbRX/Do990kFuu299972f23WrFnmd7/7\nnfnsZz9bicr8LzF9manWt9xyS+h060oU00wbi4BaLBpbdfkrTo/9pJNOMqNGjTKsFig9q/xzGpwi\nDQjTXx9//HFrpWBKXjcrQdhHenCKya74sHazKORNYiiv6w/QbdgEa1KTV0v1a4K6w9LgWnPcMK7T\nZZF+K26eRZ/3el95jpUOWbRoUeZp03HLw3v4zW9+05KJhQsX9vQpipuuhlMEdHfTrNu4tSw+u2pe\nf/31dsc7dqoMGoDCSih5BQRmYObMmZ0dQrkfNLyR+cbZ9TMysvOAfOKmFTeck3zsUzAmffmhlwg7\nSnbDQsI16eiWSeogrOxNKlOUrtRt3P8h/u/4Xyj6/w5dA+dkm9e0adNi6xdVRr2vCPgIKLHwEdFr\niwAfQz5wAfe2xzwbN9KWjygftrBGWxocvzrCwvph4lyjQ5IyEZ5fGYJelPPVV1+1+JeRZ5l5nHba\naQMXX3yxLWOSOihTxzzySvPO8N67/3d5ve+Uh7T5v4PY8WvLdu551JWmkS8Cf6ZGG0UgDAGGQdhi\nPfg42seswMePIRKGBpIKJm6WByYNTP9bt261K/oxTz7MCQ8fBe6TFyZbBJMxcbMKuiDdhlv8PMAD\nPUQX/3me1+hF2X//+9+bgHjlmXQt0jrwwAPNn//5n9syJqmDWigfU4lewx9RyfDey/8dQ2D4X+Dz\nxJL2af7v0AMnUdalYGgJn5Xbb7/dbuQXtedOlG56XxGIi4D6WMRFSsMZFvfBDyLo6dj9Rmj0RowY\nYX0AgIcpkXy8tm3bZtHasWOHDffiiy/a60mTJpnJkyebiRMnJnIQgwjwwYXkhJEQm3jMP3ycu/lT\n9EqG+Fl16JWHPIeIvf76610XB5OwTTqCIb4EbV18KS2piKpD/u9YAZaN5/ixqRuzpvbbb79OlLFj\nx5o33njDXkf93x188MGl+U11FNOTvkRAiUVfVnv2QtNzD8zYdsYCHzIEKwR7WbgfvAkTJlgrAz3+\nLPLd737XfPzjH09kZXDzE32zkgLSoeEoo6eNdQhp25LKbSYWeZMK9x3mXN5jZgp1+7+DaOy+++6l\nvKe+jnqtCCix0Heg9gjIxxqrBWQmKTkgPh/kvMgAFhSIEvoUKW0lFtQFlq1gVLdI+EpPW97TrCS6\ndMU1Q0UgZwTUxyJnQDW5/BFgCEQackgFPV4apziSxp+iV7oQFAiOSjoEiiZk6bTKFkveMyUV2XDU\n2O1AQIlFO+qxtaUI82mAXNA7lB5iVOGJy4e+iI+9EJyovPV+/yAgFqwi3rP+QVFL2iYElFi0qTZb\nVhaIQ9QsEBnWkJ6iW3SsGUJIiuwdo1svcuPqped/QADM2tIIC6ko8j3T90YRaBoCSiyaVmN9pK8M\ngUQVWawRkAgRGi1+Sf0wJH6SI/lDYuIOyyRJu81hqVecepsuSiqaXoOqf1EIvL+ohDXd9iJAQ4qP\nAdNIZWpbWGllKioe6uyLkKSXKhaHsHTde/QUZVjife97n/mrv/qr3Jw03XyizrGcxNU1Ko2o+0zf\nXbduXdTjxt4PFmpqrO6iuJIKQUKPisBQBJRYDMVE74QggBXg6aeftotcyVz6Aw44wK5hMW/evJAY\nxk5FZQoqW7KvXLnSBKv92QWf2GRMhjLCIpJX1BBIWHjuMcvgt7/9bdTjQu+zLgYNTbcypVFgzJgx\nFu80cesch/UWeBeaKkoqmlpzqndZCOh007KQbmg+rNq3bNkya52YMWOGYZGrT3/606mmWspCP+yg\nOHz4cLNgwYLQxbKSWgAIL4teQUqwqOTdyPeqPvJFklhleqVJOdo4LZNVIFkojR01myZKKppWY6pv\nFQgosagC9QbkCQkI9iywmkYRgCzFcAnLTTfd1GlkkpAKGZLx/Smi7mfRN07cJLrHSY8wLOfMEsx+\nGePGr2M4rFFr164tnfxlxUJJRVYENX6/IKDEol9qOmY56Xmz0iP+E26DHzN64mA0xuyHsOuuuxq2\nCKchjdPrj2OZKKKh71XAvPMEE3wt5s+f3yvrRjyncWa/GBw4myRKKppUW6pr1QjorJCqa6BG+WNF\noGfM+DfOmWWYqslvw4YNZurUqdY8Hmf/CD7ySK/hDtKmoceCUZbkOQUV8sSeEPfdd58tR1llKDKf\nFStWGIbUmiS8Q5BdnVLapFpTXatEQC0WVaJfo7zpGS9fvtzuOFqV2R3CcM4551jrxV133RX6Iecj\nL/4UceEjXRqFOJaQuGl2C5e2d0s8d8YEJAWdmzp0EIYRFqnrrrvONGVnzbwtUGGY6D1FoG0IKLFo\nW40mLA+9+TPOOMO8++675tvf/nZpjW+Umugzd+5c8/bbb5tVq1Z1yEVWv4k4QydROqW5H6dB8olE\nFGFiy2ym9bKddpNF/HawUDVB4tRhE8qhOioCZSOgxKJsxGuUH431lClTrEZuI14HFbGgrF+/3pIL\n9OTXa+ijl95ZyUmv9N3n5AWZcXWmoXIliki4YTgnHawWvRYM8+PV7frEE080hx9+eCN2a1VSUbe3\nR/VpEgJKLJpUWznryrQ/3zKQcxaZkoNcfP/73zf33nuvGT16dKa03Mg0GnEbdTde0nO2ekeYWotk\nGWICC6SpVgswx48G3526+yooqbCvmv5RBFIjoMQiNXTNjoh5nYWu6mapcFHFdA6pYOGrOE6dbtxe\n53n7XYg1xM1XnEezEApJr+lWC2aCQCyYcVRnUVJR59pR3ZqCgBKLptRUjnrSYJ922ml2pkFZDo1p\n1afBZrimiEYpi9+FTyRYyMod9nDLm1djdcMNN5jnnnsud5Ll6lrE+T333GMWL15sZ/8UkX5eaeZV\nT3npo+koAk1FQIlFU2supd40iAwDYAloimc+1gV6vI8++mim4YQwyIQg9LIqSDhJoxuRkDByJK7v\nbyHPkhxJ57DDDrPOraeffnqSqJWFLbLu8iyUkoo80dS0+h0BJRZ99gbgV4HccccdjSp50b1eGhbX\n7wIi4C7ilIRIhAFLA5vHWgikg574KkRZSMLyr+IeRKgoa1Oe5VFSkSeampYiYIwSiz56C/iANsWB\nLqxasFrQUy+itw6ReOGFF8yHPvQh62wpa0iE6ZH2Xl4NGAuZzZ49u/bLYkNi33vvvVoP3eRVJ2nf\nCY2nCLQRASUWbazViDI1abpfWBHyJEb0/MMWo8ridxGms3svryER0nSn49ZxlkXd9aMusPr0GgJz\n60/PFQFFIB4CSizi4dT4UHk2ylWCATk6/vjjE1stfCLhDnv45Smy0YG4IHk4zUrjXYeFzVwMRa+6\nzjjKk+C55dZzRUAR+AMCuldIjzfhwAMPNMOGDbM/dpl0Re5zfOmll9xHXc/ZL8GN2zVwTg9vvfVW\nM2/evNqvIdCruCz5zQyDXgKRcn805PRO5detl88zwhGfRihPQQ/XdyNL2qxpccABB1hdIU5VC1hB\n/GRhs24YV6WrkoqqkNd8+wmByokFjbU0sjTiKvkjwMd06dKl9qOff+rlpigzWWj0XXFJBOdCIOSY\nppEjLhYGsTK4+WU5F9KSJQ2JC7lgF1osMDi4ViUQG2as7LLLLrVdG0VJRVVvh+bbbwi8v98K3I/l\nXbNmjZk5c2Yu5vc64Pf5z3/eEiVXFxrrIoSZF0Iu8hi+EB0hAjTGeczsYBfaH/zgB3b7+dWrVxvW\nu8hTV9E57EhjzYZx559/vrn77rsTD1GFpVnEPSUVRaCqaSoC4QhUbrEIV6s+d1999VUzMDBgf7Nm\nzaqPYgk0eeyxx2xvMkGU2gZlca/PfOYzduhJrBFFkQoBQRrpPIcbxIJCg5eHgMG6devslvfMaIFc\n5JV2lH7MTsFKwaJdOEIWMVsnKu8k95VUJEFLwyoCOSAQNJqVyIsvvjgQqB/6C8aNh+h02223DXDf\njcO9bdu2hYaVcCeffLJ9fvXVV3fiSgQJwxF9+B1zzDE2HGkjbp5yLyr+E0880YlPmqTFPV8eeOCB\nji6EC5Mk5Q2L794LGsaBYFzfvdX48yrKFMwiGQgsA7lil3d6KBc08gPTpk0bAKPLL78817oHg4cf\nfnhg3Lhx9sd5nQV9wUNFEVAEykMgvFUrIf+4xALiII29SwTkfI899hh44403BmlMoyzPIRZ+fAks\nYTi6xINrIRFxicUll1zSydNN101L8u1GLNKUV9INO/JRpYFpm9BwBkM8lRQLMkCDlZcUQS7Qjbq/\n4IIL7Ht5xBFHDARDFakaWSETpMW7BPZ1JxSUX0kFKKgoAuUjUHsfC3wDnnrqqaB9DpegITbHHXec\n2bRpk2F1RF/uv/9+/1bo9cUXXxx6P+7NhQsXRgY966yzzL777mvGjx8fGUYeZC2vpCPHd955x0yY\nMEEuKznOmDHDSD0Er3guOrD9dkAoK1mWnGEHhkUYzgga2szlYegCP4480nKVwX8D584FCxYY/GwY\nEgsIsA3COwGGyNixYwf972zevNns2LHDvPbaa+aVV14xGzduNAGZsNN8v/rVr+aup1Ui5z86/JEz\noJqcIpAAgcp8LGhkaWQCy0BH3cA6YO/h14AwLdMlFYQlDr+g19+JB7lwrzsP/nhCuoGFpBPXfy7X\nfHQl/TT+FFH6kT57c/SSvMrr5kPjKw2Ie7+q86AXmUvWgaXKNny5JJYiEXG6zMPvAkKR1xTUsKJA\ngHDwZBl38lm7dq1h2i4CgViyZIlZtGhR5/f666/bZ5MnTzbM2uJ/gt1l8aHIm/zYjHL+I862Ukc5\nJ6/JKQKKQC8Ego9GpcKQQ6Cj/THs4Erw8es8Y6jBl6i47n3SZtglTCRfjuKL4YdLMhTix/X1CD7i\nNkjUUEja8vr5utfXX3/9AL8qBWwF66i6SKofwweY5KsWzO15DWXklU7VmFSZP75EbfMnqhJPzVsR\nSINAZRaLoKHpKS+//HInTFive9KkSZ3nLMoT1RuOMwTBPhRZhNUgfTn00EMH3WJYopvkVV43D8zZ\n9O7rItIbros+WfXAGsDQSB6LackU1Kw69Wt8WW+kCVaVfq0jLXd/IFBrYgFZEMGPQhbSkqPfYIYR\nC4ZB4sjOO+8cJ1hkmN13333IM9/nI0w/N1Ie5XXT45xNoMJ088OVdX3FFVcYxvDbJpALMcGnLVve\nU1DT6tHEeEoqmlhrqnNbEag1sWgr6E0sl7tCqhC7uEdx3KTc4mx7zTXXtI5gyJh+Fr8L0ghmczTx\nFalMZyUVlUGvGSsCoQjUmli41gbX+TIY8+k4WbrnVfbMcZL0xbdQ9NKviPKyxHKvIRhf76KvIRfM\nwsF60TbBDM/PX3I8STllaCVJnH4Nq6SiX2tey11nBGo93fTggw+2GxoBIGPzcXwlqgKb1QePPfbY\nQdk///zznWuGbXoRiyLKu99++1krQUcRPSkcAdfvotsuqt0UKWoKKnliUWE4CoK3detWs2XLlkGq\nQEZ5bxgeHDNmjPUhGRSgJhdKKmpSEaqGIuAhUCuLxfbt2wep97nPfa5zzVoQ7u6i9MLPPffcjt9F\n1RuYsY6Fqx9TR9FZ5NRTT5XTyGNR5WVKYduEhpEGsM6Sxe8CqwdrMeSxLDdpsPz2mWeeaSAsJ510\nklm2bJmFbs8997S73rLzrfyYZopA5rnHkBfOzSwTLo25DVDhH9FDHTUrrATNWhGIQKBWFgt6UHzE\nGBJgLYvp06fbufXi1EhD7TbWbpn4AFYt3fSTdQO66VhEeVn8iHUKsgozcBiOylN859YkaUOW6FXX\nXfCZoBHESiA+GHF1Jrzs1Bo3jhuOuKzvct1113UWuAqW+O65FgWEyBWISTAV1jz++OPWeoFec+bM\nsWtjuOHKOldSURbSmo8ikBKBNHNU84zDXhqB6oN+AbHoZBGQjSFLcvvhWS/CFXf9CDctNwznbjqs\nLREmxJdwfj5yn6O/JLj7zI8XtY4F+acpb5jeco85/UGvTi5bc6xySe80IAb+NqmX006614Ws8UG9\ns4ZJnus6UI4q9wrRdSrSvH0aRxEoFwGcICsXGt7AB6HTgIeRAcL4e36waFbYgkuElYY9LC0psITh\nmJVYQAhIwyUiWTchi1teKU/UkQYmaeMUlVZd7lNneTaYZZSLRjnNIlhxG1PSZ9MxIRRcFylCMNiH\npIz3izyaVudF4q9pKwJ1RWAYigUfaZUWI8DYOttb13Vb66TQs3V6sCGW2bBhQ9KotQjPEEVSp85e\nQyI8Zxn6UaNGWV+IgFyUVlZ8L84//3wTWEfMeeedV0i+DCVRJlnro5BMNFFFQBHIBYFaOW/mUiJN\nZAgCOOPdc889Q+439cYLL7xgfQaaqn8ap85uU1Cp26lTp5q5c+faPT3KJBXUAWQisCaYRx991DqI\n5uFw6tatkgoXDT1XBOqPgFos6l9HmTXkQ4+jZGBG7um4lzmzEhJgZgMbaSV1hixBtURZ4IRI3cQt\nR5jT4oUXXmiWL19eCzwoC+Tm7bffNqtWrcrFuqCkItErpYEVgVogoBaLWlRDsUpgPmboYOnSpcVm\nVELqTJscPnx47Ma4BJVSZ4FlgR/DGHGEsDTe/BBIBTOmsBbEJSdx8kkbhveMHVQDPyMzZcqUjp5p\n01NSkRY5jacIVIuAWiyqxb+03Gm8MJfTCDV5nPqTn/ykmT17tpk5c2Zp2JWREfUT1++CsIGjsCUV\neVkG8i6jkJ60+impyLtGND1FoDwE1GJRHtaV5sQY/YQJE8xdd91VqR5ZMsdaga8xa2rQuLq/LOnW\nIW4SvwsWB2P449vf/nZtSeK1115rRo4cac4444zE8CqpSAyZRlAEaoWAWixqVR3FKkNDjNWCY9kO\nfnmUbP/99zcLFiwIXZiJMrmCT0kdhgdcneKc9/K7oNHFslGX4Y9uZWLIhiGRYDqqmT9/fregnWdK\nKjpQ6Iki0FgElFg0turSKY6Jmq3UGQtvkjClkVkHOG3GERo1Gl9X4g41uHGqOBfdsWK4wn2mDeMg\n2ZSpwxAFlgOn7vzyuGXjXEmFj4heKwLNRECJRTPrLbXWNE40sDfddFNozz91wgVGzKuXTjrBolGD\nNO3V2A0KXPIFVhiXDLEb7MaNG+2U0pJVyZQd02EXL17cdd0Rv6yZMtTIioAiUCkCSiwqhb+azPmI\nMyTShCmbRffS/SEUprLWaZgIMiTOtujW1CnDWC1458IW0KIO6kzwqvkv1VwVgeYioMSiuXWXSXOG\nFu68806zbt26TsOVKcGCItMgMX0RZ8AyBB8HGm9XXKuBe7+sc3T62te+Zvbaa6/Yvgpl6RY3HyGz\n/qwkJRVxEdRwikBzEFBi0Zy6yl3TrFMCc1fIS7Au+vlDKGU7hkIssFagx+jRoz2UmnN54oknmsMP\nP7xjtVBS0Zy6U00VgSQIKLFIglYLw9al8XahZfjjsssuq+06DejnO4YWOYRCHSFlWW3cusjzHCLB\nfibs8aKkIk9kNS1FoF4IKLGoV31Uog0N15o1a+yiS1VP0aTRZooiknZxpSpADBtCyctvANLSBH+Y\nOLgzZfiss84y5557bpzgGkYRUAQaiMD7G6izqpwzAvSE8bnAn6HK2SL0YnHwmzFjhl2vQpwWcy5u\nIcnh8Ok7fVIeV9IMobCTK2SvasLnliPL+bRp0+xeIlnS0LiKgCJQbwTUYlHv+ilVO2nYWdny8ssv\nH9JQFqUMVopvfvOb5pZbbjELFy5szBoNSfEIG0Lp5RiKNWnnnXdurNOmjxF+IhBY30HWD6fXioAi\n0FwElFg0t+4K0ZzGD/8GloyeN2+eXZK5SMsBy3ST36hRo6zVxO/1F1LIGiXqO4aimjuEwtDB7bff\nPuhejdRPpUqbhnZSAaCRFIGWI6DEouUVnLZ4WC8WLVpkXnzxRUsw8OjPq9GHvDz88MN20ST0u+66\n68xRRx2VVtXWxZMhlN/97nfmc5/7XGPXroiqmDPPPNOuINqU1UOjyqH3FQFFIBwB3YQsHJe+v0uv\n+aGHHrJLMW/dutVOd6RBYBVFHBWTCmQC6wRp4GuwevVqSyiYIaCkYjCaYM/vQx/6kMEnAcGy0RbZ\nb7/9DO+UiiKgCLQTAbVYtLNecy8VxICZI4899ph5/PHHzfDhw+3wBesSIOyc6go7cO7YscO89tpr\n5pVXXrFLUdNIHn/88eboo4/Ozfrh5tm2c0gcC5g1bV+XXvWAQ+qSJUsatzR5r3Lpc0VAEfgDAjor\nRN+EWAjgZ3HCCSd09hehBw152LZtmyUQDJu4MmLECLPnnnuayZMnm3/8x39slY+AW84izyFm9O7b\nJlisVBQBRaC9CCixaG/dFlqyNk2BLBQoTXwIAjhv4rujoggoAu1EQH0s2lmvWipFoLYI4AScxk+n\ntgVSxRQBRWAQAkosBsGhF4qAIqAIKAKKgCKQBQElFlnQ07iKgCKQGAG1ViSGTCMoAo1CQIlFo6pL\nlVUEmo8Aq27KNNrml0ZLoAgoAj4CSix8RPRaEagJAizlres91KQyVA1FQBGIjYASi9hQaUBFoFwE\nxowZY7Zs2VJupiXkxoyQcePGlZCTZqEIKAJVIKDEogrUNU9FIAYCbFC2cuXKGCGbFYRF01jjREUR\nUATaiYASi3bWq5aqBQiwKBk9+zYt5021sBLr+PHjW1BDWgRFQBEIQ0CJRRgqek8RqAkCEydONE8+\n+WRNtMmuBiRp+/bthgXWVBQBRaCdCCixaGe9aqlagsCkSZPsRnAtKY556aWXzIwZM9pSHC2HIqAI\nhCCgxCIEFL2lCNQFAXZ+pZfflrUfFi9ebCBLKoqAItBeBJRYtLdutWQtQYAe/je+8Y3Gl+aHP/yh\nHQaBLKkoAopAexHQbdPbW7daspYggLXioIMOMj/5yU8MDp1NlRNPPNEcfvjh5rzzzmtqEVRvRUAR\niIGAWixigKRBFIEqEWDTrgkTJpi77rqrSjUy5Y21gvUrzjjjjEzpaGRFQBGoPwJqsah/HamGioD1\ns2BdC5bDhmg0TdRa0bQaU30VgfQIqMUiPXYaUxEoDQGmZ15++eWNHEa45557zFtvvaXWitLeFs1I\nEagWAbVYVIu/5q4IxEbg17/+tcFqsXDhQnP66afHjldlQPEPefTRR62fSJW6aN6KgCJQDgJKLMrB\nWXNRBHJBAF+FqVOnmrVr1zZikakjjzzSHHHEEWb+/Pm5lF8TUQQUgfojoEMh9a8j1VAR6CDA7JB5\n8+aZk046yWDBqLNceOGFVj0lFXWuJdVNEcgfAbVY5I+ppqgIFI4Ajfb69evNqlWrajkFFf3WrFlj\n1q1bV0v9Cq8gzUAR6GME1GLRx5WvRW8uAtdee6054IADzJQpU2pnuYBULF++3DzwwANKKpr7iqnm\nikBqBJRYpIZOIyoC1SLgkos67IDK0IxYUpriA1JtDWruikA7EVBi0c561VL1CQKQC5wjcZJ85pln\nKis1sz+wnsjwjO5eWllVaMaKQOUIKLGovApUAUUgGwI4R957773mtNNOM2eeeWbpQyOsU4FTKQQH\nS0WTlx3PVhMaWxFQBEBAiYW+B4pACxBgYy/2EkFY6+KGG24ovFRMfcVSwo6lrFOhsz8Kh1wzUAQa\ngYDOCmlENamSikB8BGjwFy1aZFe7/MpXvmJXvMzTivDII4+YZcuW2b0/mrRYV3wENaQioAhkQUCJ\nRRb0NK4iUGMEIBi33nqrWbp0qZk5c6aZPHmymThxYqqhCtJ64oknzC233GKGDx9u5s6da77whS+k\nSqvGkKlqioAikAMCSixyAFGTUATqjACOlU8//bRZvXq1WblypfWFYKrqnnvuacaOHWt22mmnIeqz\nE+mOHTvMxo0bbZxx48aZadOmmenTpzdixc8hBdIbioAiUBoCSixKg1ozUgTqgQDWh82bN1vi8Nxz\nz4UqNWLEiA7x2HvvvRu5o2powfSmIqAIFI6AEovCIdYMFAFFQBFQBBSB/kFAZ4X0T11rSRUBRUAR\nUAQUgcIRUGJROMSagSKgCCgCioAi0D8IKLHon7rWkioCioAioAgoAoUjoMSicIg1A0VAEVAEFAFF\noH8QUGLRP3WtJVUEFAFFQBFQBApHQIlF4RBrBoqAIqAIKAKKQP8goMSif+paS6oIKAKKgCKgCBSO\ngBKLwiHWDBQBRUARUAQUgf5BQIlF/9S1llQRUAQUAUVAESgcASUWhUOsGSgCioAioAgoAv2DgBKL\n/qlrLakioAgoAoqAIlA4AkosCodYM1AEFAFFQBFQBPoHASUW/VPXWlJFQBFQBBQBRaBwBJRYFA6x\nZqAIKAKKgCKgCPQPAv8fPYIkU9UI+YoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import Image\n",
    "Image(filename='sentiment_network_2.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO:** Create a numpy array called `layer_0` and initialize it to all zeros. You will find the [zeros](https://docs.scipy.org/doc/numpy/reference/generated/numpy.zeros.html) function particularly helpful here. Be sure you create `layer_0` as a 2-dimensional matrix with 1 row and `vocab_size` columns. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO: Create layer_0 matrix with dimensions 1 by vocab_size, initially filled with zeros\n",
    "layer_0 = None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell. It should display `(1, 74074)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layer_0.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "Image(filename='sentiment_network.png')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`layer_0` contains one entry for every word in the vocabulary, as shown in the above image. We need to make sure we know the index of each word, so run the following cell to create a lookup table that stores the index of every word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a dictionary of words in the vocabulary mapped to index positions\n",
    "# (to be used in layer_0)\n",
    "word2index = {}\n",
    "for i,word in enumerate(vocab):\n",
    "    word2index[word] = i\n",
    "    \n",
    "# display the map of words to indices\n",
    "word2index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO:**  Complete the implementation of `update_input_layer`. It should count \n",
    "          how many times each word is used in the given review, and then store\n",
    "          those counts at the appropriate indices inside `layer_0`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def update_input_layer(review):\n",
    "    \"\"\" Modify the global layer_0 to represent the vector form of review.\n",
    "    The element at a given index of layer_0 should represent\n",
    "    how many times the given word occurs in the review.\n",
    "    Args:\n",
    "        review(string) - the string of the review\n",
    "    Returns:\n",
    "        None\n",
    "    \"\"\"\n",
    "    global layer_0\n",
    "    # clear out previous state by resetting the layer to be all 0s\n",
    "    layer_0 *= 0\n",
    "    \n",
    "    # TODO: count how many times each word is used in the given review and store the results in layer_0 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to test updating the input layer with the first review. The indices assigned may not be the same as in the solution, but hopefully you'll see some non-zero values in `layer_0`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "update_input_layer(reviews[0])\n",
    "layer_0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO:** Complete the implementation of `get_target_for_labels`. It should return `0` or `1`, \n",
    "          depending on whether the given label is `NEGATIVE` or `POSITIVE`, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_target_for_label(label):\n",
    "    \"\"\"Convert a label to `0` or `1`.\n",
    "    Args:\n",
    "        label(string) - Either \"POSITIVE\" or \"NEGATIVE\".\n",
    "    Returns:\n",
    "        `0` or `1`.\n",
    "    \"\"\"\n",
    "    # TODO: Your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following two cells. They should print out`'POSITIVE'` and `1`, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "labels[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "get_target_for_label(labels[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following two cells. They should print out `'NEGATIVE'` and `0`, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "labels[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "get_target_for_label(labels[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End of Project 2. \n",
    "## Watch the next video to see Andrew's solution, then continue on to the next lesson."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project 3: Building a Neural Network<a id='project_3'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**TODO:** We've included the framework of a class called `SentimentNetork`. Implement all of the items marked `TODO` in the code. These include doing the following:\n",
    "- Create a basic neural network much like the networks you've seen in earlier lessons and in Project 1, with an input layer, a hidden layer, and an output layer. \n",
    "- Do **not** add a non-linearity in the hidden layer. That is, do not use an activation function when calculating the hidden layer outputs.\n",
    "- Re-use the code from earlier in this notebook to create the training data (see `TODO`s in the code)\n",
    "- Implement the `pre_process_data` function to create the vocabulary for our training data generating functions\n",
    "- Ensure `train` trains over the entire corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Where to Get Help if You Need it\n",
    "- Re-watch earlier Udacity lectures\n",
    "- Chapters 3-5 - [Grokking Deep Learning](https://www.manning.com/books/grokking-deep-learning) - (Check inside your classroom for a discount code)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import time\n",
    "import sys\n",
    "import numpy as np\n",
    "\n",
    "# Encapsulate our neural network in a class\n",
    "class SentimentNetwork:\n",
    "    def __init__(self, reviews, labels, hidden_nodes = 10, learning_rate = 0.1):\n",
    "        \"\"\"Create a SentimenNetwork with the given settings\n",
    "        Args:\n",
    "            reviews(list) - List of reviews used for training\n",
    "            labels(list) - List of POSITIVE/NEGATIVE labels associated with the given reviews\n",
    "            hidden_nodes(int) - Number of nodes to create in the hidden layer\n",
    "            learning_rate(float) - Learning rate to use while training\n",
    "        \n",
    "        \"\"\"\n",
    "        # Assign a seed to our random number generator to ensure we get\n",
    "        # reproducable results during development \n",
    "        np.random.seed(1)\n",
    "\n",
    "        # process the reviews and their associated labels so that everything\n",
    "        # is ready for training\n",
    "        self.pre_process_data(reviews, labels)\n",
    "        \n",
    "        # Build the network to have the number of hidden nodes and the learning rate that\n",
    "        # were passed into this initializer. Make the same number of input nodes as\n",
    "        # there are vocabulary words and create a single output node.\n",
    "        self.init_network(len(self.review_vocab),hidden_nodes, 1, learning_rate)\n",
    "\n",
    "    def pre_process_data(self, reviews, labels):\n",
    "        \n",
    "        review_vocab = set()\n",
    "        # TODO: populate review_vocab with all of the words in the given reviews\n",
    "        #       Remember to split reviews into individual words \n",
    "        #       using \"split(' ')\" instead of \"split()\".\n",
    "        \n",
    "        # Convert the vocabulary set to a list so we can access words via indices\n",
    "        self.review_vocab = list(review_vocab)\n",
    "        \n",
    "        label_vocab = set()\n",
    "        # TODO: populate label_vocab with all of the words in the given labels.\n",
    "        #       There is no need to split the labels because each one is a single word.\n",
    "        \n",
    "        # Convert the label vocabulary set to a list so we can access labels via indices\n",
    "        self.label_vocab = list(label_vocab)\n",
    "        \n",
    "        # Store the sizes of the review and label vocabularies.\n",
    "        self.review_vocab_size = len(self.review_vocab)\n",
    "        self.label_vocab_size = len(self.label_vocab)\n",
    "        \n",
    "        # Create a dictionary of words in the vocabulary mapped to index positions\n",
    "        self.word2index = {}\n",
    "        # TODO: populate self.word2index with indices for all the words in self.review_vocab\n",
    "        #       like you saw earlier in the notebook\n",
    "        \n",
    "        # Create a dictionary of labels mapped to index positions\n",
    "        self.label2index = {}\n",
    "        # TODO: do the same thing you did for self.word2index and self.review_vocab, \n",
    "        #       but for self.label2index and self.label_vocab instead\n",
    "         \n",
    "        \n",
    "    def init_network(self, input_nodes, hidden_nodes, output_nodes, learning_rate):\n",
    "        # Store the number of nodes in input, hidden, and output layers.\n",
    "        self.input_nodes = input_nodes\n",
    "        self.hidden_nodes = hidden_nodes\n",
    "        self.output_nodes = output_nodes\n",
    "\n",
    "        # Store the learning rate\n",
    "        self.learning_rate = learning_rate\n",
    "\n",
    "        # Initialize weights\n",
    "        \n",
    "        # TODO: initialize self.weights_0_1 as a matrix of zeros. These are the weights between\n",
    "        #       the input layer and the hidden layer.\n",
    "        self.weights_0_1 = None\n",
    "        \n",
    "        # TODO: initialize self.weights_1_2 as a matrix of random values. \n",
    "        #       These are the weights between the hidden layer and the output layer.\n",
    "        self.weights_1_2 = None\n",
    "        \n",
    "        # TODO: Create the input layer, a two-dimensional matrix with shape \n",
    "        #       1 x input_nodes, with all values initialized to zero\n",
    "        self.layer_0 = None\n",
    "    \n",
    "        \n",
    "    def update_input_layer(self,review):\n",
    "        # TODO: You can copy most of the code you wrote for update_input_layer \n",
    "        #       earlier in this notebook. \n",
    "        #\n",
    "        #       However, MAKE SURE YOU CHANGE ALL VARIABLES TO REFERENCE\n",
    "        #       THE VERSIONS STORED IN THIS OBJECT, NOT THE GLOBAL OBJECTS.\n",
    "        #       For example, replace \"layer_0 *= 0\" with \"self.layer_0 *= 0\"\n",
    "        pass\n",
    "                \n",
    "    def get_target_for_label(self,label):\n",
    "        # TODO: Copy the code you wrote for get_target_for_label \n",
    "        #       earlier in this notebook. \n",
    "        pass\n",
    "        \n",
    "    def sigmoid(self,x):\n",
    "        # TODO: Return the result of calculating the sigmoid activation function\n",
    "        #       shown in the lectures\n",
    "        pass\n",
    "    \n",
    "    def sigmoid_output_2_derivative(self,output):\n",
    "        # TODO: Return the derivative of the sigmoid activation function, \n",
    "        #       where \"output\" is the original output from the sigmoid function \n",
    "        pass\n",
    "\n",
    "    def train(self, training_reviews, training_labels):\n",
    "        \n",
    "        # make sure we have a matching number of reviews and labels\n",
    "        assert(len(training_reviews) == len(training_labels))\n",
    "        \n",
    "        # Keep track of correct predictions to display accuracy during training \n",
    "        correct_so_far = 0\n",
    "        \n",
    "        # Remember when we started for printing time statistics\n",
    "        start = time.time()\n",
    "\n",
    "        # loop through all the given reviews and run a forward and backward pass,\n",
    "        # updating weights for every item\n",
    "        for i in range(len(training_reviews)):\n",
    "            \n",
    "            # TODO: Get the next review and its correct label\n",
    "            \n",
    "            # TODO: Implement the forward pass through the network. \n",
    "            #       That means use the given review to update the input layer, \n",
    "            #       then calculate values for the hidden layer,\n",
    "            #       and finally calculate the output layer.\n",
    "            # \n",
    "            #       Do not use an activation function for the hidden layer,\n",
    "            #       but use the sigmoid activation function for the output layer.\n",
    "            \n",
    "            # TODO: Implement the back propagation pass here. \n",
    "            #       That means calculate the error for the forward pass's prediction\n",
    "            #       and update the weights in the network according to their\n",
    "            #       contributions toward the error, as calculated via the\n",
    "            #       gradient descent and back propagation algorithms you \n",
    "            #       learned in class.\n",
    "            \n",
    "            # TODO: Keep track of correct predictions. To determine if the prediction was\n",
    "            #       correct, check that the absolute value of the output error \n",
    "            #       is less than 0.5. If so, add one to the correct_so_far count.\n",
    "            \n",
    "            # For debug purposes, print out our prediction accuracy and speed \n",
    "            # throughout the training process. \n",
    "\n",
    "            elapsed_time = float(time.time() - start)\n",
    "            reviews_per_second = i / elapsed_time if elapsed_time > 0 else 0\n",
    "            \n",
    "            sys.stdout.write(\"\\rProgress:\" + str(100 * i/float(len(training_reviews)))[:4] \\\n",
    "                             + \"% Speed(reviews/sec):\" + str(reviews_per_second)[0:5] \\\n",
    "                             + \" #Correct:\" + str(correct_so_far) + \" #Trained:\" + str(i+1) \\\n",
    "                             + \" Training Accuracy:\" + str(correct_so_far * 100 / float(i+1))[:4] + \"%\")\n",
    "            if(i % 2500 == 0):\n",
    "                print(\"\")\n",
    "    \n",
    "    def test(self, testing_reviews, testing_labels):\n",
    "        \"\"\"\n",
    "        Attempts to predict the labels for the given testing_reviews,\n",
    "        and uses the test_labels to calculate the accuracy of those predictions.\n",
    "        \"\"\"\n",
    "        \n",
    "        # keep track of how many correct predictions we make\n",
    "        correct = 0\n",
    "\n",
    "        # we'll time how many predictions per second we make\n",
    "        start = time.time()\n",
    "\n",
    "        # Loop through each of the given reviews and call run to predict\n",
    "        # its label. \n",
    "        for i in range(len(testing_reviews)):\n",
    "            pred = self.run(testing_reviews[i])\n",
    "            if(pred == testing_labels[i]):\n",
    "                correct += 1\n",
    "            \n",
    "            # For debug purposes, print out our prediction accuracy and speed \n",
    "            # throughout the prediction process. \n",
    "\n",
    "            elapsed_time = float(time.time() - start)\n",
    "            reviews_per_second = i / elapsed_time if elapsed_time > 0 else 0\n",
    "            \n",
    "            sys.stdout.write(\"\\rProgress:\" + str(100 * i/float(len(testing_reviews)))[:4] \\\n",
    "                             + \"% Speed(reviews/sec):\" + str(reviews_per_second)[0:5] \\\n",
    "                             + \" #Correct:\" + str(correct) + \" #Tested:\" + str(i+1) \\\n",
    "                             + \" Testing Accuracy:\" + str(correct * 100 / float(i+1))[:4] + \"%\")\n",
    "    \n",
    "    def run(self, review):\n",
    "        \"\"\"\n",
    "        Returns a POSITIVE or NEGATIVE prediction for the given review.\n",
    "        \"\"\"\n",
    "        # TODO: Run a forward pass through the network, like you did in the\n",
    "        #       \"train\" function. That means use the given review to \n",
    "        #       update the input layer, then calculate values for the hidden layer,\n",
    "        #       and finally calculate the output layer.\n",
    "        #\n",
    "        #       Note: The review passed into this function for prediction \n",
    "        #             might come from anywhere, so you should convert it \n",
    "        #             to lower case prior to using it.\n",
    "        \n",
    "        # TODO: The output layer should now contain a prediction. \n",
    "        #       Return `POSITIVE` for predictions greater-than-or-equal-to `0.5`, \n",
    "        #       and `NEGATIVE` otherwise.\n",
    "        pass\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to create a `SentimentNetwork` that will train on all but the last 1000 reviews (we're saving those for testing). Here we use a learning rate of `0.1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp = SentimentNetwork(reviews[:-1000],labels[:-1000], learning_rate=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to test the network's performance against the last 1000 reviews (the ones we held out from our training set). \n",
    "\n",
    "**We have not trained the model yet, so the results should be about 50% as it will just be guessing and there are only two possible values to choose from.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp.test(reviews[-1000:],labels[-1000:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to actually train the network. During training, it will display the model's accuracy repeatedly as it trains so you can see how well it's doing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp.train(reviews[:-1000],labels[:-1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That most likely didn't train very well. Part of the reason may be because the learning rate is too high. Run the following cell to recreate the network with a smaller learning rate, `0.01`, and then train the new network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp = SentimentNetwork(reviews[:-1000],labels[:-1000], learning_rate=0.01)\n",
    "mlp.train(reviews[:-1000],labels[:-1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That probably wasn't much different. Run the following cell to recreate the network one more time with an even smaller learning rate, `0.001`, and then train the new network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp = SentimentNetwork(reviews[:-1000],labels[:-1000], learning_rate=0.001)\n",
    "mlp.train(reviews[:-1000],labels[:-1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a learning rate of `0.001`, the network should finally have started to improve during training. It's still not very good, but it shows that this solution has potential. We will improve it in the next lesson."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End of Project 3. \n",
    "## Watch the next video to see Andrew's solution, then continue on to the next lesson."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Understanding Neural Noise<a id='lesson_4'></a>\n",
    "\n",
    "The following cells include includes the code Andrew shows in the next video. We've included it here so you can run the cells along with the video without having to type in everything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "Image(filename='sentiment_network.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def update_input_layer(review):\n",
    "    \n",
    "    global layer_0\n",
    "    \n",
    "    # clear out previous state, reset the layer to be all 0s\n",
    "    layer_0 *= 0\n",
    "    for word in review.split(\" \"):\n",
    "        layer_0[0][word2index[word]] += 1\n",
    "\n",
    "update_input_layer(reviews[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layer_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "review_counter = Counter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for word in reviews[0].split(\" \"):\n",
    "    review_counter[word] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "review_counter.most_common()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project 4: Reducing Noise in Our Input Data<a id='project_4'></a>\n",
    "\n",
    "**TODO:** Attempt to reduce the noise in the input data like Andrew did in the previous video. Specifically, do the following:\n",
    "* Copy the `SentimentNetwork` class you created earlier into the following cell.\n",
    "* Modify `update_input_layer` so it does not count how many times each word is used, but rather just stores whether or not a word was used. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO: -Copy the SentimentNetwork class from Projet 3 lesson\n",
    "#       -Modify it to reduce noise, like in the video "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to recreate the network and train it. Notice we've gone back to the higher learning rate of `0.1`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp = SentimentNetwork(reviews[:-1000],labels[:-1000], learning_rate=0.1)\n",
    "mlp.train(reviews[:-1000],labels[:-1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That should have trained much better than the earlier attempts. It's still not wonderful, but it should have improved dramatically. Run the following cell to test your model with 1000 predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp.test(reviews[-1000:],labels[-1000:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End of Project 4. \n",
    "## Andrew's solution was actually in the previous video, so rewatch that video if you had any problems with that project. Then continue on to the next lesson.\n",
    "# Analyzing Inefficiencies in our Network<a id='lesson_5'></a>\n",
    "The following cells include the code Andrew shows in the next video. We've included it here so you can run the cells along with the video without having to type in everything."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Image(filename='sentiment_network_sparse.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layer_0 = np.zeros(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layer_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layer_0[4] = 1\n",
    "layer_0[9] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layer_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "weights_0_1 = np.random.randn(10,5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layer_0.dot(weights_0_1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "indices = [4,9]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layer_1 = np.zeros(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for index in indices:\n",
    "    layer_1 += (1 * weights_0_1[index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layer_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Image(filename='sentiment_network_sparse_2.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layer_1 = np.zeros(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for index in indices:\n",
    "    layer_1 += (weights_0_1[index])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "layer_1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project 5: Making our Network More Efficient<a id='project_5'></a>\n",
    "**TODO:** Make the `SentimentNetwork` class more efficient by eliminating unnecessary multiplications and additions that occur during forward and backward propagation. To do that, you can do the following:\n",
    "* Copy the `SentimentNetwork` class from the previous project into the following cell.\n",
    "* Remove the `update_input_layer` function - you will not need it in this version.\n",
    "* Modify `init_network`:\n",
    ">* You no longer need a separate input layer, so remove any mention of `self.layer_0`\n",
    ">* You will be dealing with the old hidden layer more directly, so create `self.layer_1`, a two-dimensional matrix with shape 1 x hidden_nodes, with all values initialized to zero\n",
    "* Modify `train`:\n",
    ">* Change the name of the input parameter `training_reviews` to `training_reviews_raw`. This will help with the next step.\n",
    ">* At the beginning of the function, you'll want to preprocess your reviews to convert them to a list of indices (from `word2index`) that are actually used in the review. This is equivalent to what you saw in the video when Andrew set specific indices to 1. Your code should create a local `list` variable named `training_reviews` that should contain a `list` for each review in `training_reviews_raw`. Those lists should contain the indices for words found in the review.\n",
    ">* Remove call to `update_input_layer`\n",
    ">* Use `self`'s  `layer_1` instead of a local `layer_1` object.\n",
    ">* In the forward pass, replace the code that updates `layer_1` with new logic that only adds the weights for the indices used in the review.\n",
    ">* When updating `weights_0_1`, only update the individual weights that were used in the forward pass.\n",
    "* Modify `run`:\n",
    ">* Remove call to `update_input_layer` \n",
    ">* Use `self`'s  `layer_1` instead of a local `layer_1` object.\n",
    ">* Much like you did in `train`, you will need to pre-process the `review` so you can work with word indices, then update `layer_1` by adding weights for the indices used in the review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO: -Copy the SentimentNetwork class from Project 4 lesson\n",
    "#       -Modify it according to the above instructions "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to recreate the network and train it once again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp = SentimentNetwork(reviews[:-1000],labels[:-1000], learning_rate=0.1)\n",
    "mlp.train(reviews[:-1000],labels[:-1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That should have trained much better than the earlier attempts. Run the following cell to test your model with 1000 predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp.test(reviews[-1000:],labels[-1000:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End of Project 5. \n",
    "## Watch the next video to see Andrew's solution, then continue on to the next lesson.\n",
    "# Further Noise Reduction<a id='lesson_6'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Image(filename='sentiment_network_sparse_2.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# words most frequently seen in a review with a \"POSITIVE\" label\n",
    "pos_neg_ratios.most_common()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# words most frequently seen in a review with a \"NEGATIVE\" label\n",
    "list(reversed(pos_neg_ratios.most_common()))[0:30]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from bokeh.models import ColumnDataSource, LabelSet\n",
    "from bokeh.plotting import figure, show, output_file\n",
    "from bokeh.io import output_notebook\n",
    "output_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hist, edges = np.histogram(list(map(lambda x:x[1],pos_neg_ratios.most_common())), density=True, bins=100, normed=True)\n",
    "\n",
    "p = figure(tools=\"pan,wheel_zoom,reset,save\",\n",
    "           toolbar_location=\"above\",\n",
    "           title=\"Word Positive/Negative Affinity Distribution\")\n",
    "p.quad(top=hist, bottom=0, left=edges[:-1], right=edges[1:], line_color=\"#555555\")\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "frequency_frequency = Counter()\n",
    "\n",
    "for word, cnt in total_counts.most_common():\n",
    "    frequency_frequency[cnt] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "hist, edges = np.histogram(list(map(lambda x:x[1],frequency_frequency.most_common())), density=True, bins=100, normed=True)\n",
    "\n",
    "p = figure(tools=\"pan,wheel_zoom,reset,save\",\n",
    "           toolbar_location=\"above\",\n",
    "           title=\"The frequency distribution of the words in our corpus\")\n",
    "p.quad(top=hist, bottom=0, left=edges[:-1], right=edges[1:], line_color=\"#555555\")\n",
    "show(p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project 6: Reducing Noise by Strategically Reducing the Vocabulary<a id='project_6'></a>\n",
    "\n",
    "**TODO:** Improve `SentimentNetwork`'s performance by reducing more noise in the vocabulary. Specifically, do the following:\n",
    "* Copy the `SentimentNetwork` class from the previous project into the following cell.\n",
    "* Modify `pre_process_data`:\n",
    ">* Add two additional parameters: `min_count` and `polarity_cutoff`\n",
    ">* Calculate the positive-to-negative ratios of words used in the reviews. (You can use code you've written elsewhere in the notebook, but we are moving it into the class like we did with other helper code earlier.)\n",
    ">* Andrew's solution only calculates a postive-to-negative ratio for words that occur at least 50 times. This keeps the network from attributing too much sentiment to rarer words. You can choose to add this to your solution if you would like.  \n",
    ">* Change so words are only added to the vocabulary if they occur in the vocabulary more than `min_count` times.\n",
    ">* Change so words are only added to the vocabulary if the absolute value of their postive-to-negative ratio is at least `polarity_cutoff`\n",
    "* Modify `__init__`:\n",
    ">* Add the same two parameters (`min_count` and `polarity_cutoff`) and use them when you call `pre_process_data`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# TODO: -Copy the SentimentNetwork class from Project 5 lesson\n",
    "#       -Modify it according to the above instructions "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to train your network with a small polarity cutoff."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp = SentimentNetwork(reviews[:-1000],labels[:-1000],min_count=20,polarity_cutoff=0.05,learning_rate=0.01)\n",
    "mlp.train(reviews[:-1000],labels[:-1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And run the following cell to test it's performance. It should be "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp.test(reviews[-1000:],labels[-1000:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following cell to train your network with a much larger polarity cutoff."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp = SentimentNetwork(reviews[:-1000],labels[:-1000],min_count=20,polarity_cutoff=0.8,learning_rate=0.01)\n",
    "mlp.train(reviews[:-1000],labels[:-1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And run the following cell to test it's performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp.test(reviews[-1000:],labels[-1000:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End of Project 6. \n",
    "## Watch the next video to see Andrew's solution, then continue on to the next lesson."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Analysis: What's Going on in the Weights?<a id='lesson_7'></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp_full = SentimentNetwork(reviews[:-1000],labels[:-1000],min_count=0,polarity_cutoff=0,learning_rate=0.01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mlp_full.train(reviews[:-1000],labels[:-1000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Image(filename='sentiment_network_sparse.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_most_similar_words(focus = \"horrible\"):\n",
    "    most_similar = Counter()\n",
    "\n",
    "    for word in mlp_full.word2index.keys():\n",
    "        most_similar[word] = np.dot(mlp_full.weights_0_1[mlp_full.word2index[word]],mlp_full.weights_0_1[mlp_full.word2index[focus]])\n",
    "    \n",
    "    return most_similar.most_common()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "get_most_similar_words(\"excellent\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "get_most_similar_words(\"terrible\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.colors as colors\n",
    "\n",
    "words_to_visualize = list()\n",
    "for word, ratio in pos_neg_ratios.most_common(500):\n",
    "    if(word in mlp_full.word2index.keys()):\n",
    "        words_to_visualize.append(word)\n",
    "    \n",
    "for word, ratio in list(reversed(pos_neg_ratios.most_common()))[0:500]:\n",
    "    if(word in mlp_full.word2index.keys()):\n",
    "        words_to_visualize.append(word)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pos = 0\n",
    "neg = 0\n",
    "\n",
    "colors_list = list()\n",
    "vectors_list = list()\n",
    "for word in words_to_visualize:\n",
    "    if word in pos_neg_ratios.keys():\n",
    "        vectors_list.append(mlp_full.weights_0_1[mlp_full.word2index[word]])\n",
    "        if(pos_neg_ratios[word] > 0):\n",
    "            pos+=1\n",
    "            colors_list.append(\"#00ff00\")\n",
    "        else:\n",
    "            neg+=1\n",
    "            colors_list.append(\"#000000\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.manifold import TSNE\n",
    "tsne = TSNE(n_components=2, random_state=0)\n",
    "words_top_ted_tsne = tsne.fit_transform(vectors_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "p = figure(tools=\"pan,wheel_zoom,reset,save\",\n",
    "           toolbar_location=\"above\",\n",
    "           title=\"vector T-SNE for most polarized words\")\n",
    "\n",
    "source = ColumnDataSource(data=dict(x1=words_top_ted_tsne[:,0],\n",
    "                                    x2=words_top_ted_tsne[:,1],\n",
    "                                    names=words_to_visualize,\n",
    "                                    color=colors_list))\n",
    "\n",
    "p.scatter(x=\"x1\", y=\"x2\", size=8, source=source, fill_color=\"color\")\n",
    "\n",
    "word_labels = LabelSet(x=\"x1\", y=\"x2\", text=\"names\", y_offset=6,\n",
    "                  text_font_size=\"8pt\", text_color=\"#555555\",\n",
    "                  source=source, text_align='center')\n",
    "p.add_layout(word_labels)\n",
    "\n",
    "show(p)\n",
    "\n",
    "# green indicates positive words, black indicates negative words"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}