NaiveBayes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why to Read this Tutorial?\n",
    "\n",
    "* In this tutorial, you will learn how to train a Naive Bayes classifier to make it discriminate positive from negative reviews. The used data is taken from [rotten tomatoes](http://www.rottentomatoes.com/), a movie and TV reviews site.\n",
    "* It will introduce some basic concepts of text representation that are also used in information retrieval. In principle, you can transfer the described ideas to other use cases, e.g., text similarity or image similarity search.\n",
    "* The tutorial will summarize some concepts of probability theory that are needed to understand the theory behind the Naive Bayes classifier.\n",
    "* At the end of this tutorial, you will be able to input an arbitrary text to the classifier and the classifier will tell you whether it is positive or not. However, the classifier is not limited to decisions between two classes. You can train it to as many classes you want to..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparation\n",
    "\n",
    "The following code snippet initializes your Python run-time enviroment in order to run all of the subsequent actions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# The %... is an iPython thing, and is not part of the Python language.\n",
    "# In this case we're just telling the plotting library to draw things on\n",
    "# the notebook, instead of on a separate window.\n",
    "%matplotlib inline\n",
    "# See all the \"as ...\" contructs? They're just aliasing the package names.\n",
    "# That way we can call methods like plt.plot() instead of matplotlib.pyplot.plot().\n",
    "import numpy as np\n",
    "import scipy as sp\n",
    "import matplotlib as mpl\n",
    "import matplotlib.cm as cm\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import time\n",
    "pd.set_option('display.width', 500)\n",
    "pd.set_option('display.max_columns', 100)\n",
    "pd.set_option('display.notebook_repr_html', True)\n",
    "import seaborn as sns\n",
    "sns.set_style(\"whitegrid\")\n",
    "sns.set_context(\"poster\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Brief Introduction to the Vector Space Model\n",
    "\n",
    "See http://nlp.stanford.edu/IR-book/ from which most of this text is taken. The text has been augmented with hyperlinks to information retrieval (IR) or machine learning (ML) concepts for the interested reader. Text parts in italics have been taken from http://nbn-resolving.de/urn/resolver.pl?urn:nbn:de:kobv:co1-opus4-35218.\n",
    "\n",
    "_The vector space model (VSM) is the most prominent algebraic retrieval model in IR. It is called algebraic because both queries and document representations are modeled in an $t$-dimensional vector space. The dimensionality $t$ is determined by the number of index terms, i.e., $t=|K|$. The determination of the similarity between a query vector and a document representation vector is solved by using methods from linear algebra, e.g., the cosine of the angle between both vectors, i.e., the cosine similarity that we will use below._\n",
    "\n",
    "_In traditional IR, the index vocabulary $\\mathbf{K}$ consists of index terms or keywords that can be used during retrieval. Only a keyword $\\mathbf{k_{j}}$ out of $\\mathbf{K}~(\\mathbf{k_{j}}\\in \\mathbf{K})$ can be used for the representation of a document. In order to indicate whether a specific index term $\\mathbf{k_{j}}$ is present in a document representation $\\mathbf{d_{i}}$, an index term weight $\\mathbf{w_{i,j}} \\geq 0$ is used. If $\\mathbf{w_{i,j}} = 0$ then $\\mathbf{d_{i}}$ does not contain $\\mathbf{k_{j}}$. Hence, a document representation has the following (vector) form: $\\mathbf{d_{i}}=(\\mathbf{w_{i,1}},\\mathbf{w_{i,2}},\\dots,\\mathbf{w_{i,j}})$._\n",
    "\n",
    "In other words, there is one axis for each index term in the vector space containing all document vectors.\n",
    "\n",
    ">This representation loses the relative ordering of the terms in each document. The documents *Mary is quicker than John* and *John is quicker than Mary* are identical in such a *[bag of words](https://en.wikipedia.org/wiki/Bag-of-words_model)* representation. the standard way of quantifying the similarity between two documents $d_1$ and $d_2$  is to compute the *[cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity)* of their vector representations $\\bar V(d_1)$ and $\\bar V(d_2)$:\n",
    "\n",
    "$$sim_{d_1,d_2} = \\frac{\\bar V(d_1) \\cdot \\bar V(d_2)}{|\\bar V(d_1)| |\\bar V(d_2)|}$$\n",
    "\n",
    ">The formula can be viewed as the [dot product](https://en.wikipedia.org/wiki/Dot_product) of the normalized versions of the two document vectors (The normalization happens in the denominator, right below the line of the fraction). \n",
    "\n",
    "This equation equals to the following if we transform the vector form into a simple algebraic evaluation:\n",
    "\n",
    "$$sim(d_{1},d_2)=\\frac{\\sum^{t}_{j=1}d_{1,j}\\cdot d_{2,j}}{\\sqrt{\\sum^{t}_{j=1}d^{2}_{1,j}\\cdot \\sum^{t}_{j=1}d^{2}_{2,j}}}$$\n",
    "\n",
    "_Obviously, $sim(d_{1},d_{2})\\rightarrow[0,1]$ holds (because of the normalization), where $1$ denotes a full match or the highest degree of similarity. This value is used to sort the retrieved documents in decreasing order to present a ranking to the user that also includes partial matches, e.g., documents that do not contain all index terms of the query. The index term weights can be calculated, e.g., by counting the occurrence of terms in a document (as we will do below) or by using the well-known [$tf\\ast idf$ formula](https://en.wikipedia.org/wiki/Tf%E2%80%93idf). The $tf\\ast idf$ formula assigns each $w_{i,j}$ with a product of the term frequency ($tf$) of $k_{j}$ in $d_{i}$ and the inverse of frequency ($idf$) of $k_{j}$ in the collection . The $idf$ expresses how rare an index term is within the collection._\n",
    "\n",
    ">Viewing a collection of $N$ documents as a collection of vectors leads to a TERM-DOCUMENT natural view of a collection as a term-document matrix: this is an $|K| \\times N$ matrix whose rows represent the $|K|$ terms (dimensions) of the $N$ columns, each of which corresponds to a document.\n",
    "\n",
    "![novel terms](terms.png)\n",
    "\n",
    ">There is a far more compelling reason to represent documents as vectors: we can also view a query as a vector. Consider the query q = _jealous gossip_. This query turns into the unit vector $\\bar V(q) = (0, 0.707, 0.707)$ on the three coordinates below. \n",
    "\n",
    "![novel terms](terms2.png)\n",
    "\n",
    ">The key idea now: to assign to each document $d_i$ a score equal to the dot product:\n",
    "\n",
    "$$\\bar V(q) \\cdot \\bar V(d_i)$$\n",
    "\n",
    "Then, you repeat this calculation for all three documents.\n",
    "\n",
    "You can visualize the outcome of this equation as follows. Basically, you are trying to measure the angle between the documents and the query to determine the relevance (or the similarity) of a document with respect to the query. \n",
    "![Vector Space Model](vsm.png)\n",
    "\n",
    ">\"Wuthering Heights\" (WH) is the top-scoring document for this query with a score of $0.509$, with \"Pride and Prejudice\" (PaP) a distant second with a score of $0.085$, and \"Sense and Sensibility\" (SaS) last with a score of $0.074$. This simple example is somewhat misleading: the number of dimensions in practice will be far larger than three: it will equal the vocabulary size $|K|$.\n",
    "\n",
    "### Working with the Vector Space Model in Python\n",
    "\n",
    "In this example, you will use the [CountVectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html) of the *scikit-learn* package (a ML package for Python). The *CountVectorizer* creates document vector with index term weights that reflect the frequency of a terms occurence in a document as the following example illustrates:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original documents are\n",
      "Hop on pop\n",
      "Hop off pop\n",
      "Hop Hop hop\n",
      "The vocabulary's size is: 4\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# create three different documents\n",
    "documents = ['Hop on pop', 'Hop off pop', 'Hop Hop hop']\n",
    "print \"Original documents are\\n\", '\\n'.join(documents)\n",
    "\n",
    "vectorizer = CountVectorizer(min_df=0)\n",
    "\n",
    "# call `fit` to build the vocabulary from the given documents\n",
    "vectorizer.fit(documents)\n",
    "# as there are four different terms in the documents, the vocabulary's size should be four as well\n",
    "# the str() \"casts\" the number returned by len() to a string in order to display it\n",
    "print \"The vocabulary's size is: \"+str(len(vectorizer.vocabulary_.keys())) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original documents are\n",
      "Hop on pop\n",
      "Hop off pop\n",
      "Hop Hop hop\n",
      "\n",
      "Index term weights for each document:\n",
      "Transformed text vector is \n",
      "[[1 0 1 1]\n",
      " [1 1 0 1]\n",
      " [3 0 0 0]]\n",
      "\n",
      "Words for each feature:\n",
      "[u'hop', u'off', u'on', u'pop']\n"
     ]
    }
   ],
   "source": [
    "# call `transform` to convert text to a bag of words\n",
    "x = vectorizer.transform(documents)\n",
    "\n",
    "# CountVectorizer uses a sparse array to save memory, but it's easier in this assignment to \n",
    "# convert back to a \"normal\" NumPy array\n",
    "x = x.toarray()\n",
    "\n",
    "print \"Original documents are\\n\", '\\n'.join(documents)\n",
    "print \"\\nIndex term weights for each document:\"\n",
    "print \"Transformed text vector is \\n\", x\n",
    "\n",
    "# `get_feature_names` tracks which word is associated with each column of the transformed x\n",
    "print\n",
    "print \"Words for each feature:\"\n",
    "print vectorizer.get_feature_names()\n",
    "\n",
    "# Notice that the bag of words treatment doesn't preserve information about the *order* of words, \n",
    "# just their frequency"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is easy to see that the last row represents the document *Hop Hop hop* as it has the value $3$ for the first index term weight associated with the term *hop*. Note that the *CountVectorizer* converts all characters to lowercase by default.\n",
    "\n",
    "Now, we will check which document is most relevant for the query *hop on*. For the sake of brevity, we will directly input the query in vector format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.57735027  0.          0.57735027  0.57735027]\n",
      " [ 0.57735027  0.57735027  0.          0.57735027]\n",
      " [ 1.          0.          0.          0.        ]]\n",
      "\n",
      "The most similar document has a similarity of: 0.816496580928\n",
      "It is: Hop on pop[ 0.57735027  0.          0.57735027  0.57735027]\n"
     ]
    }
   ],
   "source": [
    "from scipy.spatial.distance import cosine\n",
    "from sklearn.preprocessing import normalize\n",
    "\n",
    "query=np.array([1,0,1,0]).astype(float) # we have to use a NumPy floating point array\n",
    "x=x.astype(float) # for the documents, we need floating points as well as we want to normalize later\n",
    "x=normalize(x)\n",
    "print x\n",
    "# this variables is set to the maximum distance to find the best, i.e., the nearest, document to the query \n",
    "dist=1\n",
    "documentIndex=0\n",
    "for index,document in enumerate(x):\n",
    "    currentDist=cosine(query,document)\n",
    "    if currentDist<= dist:\n",
    "        dist=currentDist\n",
    "        documentIndex=index\n",
    "# we have to take 1-dist, because we have calculated a distance above but want to display a similarity\n",
    "# a distance is the logical counterpart of a distance and as both values are in the range of 0 and 1, we can simply invert\n",
    "# them to yield the logical counterpart\n",
    "print \n",
    "print \"The most similar document has a similarity of: \" + str(1-dist) \n",
    "print \"It is: \"+str(documents[documentIndex])+str(x[documentIndex])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Probability Theory\n",
    "_This informal introduction of probability theory is oriented on [ASH, Robert B.: Basic Probability Theory. Dover ed. Mineola, NY : Dover Publ., 2008]. It aims at summarizing classical probability theory being based on Kolmogorov's axioms of probability theory._\n",
    "\n",
    "### Basic Terminology\n",
    "_Classical probability theory can be best described with the help of an observation of a random experiment, e.g., the rolling of a die._\n",
    "\n",
    "_The **sample space** $\\Omega$ represents all possible outcomes of a random experiment , i.e., $\\Omega=\\{1,2,\\dots,6\\}$ for the rolling of a die._\n",
    "\n",
    "_**Events** are sentences about the experiment involving a number of outcomes that can be assigned with a truth value, e.g., that an even number has been rolled, $A=\\{2,4,6\\}$. Hence, $A\\subseteq\\Omega$ holds._\n",
    "\n",
    "_The **probability** of an event $P(A)$ is a **probability measure** $P$ that assigns an event $A$ a value out of $[0,1]$, i.e., $P:A\\subseteq\\Omega\\rightarrow[0,1]$, while $P(\\Omega)=1$ and $P(\\varnothing)=0$, i.e., an event that is not in $\\Omega$ cannot occur during the random experiment. Furthermore, for a probability measure countable additivity must hold. That is, for all countable sequences of disjoint events $\\{A_{i}\\}$ (see below) the following holds:_\n",
    "$$\n",
    "P(\\bigcup_{i\\geq1}A_{i})=\\sum_{i\\geq1}P(A_{i}).\n",
    "$$\n",
    "\n",
    "_To simplify the handling of $P(A)$, we commonly assign it a probability based on the frequency or chance of the event, e.g., $P(\\mbox{``1 is rolled''})=\\frac{1}{6}$._\n",
    "\n",
    "### Event Algebra \n",
    "_Being a Boolean algebra, the algebra of events and probabilities resembles the algebra of real numbers, i.e., a union ($\\cup$) corresponds to an addition while an intersection ($\\cap$) corresponds to a multiplication, if the events are independent._\n",
    " \n",
    "_Two events $A$ and $B$ are *independent* if and only if\n",
    "$$\n",
    "P(A\\cap B)=P(A)\\cdot P(B).\n",
    "$$\n",
    "_That is, the occurrence (or non-occurrence) of $A$ has no effect on the occurrence of $B$. Otherwise, we have to deal with a conditional probability (see below)._\n",
    " \n",
    "_The probability of mutually exclusive/disjoint events, i.e., $E_{i}\\cap E_{j}=\\varnothing;~\\mbox{for~} i\\neq j$, is defined as follows:_\n",
    "$$\n",
    "P(E_{1}\\cup E_{2}\\cup \\dots \\cup E_{n})=\\sum^{n}_{i=1}P(E_{i})\n",
    "$$\n",
    "_If two events are *not* mutually exclusive the following applies:_\n",
    "$$\n",
    "P(A\\cup B)=P(A)+P(B)-P(A\\cap B)\n",
    "$$\n",
    "_To conclude, let the complement be:_\n",
    "\n",
    "$$\n",
    "P(\\Omega\\backslash A)=P(\\neg A)=1-P(A)\n",
    "$$\n",
    "\n",
    "### Conditional Probabilities\n",
    "\n",
    "_If two probabilities are affected by each other, they are called *conditional*. That is, the probability of $A$ on the condition that $B$ has occurred is defined as follows:_\n",
    "$$\n",
    "P(A | B)=\\frac{P(A \\cap B)}{P(B)} \\quad\\mbox{if } P(B)\\neq 0.\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bayes' Theorem\n",
    "\n",
    "_The Bayes' theorem can be interpreted as a means to calculate the conditional probabilities of two events $A$ and $B$ by \"inverting\" them. The following transformation can become useful in the domain of IR (as well as in other fields) if certain conditional probabilities are easier to calculate than others._\n",
    "$$\n",
    "P(A | B)=\\frac{P(B | A)\\cdot P(A)}{P(B)}\n",
    "$$\n",
    "\n",
    "Further details can be found in \"Think Bayes\", a very fun O'Reilly book(online at http://www.greenteapress.com/thinkbayes/.\n",
    "\n",
    "The theorem is named after [Thomas Bayes](https://en.wikipedia.org/wiki/Thomas_Bayes), an English Presbyterian minister and mathematican who lived in the 1700s.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Naive Bayes Classifier\n",
    "\n",
    "The Naive Bayes classifier is a [classifier](https://en.wikipedia.org/wiki/Statistical_classification) based on Bayes' theorem. In computer science and ML, a classifier is an algorithm or method that assigns an observation (e.g., a new review) of a population (e.g., your set of reviews) to a certain sub-population, class, or category (e.g., whether the review is positive or not). Wikipedia provides a good overview of the [_Naive Bayes classifier_](https://en.wikipedia.org/wiki/Naive_Bayes_classifier) and its history.\n",
    "\n",
    "Each observation to be used with the Naive Bayes classifier is represented by characteristic, statistical features. In this tutorial, we will use the vector space representation $x$ of a review. Based on these features, we expect the classifier to predict correctly to which class $C_k \\in C$ a specific feature characteristic belongs to - or in other words, whether a review is positive or negative. Using the rules from probability theory introduced before, we can formulize this as follows:\n",
    "\n",
    "$$P(C_k \\vert x) = \\frac{P(C_k) \\ P(x \\vert C_k)}{P(x)}$$\n",
    "\n",
    "That is, we calculate the probability that $C_k$ (e.g., whether a review is positive) occurs given that $x$ (a specific feature of the vector space) has occured before.\n",
    "In practice, we skip the denominator (see the Wikipedia article) and obtain the following formula:\n",
    "\n",
    "$$P(C_k) \\ P(x \\vert C_k)$$\n",
    "\n",
    "The next important assumption of the Naive Bayes classifier is that all events (encoded in each dimension $x_i$ of $x$) occur indenpently from each other. By using this \"trick\", the actual calculation of the probability a feature belongs to a class $C_k$ becomes pretty simple:\n",
    "\n",
    "$$\n",
    "P(C_k \\vert x_1, \\dots, x_n) \\varpropto P(C_k) \\prod_{i=1}^n P(x_i \\vert C_k)\n",
    "$$\n",
    "\n",
    "Mathematically interested readers may now object that this will become problematic if certain feature components become 0 (this is the case if an index term did not occur in a review). That's true. But the implementation of the Naive Bayes classifier takes care of this by introducing a small offset that prevents components from coming 0. However, an explanation of these processes is not necessary to follow this tutorial. As they would also require a deeper statistical discussion, we will skip this issue for now. \n",
    "\n",
    "As you can imagine, a multiplication with very small numbers will yield even smaller results. This may result in an effect called [floating point underflow](https://en.wikipedia.org/wiki/Arithmetic_underflow). Hence, we change the product into a sum and handle non-existent terms by adding a small offset $\\alpha$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Implementing the Classifier\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rotten Tomatoes Data Set\n",
    "\n",
    "In order to build our classifier, we need to read in data. The data file is provided in CSV format in the current directory. The \"./\" is a shortcut for the current working directory in which this notebook resides. Mac or Linux user can check this by running the following cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/Users/david/Documents/src/python/dst4l-copenhagen\n",
      "NaiveBayes.ipynb              all.csv                       english.stop.txt              terms2.png\n",
      "README.md                     babypython.ipynb              hamlet.txt                    vsm.png\n",
      "Teaser.ipynb                  critics.csv                   obfuscatedpython.md\n",
      "TextAnalysisScikitLearn.ipynb dst4l0.ipynb                  terms.png\n"
     ]
    }
   ],
   "source": [
    "!pwd\n",
    "!ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Windows users will have to run this cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/bin/sh: dir: command not found\r\n"
     ]
    }
   ],
   "source": [
    "!dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 235,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>critic</th>\n",
       "      <th>fresh</th>\n",
       "      <th>imdb</th>\n",
       "      <th>publication</th>\n",
       "      <th>quote</th>\n",
       "      <th>review_date</th>\n",
       "      <th>rtid</th>\n",
       "      <th>title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Derek Adams</td>\n",
       "      <td>fresh</td>\n",
       "      <td>114709</td>\n",
       "      <td>Time Out</td>\n",
       "      <td>So ingenious in concept, design and execution ...</td>\n",
       "      <td>2009-10-04</td>\n",
       "      <td>9559</td>\n",
       "      <td>Toy story</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Richard Corliss</td>\n",
       "      <td>fresh</td>\n",
       "      <td>114709</td>\n",
       "      <td>TIME Magazine</td>\n",
       "      <td>The year's most inventive comedy.</td>\n",
       "      <td>2008-08-31</td>\n",
       "      <td>9559</td>\n",
       "      <td>Toy story</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>David Ansen</td>\n",
       "      <td>fresh</td>\n",
       "      <td>114709</td>\n",
       "      <td>Newsweek</td>\n",
       "      <td>A winning animated feature that has something ...</td>\n",
       "      <td>2008-08-18</td>\n",
       "      <td>9559</td>\n",
       "      <td>Toy story</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Leonard Klady</td>\n",
       "      <td>fresh</td>\n",
       "      <td>114709</td>\n",
       "      <td>Variety</td>\n",
       "      <td>The film sports a provocative and appealing st...</td>\n",
       "      <td>2008-06-09</td>\n",
       "      <td>9559</td>\n",
       "      <td>Toy story</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Jonathan Rosenbaum</td>\n",
       "      <td>fresh</td>\n",
       "      <td>114709</td>\n",
       "      <td>Chicago Reader</td>\n",
       "      <td>An entertaining computer-generated, hyperreali...</td>\n",
       "      <td>2008-03-10</td>\n",
       "      <td>9559</td>\n",
       "      <td>Toy story</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               critic  fresh    imdb     publication                                              quote review_date  rtid      title\n",
       "1         Derek Adams  fresh  114709        Time Out  So ingenious in concept, design and execution ...  2009-10-04  9559  Toy story\n",
       "2     Richard Corliss  fresh  114709   TIME Magazine                  The year's most inventive comedy.  2008-08-31  9559  Toy story\n",
       "3         David Ansen  fresh  114709        Newsweek  A winning animated feature that has something ...  2008-08-18  9559  Toy story\n",
       "4       Leonard Klady  fresh  114709         Variety  The film sports a provocative and appealing st...  2008-06-09  9559  Toy story\n",
       "5  Jonathan Rosenbaum  fresh  114709  Chicago Reader  An entertaining computer-generated, hyperreali...  2008-03-10  9559  Toy story"
      ]
     },
     "execution_count": 235,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# reads the CSV using the pandas Python package, this creates a variable called \"critics\" that will store the\n",
    "# CSV contents in form of a table\n",
    "critics = pd.read_csv('./critics.csv')\n",
    "# let's drop rows with missing quotes by using the negation operator ~\n",
    "# an alternative way of expressing this would be: critics = critics[~critics.quote.notnull()]\n",
    "critics = critics[~critics.quote.isnull()]\n",
    "# display the first rows of the table\n",
    "critics.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The *read_csv()* function is a powerful tool and can deal with a variety of CSV file formats as its [documentation](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) shows.\n",
    "The function returns a so-called [DataFrame](http://pandas.pydata.org/pandas-docs/stable/api.html#dataframe) which itself offers a lot of functionality as the *.head()* function used above. Another function is *.dtypes()* that displays the data types of the columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "critic         object\n",
       "fresh          object\n",
       "imdb            int64\n",
       "publication    object\n",
       "quote          object\n",
       "review_date    object\n",
       "rtid            int64\n",
       "title          object\n",
       "dtype: object"
      ]
     },
     "execution_count": 188,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "    critics.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explore Your Data Set\n",
    "Typically, the first step in IR and MR is to learn more about your data, i.e., to get a feeling for it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of reviews: 15561\n",
      "There are 623 active critics. \n",
      "1921 movies were found.\n"
     ]
    }
   ],
   "source": [
    "# the len() function return the length of an object, i.e., the number of rows stored in critics coming from your CSV\n",
    "n_reviews = len(critics)\n",
    "# find the number of unique entries in the rtid column\n",
    "n_movies = critics.rtid.unique().size\n",
    "# find the number of unique entries in the critic column\n",
    "n_critics = critics.critic.unique().size\n",
    "\n",
    "\n",
    "print \"Number of reviews: %i\" % n_reviews # %i is a placeholder for the number following after the string\n",
    "print \"There are %i active critics. \" % n_critics # you can place the placeholder wherever you want\n",
    "print \"%i movies were found.\" % n_movies "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x111c78b50>"
      ]
     },
     "execution_count": 190,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxQAAAI9CAYAAACjVz7bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd4FOXexvF7E4r0g3QOIH1DCQklQAAhhSPNgDQDBGmG\nomBBiqCgAgoe2hERKRGkCEJE6gFRQhUIIEgTI0oLTYrSSTAhmfcP3uxhSQKbIXE34fu5Lq6LPDM7\n89uZ3dm5Z56ZsRiGYQgAAAAATHBzdgEAAAAAMi8CBQAAAADTCBQAAAAATCNQAAAAADCNQAEAAADA\nNAIFAAAAANMIFICkO3fuqGHDhvLw8ND8+fOdXU6Wd+rUKdv/z5w5Iw8PD40cOdKJFWVuw4YNk4eH\nhy5cuGDXHh0dbfe3h4eHevXqlW7zXbhwofz9/eXp6al27dql23QzgrOWUUbZtWuXPDw8NGPGDLv2\nCxcu6Pbt27a/U3vfzjR79mxVq1Yt1eERERHq0KGDatasqYYNG+qdd97R1atXk413584dzZo1S82a\nNZOXl5eeeeYZzZo1SwkJCRlZfppNnTpVHh4e+vHHH51dCpBhsjm7AMAVbN26VX/++ady586t8PBw\ndevWzdklZUk3b95UaGioKlasqPfff1+SVKhQIU2YMEFPPfWUk6vLvDp16qSGDRuqQIECtra+ffsq\nPj5ec+bMsbVNmDBBRYoUSZd5HjlyRGPGjFHZsmU1cuRIPfnkk+ky3YzijGWUkSpWrKgJEyaoSpUq\ntrbly5dr9OjRWrdunZ544glbu8VicUaJKdqwYYMmT56cak0rVqzQsGHDVLt2bQ0ePFjnz5/XvHnz\ntG/fPn311Vd272vkyJFavny5WrdurRdffFG7du3S5MmTdfLkSY0dO/bveksP9cwzz6hs2bIqW7as\ns0sBMgyBApC0bNky5c6dW506ddKcOXO0Z88e1alTx9llZTlXr17V/v37ValSJVtbrly5FBQU5MSq\nMj9vb295e3vbtW3ZskUNGjSwa0vP5fzrr79Kkrp166bnn38+3aabUZyxjDJSoUKFktW6a9cuxcbG\nJhv3UZ5fe/PmTeXNm9f065MkJCRo9uzZ+uijj5SYmKhs2ZLvfsTExGjcuHHy9vbW/Pnz5e7uLkmq\nVq2aXn/9dS1YsEC9e/eWJO3fv1/Lly9Xz5499eabb0qSnn/+eRUoUECLFi1Sp06dVKNGjUeuOz1Y\nrVZZrVZnlwFkKLo84bF3+fJlbd68WT4+PmrevLkkacmSJU6uKmt7lB0cuIb4+HhJSpedTbiu9u3b\na9CgQdq7d6/paVy7dk2tW7fW5MmT5e/vn2p3p82bN+vatWvq3LmzLUxIUvPmzfXPf/5Tq1atsrWt\nWLFCktS9e3e7aYSGhkqSVq5cabpeAGlHoMBjb9WqVbpz544aNGigGjVqqGTJkvr2229tfXYvX76s\n6tWrq2/fvslee+PGDdWoUcN21Ey6exT+gw8+kL+/v6pXry4/Pz+NHj1aly9ftnttQECA+vfvr48/\n/li1a9eWj4+P7QfzwoULGj16tP71r3+pRo0aqlmzptq0aaOFCxcmq+HYsWN69dVXVb9+fdWsWVN9\n+vTRsWPHVLVqVQ0fPtxu3KNHj+q1115TvXr15OnpqVatWiksLMyhPscvvPCCnnvuOS1evNg2r1mz\nZkm6u8MwadIktWrVSt7e3vLy8lLLli01bdo027SXLVumpk2bSpKWLl0qDw8P/fDDD7ZrKEaMGJFs\n2ezYsUOdOnWSt7e36tWrpyFDhiTrC24YhubNm6dWrVrJy8tLTZs21WeffaZp06bJw8ND586de+D7\n8vDw0EcffaSFCxcqMDBQXl5eCgoK0ldffZVs3ISEBM2dO1dBQUGqUaOG6tWrpwEDBujIkSN24w0b\nNkw+Pj6KiIhQkyZN5OXlpVGjRj2wjhMnTmjIkCFq1KiRatasqaCgIC1YsMAWvpKW0/Tp0/Xqq6/K\n09NTjRo10vHjx+36ySf1rZekHTt2yMPDw7bz5eHhoZ49e6ZpvikJCAjQW2+9JUkaOnSobV1KUmxs\nrKZMmaJmzZqpevXq8vX11euvv66jR49m6WV06NAh9e7dW40aNVKNGjXUrFkzTZo0KcUzBkmOHTsm\nDw8PW/e/JAsXLpSHh4dGjx6dYvvBgweTXUMREBBgew9NmjRJ1m3z1KlTevXVV+Xj46NatWqpR48e\n+umnnx64vKW7O/Pff/+9QkJC1Lp1a3355Ze6devWQ193rxs3biguLk6TJk3StGnTlDt37hTHO3Dg\ngCTJy8sr2TBPT08dPXrUdn3IgQMHVKRIERUvXtxuvJIlS6pQoUI6dOjQA2tatmyZPDw8tHbtWrVr\n106enp5q27atEhMTJT18WxkfHy9fX189++yzyaadmJioxo0bq2XLlpJSv4ZixYoVat++vby9vVW7\ndm316tVLe/bssQ2fMGGCPDw8dPjwYbvXtW/fXh4eHsneY4cOHdSiRQtJd7dVH3/8sW2bXK9ePb34\n4ovavXv3A5cLYBZdnvDYW7Zsmdzc3GxnJ1q0aKHZs2drxYoV6tGjh5588kn5+/tr06ZNunz5sl1f\n8XXr1ikuLk7t27eXdDdMBAcH69KlSwoODla5cuX066+/Kjw8XFu3blV4eLjd6yMjI3XkyBG98cYb\nunjxourWravr16+rY8eOiouLU5cuXfTPf/5Tly5dUnh4uMaMGaPs2bPbupgcO3ZMnTp1kmEY6tq1\nqwoXLqw1a9aoS5cuyXYI9+/fr549e6pgwYLq2bOnChQooMjISE2aNEn79u3TtGnTHtrXOjo6Wh9/\n/LH69eun27dvq2HDhoqPj1f37t118uRJde7cWRUrVtTVq1e1YsUKTZ06VXFxcRo4cKB8fHw0fPhw\njRs3TvXq1VOHDh1Uvnx52w7X/fM+fPiw+vfvr/bt26t9+/bas2ePVq5cqdOnT2vx4sW28d59912F\nh4erQYMG6tKli86ePatPPvlETzzxhMN9x1evXq0//vhDXbt2VYkSJbR69WqNHDlSZ86c0cCBAyXd\nDS4DBw7U+vXr1apVK4WEhOiPP/7QkiVLFBwcrNmzZ6t27dq2acbGxuqtt95Sz549lStXLlWoUCHV\n+f/yyy/q0qWL3Nzc1KVLF5UqVUpbt27VBx98oFOnTuntt9+2jRsWFqbq1avrnXfe0alTp1S+fHm7\n5VexYkWNHz9eQ4cOVaVKldSnTx/VrFnT9vp7l0la5nuvt956y/Z57tKli2rWrKny5csrJiZGL7zw\ngg4fPqwWLVqoR48e+v3337Vo0SJt2bJFn332WZZcRtHR0erZs6eKFy+u0NBQ5c2bV7t27VJYWJhO\nnDihTz75JMX3VKFCBT311FPatm2bXXvS3zt37rRr37Rpk4oWLaoaNWpo165dydbJ3LlztWfPHo0c\nOdL2npP06dNHjRs31tChQxUdHa358+erZ8+e+vbbbx94/cvAgQP10ksvae3atQoPD9eoUaM0ceJE\ntW7dWl26dLHrvpiaEiVKaP369Q8d7/z585KULCRIUrFixWQYhs6ePasKFSro/PnzKlWqVIrTKVq0\nqM6cOfPQ+UnSiBEj9Oyzzyo4OFh37tyRm5ubQ9vK7Nmzq02bNpo7d66ioqLsrmXZuXOnLl68mOzs\nyb0+/PBDzZ07V35+fmrXrp1u3ryp5cuXq3v37po4caJatGihwMBAzZ49W9u3b7ed1bly5Yp+/vln\nWSwW7dy5U56enpKkS5cu6fDhw7aDW++//76WLFmiTp06qWrVqrpy5YoWLVqkXr16afHixapevbpD\nywdwmAE8xn766SfDarUaXbp0sbUdPnzYsFqtRvPmzW1tGzduNKxWq/HFF1/YvT4kJMSoW7euERcX\nZxiGYYwcOdKoVq2a8dNPP9mN98MPPxgeHh7Gu+++a2vz9/c3PDw8jP3799uNO3fuXMPDw8PYsmWL\nXfuJEycMq9VqvPTSS7a2Pn36GNWqVTOioqJsbQkJCUavXr0Mq9VqDBs2zDAMw0hMTDRatGhhBAYG\nGjdu3LCb7qeffmpYrVZjzZo1D1xWXbt2TXG89evXG1ar1Vi0aJFd+40bN4waNWoYrVu3trWdPn3a\nsFqtxogRIx7Y5u/vb1itViMiIsJumqGhoYbVajWio6MNwzCMAwcOGFar1XjllVfsxvvhhx8Mq9Vq\neHh4GGfPnn3g+0oaLzIy0tYWHx9vBAcHG1WrVjVOnTplGIZh/Pe//zWsVquxYMECu9f/+eefRsOG\nDY2WLVva2t58803DarUaM2fOfOC8k4SEhBje3t7GsWPH7Nr79u1rVKtWzfjjjz9sy6lu3brG7du3\n7cZLmt/58+ft3lfPnj2Tvdd72xyZb2q+/vprw2q1GqtWrbK1TZ06NcX3fezYMcPT09N45plnjMTE\nRLuaM/syunTpkhEWFmZYrVbj0KFDduMMHTrUCA4ONv76669U39eHH35oWK1W2+c0Pj7eqFWrluHn\n52dYrVbj4sWLhmEYxq1btwxPT0/bNmTnzp2G1Wo1pk+f/sD3mNQ2evRou/lOmzbNsFqtxvLly1Ot\nLSW//PKLMWrUKKN27dqG1Wo1QkJCjDNnzqRpGl27djWqVauWrL1Xr15GlSpVUnzN5MmTDavVahw8\neNAwDMOoVq2a0a1btxTH7dSpk+Ht7f3AGpI+vy+++KJde1q2lUeOHDGsVqvx4Ycf2o03dOhQ22fD\nMAzj448/NqxWq7F3717DMAxj3759Kb4uNjbWCAoKMurVq2fExsYaCQkJhq+vr/HCCy/YxlmzZo1h\ntVoNPz8/o1evXrb28PBww2q1GgcOHDAMwzC8vLyMvn372k3/559/Npo1a2aEh4c/cNkAZtDlCY+1\n5cuXS5Lt1LQkVa1aVWXLltWJEydsRwEbN26swoUL2/XhPXv2rPbu3augoCBlz55dhmFo3bp1qly5\nskqWLKnLly/b/pUvX17lypVLdpQuV65cyU7vd+/eXdu3b1fjxo1tbYZhKC4uThaLRTExMZLudiPY\ntm2bGjdubOu+IUlubm7q37+/3TR/+eUXHT9+XH5+foqLi7OrLenMjCNHEC0Wi+rXr2/X1rRpU+3e\nvVsdOnSwa798+bLy5ctnqzet8uTJo8DAQLu2pKN0ly5dkiR98803kqSXXnrJbrw6deqoQYMGDl+r\n4ePjY/e+smXLphdffFEJCQnasGGDJGnNmjWyWCwKDAy0W36S5Ofnp2PHjun48eN20/X19X3ovC9f\nvqw9e/aoSZMmyY4qjxo1SqtWrdI//vEPW5u3t7dy5szp0PtKz/k6Yt26dcqfP3+y266WL19ebdq0\nUXR0tKKiouyGZfZlVLBgQZUsWVLS3S4qkZGRiouLkyT9+9//1uLFi5UjR45U5xEQECBJ2r59u6S7\nZxJv3bpluxYg6SzFjh07FBcXZ+s2mFZt2rSx+ztpu3Px4sU0Tcdqtertt9/WiBEjlCdPHu3du/eh\n3Qod5cj31c3t4bsthmE4fHby/s9fWraVlStXVrVq1bRmzRpb7bGxsfruu+/UqFEjFS5cOMV5rlmz\nRtLduz/dO/2YmBg988wzunr1qn744Qe5ubmpSZMm2rdvn62r1/bt21W+fHkFBAToxx9/tHXB2rx5\ns+3slXS369euXbs0Z84c2/qpUqWK1q1bp44dOzq0bIC0oMsTHltxcXFavXq1pLv9c+89RV6/fn2d\nPHlSS5YsUb169eTu7q7WrVtrzpw5On36tEqXLq2VK1fKMAy1bdtW0t2dj+vXr+vnn39OdSfJYrEo\nLi7OtoORWlcDwzA0c+ZMHThwQKdPn9aZM2dsXYOS+vieOnVKCQkJyXZ0JCXrOnLixAlJ0hdffKEv\nvvgixXn+/vvvKS+o++pPqWY3NzctXrxYP/zwg06fPq3Tp0/r5s2bkqR//vOfD51uSlKaT9JyS1oG\nJ06ckMViSXEZlC9fXjt27HBoXpUrV07WVq5cOUnS6dOnJUknT56UYRjy9/dPcRoWi0Xnzp2zqyW1\nHYp7Jf3Yp/QeihUrlqytUKFCD52mI9I6X0ecPn1alSpVSvEOPkmfyTNnzqhq1aq29qywjJKuM1i+\nfLl27dqlXLlyqW7dugoMDFTr1q3tbnV6v1q1aqlAgQLatm2bOnbsqO3btyt//vzq2LGj/vOf/2jX\nrl0KCgrS5s2blS9fvmSB3lH33wo3KXAlhR9HnDt3Tl999ZWWLl2qS5cuqXLlyurSpUu63U0pd+7c\ntoMn94ewpO1fvnz5bOPe+7yNe92+fds23sPc/1lJ67ayXbt2GjNmjHbu3ClfX1999913io2NfeBz\nWU6ePClJ6ty5c4rDk7YlkhQYGGj7XDVp0kQ7duxQYGCgateurYULF2r//v3y9PTUjh077ELj+++/\nr1dffVXjx4/X+PHjVb58eT399NNq3br1A58BAphFoMBja+PGjbp27ZokpXrby/Xr19uum2jfvr3m\nzJmjlStXasCAAVq5cqXtCJX0v53cunXrJjtifq97715y7/+T7N+/33aEt0GDBgoMDFTlypVVu3Zt\n+fn52cZLustOSkc/c+XKZfd30tGzbt26pbpDnCdPnlRrTpLS0cFTp06pS5cuunbtmnx9fdWwYUNV\nqlTJduGnWY4ciUxaBtmzZ0827P5l8CApLcM7d+5I+t86SkxMVP78+TVlypRUp3P/rSEdeQ9J83H0\niGpKnxkz0jpfRzzoCHPSsPuXdVZYRm5ubho3bpxefvllbdiwQZGRkfrhhx+0ZcsWhYWFKTw8XAUL\nFky11iZNmmjLli0yDEM7duxQ3bp1lSNHDtWpU8cWirds2aLGjRunGNYc4chyTkliYqK2bt2qxYsX\na+vWrXJzc1PTpk3VtWvXdL+1dqlSpWQYhs6fP68yZcrYDbtw4YKyZcumokWL2sZN7SDIhQsXHH6u\nzf2flbRuK4OCgvTvf/9bq1atkq+vr+2sVdKZp5Qk/VbMnDkz1bNXSQc0GjRooJw5c2r79u0qXbq0\nfv/9d/n6+qp27dqyWCyKjIxUbGysYmNj7c5e1apVSxs3btSOHTv0/fffKzIyUvPmzdO8efM0bNiw\nR9o2AykhUOCxtWzZMknSgAED7C6ok+7+qMyZM0c//vijli1bptDQUFWoUEFeXl5at26d/Pz8FB0d\nrWHDhtle8+STTyp37ty6efNmimcoNm3apHz58j10Z+ejjz5SXFyc1qxZY/ej+Mcff9jtsD311FOy\nWCw6duxYsmnc3/Um6eJFi8WSrLa//vpLGzduNP0wr5kzZ+qPP/7QvHnzVK9ePVt7QkJCsovY01u5\ncuW0fft2HT9+XBUrVrQbdv8yeJCko5IpvT7ph71UqVKKjo5W9erVkx39TOqmkpYQkyTpDE5KNezZ\ns0dffvmlevbsmebuR+k137RcvFm6dGlFR0crPj4+Wcj77bffJN29QDejanXWMipSpIiOHz8uX19f\n9ejRQz169FBcXJwmTpyo+fPna+3atQoJCUl1PgEBAVq1apV27typw4cP2+7O1qBBA23evFnr1q3T\npUuXknUB/DsEBwfr0KFDKlKkiF5++WUFBwdn2IP/ks50HDhwIFmgOHTokCpWrGg72+Pp6aklS5bo\nwoULdmeLzpw5oytXrph+nkjp0qUlOb6tzJ8/v5o2baoNGzbowoULioyMVEhIyAODX9L2uFixYnbd\nVaW7D4w8f/68bVuSO3du1a9fX9u2bVPZsmXl7u6uevXqKW/evKpSpYp27typq1ev2p29io+P15Ej\nR1SgQAH5+fnZDkQdPXpUL7zwgmbMmEGgQLrjGgo8li5evKht27apcOHCeumllxQYGGj3r2nTpurT\np48kKTw83Pa6du3a6ejRo5oxY4ayZcum1q1b24a5u7srMDBQP//8syIiIuzmt2vXLr300ku226w+\nyJUrV5QnTx5bv+wkYWFhkmTrM1uwYEH5+vpqy5YttlPo0v/C0L08PT1VsmRJLV26NFl/57CwMA0c\nOFCbN29+aG2p1Ssp2d1eFi5cqNjYWLtb0t57tD89JF37MnfuXLv23377TVu3bnX4iPbWrVttD2qT\n7nYD+eyzz5QzZ07bUb9mzZrJMAx9/PHHdq/9888/1b9/fw0aNMjUkfEiRYqoRo0a2rRpk617VZL5\n8+frm2++MbUDZ7FYHricM2K+zZo1040bN5J9/k6cOKHVq1erTJkyph7w5erLaMaMGerZs6cOHjxo\nG54jRw7b2cuHnVV4+umnlT17dk2ZMsV2C2tJatiwoSTpP//5j3LkyKEmTZo8cDpJZyHS6/sl3T1w\nMXnyZG3evFkDBgzI0KeIN2nSRHnz5tX8+fNtZ4eku9dKnTt3Ts8995ytLSkwzJ49224aSX8ndUVN\nq+rVq6d5W9muXTtdv35dY8aMUWJiou2uf6lJuhZj2rRpdgeJYmJiNGjQIPXv319//fWXrT0gIEDH\njx/XqlWr5OnpaXv2i6+vr/bv36+NGzfanb26evWqOnbsmOx2xOXLl1f+/PlNn+UCHoRPFR5LK1as\nUGJiojp06JDqTqCfn5+eeuopRUdHKzIyUr6+vmrVqpXGjRuniIgINW3aNNnR98GDB2v37t167bXX\n1L59e1WrVk2nTp3SokWLVKBAAQ0dOvShtQUEBGj69OkKDQ1V8+bNFR8fr/Xr1+v48eMqXLiwrl+/\nbht3+PDh6tSpk55//nmFhISoUKFCioiI0P79+yX9r6uGm5ubxowZo379+qlt27bq1KmTSpYsqb17\n92r16tWqWrWqunTp8tDaUurSEhAQoI0bNyo0NFTt2rWTxWLR9u3btXPnTpUqVcru+RsFCxaUu7u7\nIiMj9dVXX6lRo0YPneeD1KpVS23bttXSpUt1/vx5+fv76+LFi/riiy/k7u6uO3fuOBQqsmXLpq5d\nu+qFF15Qvnz5tGrVKkVFRWn48OG2Hah27drpm2++0YIFC3Ty5Ek1adJEMTExWrx4sa5cuaJ///vf\nD7z49kFGjhypbt26qUOHDgoJCVGRIkW0detWbdq0SQMGDFCxYsUcvg1mksKFC+vw4cP68ssv5ePj\nk+wMjqPzTYvQ0FBt3LhR//nPf/TLL7+oTp06unDhghYtWqRs2bJp7NixaZpeWmt11jLq1q2bVq9e\nrb59+9q+W2fOnNGiRYtUokQJu5s+pCRPnjyqW7eutm/fruLFi9vOilWoUEFFihRRdHS0nn766Yd2\nS0z6rIaFhalRo0YP7HbjqIkTJz7yNFKS0rYkT548GjJkiN5991117dpVzz33nM6ePav58+eratWq\n6tSpk23cOnXqqFWrVpo/f76uXr0qHx8f7dy5U2vWrFFwcHCys86OMrOtbNiwoYoXL66IiAhVqVIl\n2VmH+/n6+qpt27Zavny5OnfubAsYX3/9tY4dO6aBAwfaunZJkr+/v959913t379f/fr1s7U3aNBA\ns2fP1rlz5zR48GBbe5EiRdSxY0eFh4erb9++tht8rF+/XtHR0bYniwPpiTMUeCytWLFC7u7uCg4O\nfuB4SQ+HSnruQd68edW0aVNZLJYUj4AVK1ZMX3/9tZ5//nl9//33GjNmjNauXat//etfWrJkSYo7\nLPfr37+/+vfvr3Pnzmns2LH6/PPPVblyZa1atUqNGzfW8ePHbXdmqVSpkhYuXKiaNWtq/vz5mjRp\nknLlyqWPPvpIkn1/9YYNG2rx4sXy8fHRkiVL9MEHH+jgwYPq3bu35s2bl+rDpu6V0s55hw4d9Pbb\nb+v27dsaP368Pv30U+XPn18rV65U27ZtFRsba3tg1RNPPKHBgwcrNjZW77//vnbt2pWmPvwWiyXZ\n+O+//74GDhyoU6dOady4cVq7dq1ee+01BQQEyDAMh3bymzZtqldeeUVff/21PvroI+XIkUNTp061\neziYu7u7Zs6cqTfeeEPnz5/X+PHj9fnnn6ts2bKaPXu2XReLlOp8EE9PT4WHh6tevXpauHChxo8f\nr4sXL+rDDz/UgAEDTC2XoUOHKnfu3Bo3blyqd/B6lPmmNM/cuXNr4cKF6tOnjw4fPqxx48Zp2bJl\n8vPz09KlS+363GelZVSuXDl98cUX8vHx0bJlyzRq1CitWLFCzZs31+LFix26QDhp5//+bjYNGjSQ\nxWJx6O5OnTt3lre3t7766itNmDAh1fftClKrKTg4WBMnTtTt27c1duxYrV69Wu3atdOcOXOSXdz+\n4Ycfqn///tqzZ4/GjBmjw4cPa8iQIXrvvfccmn9qNaR1W2mxWNSmTZtUfxdSmte4ceP03nvvKT4+\nXh999JE+/fRT5cuXT5MnT7adHU9StGhRVa9eXRaLxXb2SrobqnLkyJHi2at33nlHQ4YM0e+//67J\nkyfblun48eOTPbgRSA8Ww9H7KgJwOZcuXUqxC8KBAwcUHBysAQMGOLSzlVndvHlT7u7uKV67EBoa\nqp07d+rgwYMPvCDVw8NDrVq10qRJkzKyVAAAsizOUACZWEhIiFq0aJGsz/SKFSskye7pv1nRxo0b\nVbNmTbvng0h3b+u4e/duVa9e3fTdbQAAgGO4hgLIxIKDgzVhwgR1795dzZs3l5ubm3bt2qV169bJ\n39/fdlFnVhUQEKCiRYtq9OjR+u2331SmTBldvHhR4eHhslgs9BUGAOBvQJcnIJNbs2aNvvjiCx0/\nflxxcXEqU6aM2rRpox49ejwWR+fPnz+vGTNmaNu2bbp48aLy58+vOnXqqF+/fg+9OFKiyxMAAI+K\nQAEAAADAtMeiy9PevXudXQIAAADgkmrXrv1Ir38sAoX06AsKricqKkqSTN9vHK6LdZt1sW6zLtZt\n1sW6zbqioqIUExPzyNPJ+h2sAQAAAGQYAgUAAAAA0wgUAAAAAEwjUAAAAAAwjUABAAAAwDQCBQAA\nAADTCBQAAAAATCNQAAAAADCNQAEAAADANAIFAAAAANMIFAAAAABMI1AAAAAAMI1AAQAAAMA0AgUA\nAAAA0wgUAAAAAEwjUAAAAAAwjUABAAAAwDQCBQAAAADTCBQAAAAATCNQAAAAADCNQAEAAADANAIF\nAAAAANMxGO5uAAAgAElEQVQIFAAAAABMI1AAAAAAMI1AAQAAAMA0AgUAAAAA0wgUAAAAAEwjUAAA\nAAAwjUABAAAAwDQCBQAAAADTCBQAAAAATCNQAAAAADCNQAEAAADANAIFAAAAANMIFAAAAABMI1AA\nAAAAMI1AAQAAAMA0AgUAAAAA07I5uwAAwOOjSpUqzi4BAJDOCBQAgDQLGrTS2SWkq9WT2ji7BADI\ntOjyBAAAAMA0AgUAAAAA0wgUAAAAAEwjUAAAAAAwjUABAAAAwDQCBQAAAADTCBQAAAAATCNQAAAA\nADCNQAEAAADANAIFAAAAANMIFAAAAABMI1AAAAAAMI1AAQAAAMA0AgUAAAAA0wgUAAAAAEwjUAAA\nAAAwjUABAAAAwDQCBQAAAADTCBQAAAAATCNQAAAAADCNQAEAAADANAIFAAAAANMIFAAAAABMI1AA\nAAAAMI1AAQAAAMA0AgUAAAAA0wgUAAAAAEwjUAAAAAAwjUABAAAAwDQCBQAAAADTCBQAAAAATCNQ\nAAAAADCNQAEAAADANJcIFBs2bFCtWrWStU+fPl1+fn7y9vZWr169dPz4cSdUBwAAACA1Tg8UP/74\no4YMGZKs/ZNPPtGMGTMUGhqqyZMn68aNG+rRo4du3rzphCoBAAAApMRpgSIuLk5hYWHq3r27smfP\nbjfs5s2bmj17tl555RV17dpVAQEBmj17tm7duqWlS5c6qWIAAAAA93NaoNi6davCwsL05ptvqmvX\nrjIMwzbswIEDio2NVUBAgK0tf/788vHx0ffff++McgEAAACkwGmBwtPTUxs3blTXrl2TDTt58qQk\nqUyZMnbtpUqV0okTJ/6O8gAAAAA4IJuzZlysWLFUh928eVM5cuRQtmz25eXJk0e3bt0yNb+oqChT\nr4Prio2NlcS6zYpYt66tSpUqzi4hQ/B5ezR8b7Mu1m3WlbRuH5XTL8pOiWEYslgsKQ5LrR0AAADA\n389pZygeJF++fIqLi1NCQoLc3d1t7bdu3VL+/PlNTTOrHlF7nCUdKWHdZj2sWzgDn7dHw/c262Ld\nZl1RUVGKiYl55Om45BmKp556SoZh6MyZM3btZ86cUbly5ZxUFQAAAID7uWSgqFmzpnLmzKn169fb\n2q5du6bdu3fL19fXiZUBAAAAuJdLdnnKkyePunbtqilTpsjNzU1PPfWUZsyYofz586tDhw7OLg8A\nAADA/3OJQGGxWJJdbP3GG2/Izc1Nc+bM0a1bt1SrVi2NHz9eefPmdVKVAAAAAO7nEoFiwIABGjBg\ngF2bu7u7Bg0apEGDBjmpKgAAAAAP45LXUAAAAADIHAgUAAAAAEwjUAAAAAAwjUABAAAAwDQCBQAA\nAADTCBQAAAAATCNQAAAAADCNQAEAAADANAIFAAAAANMIFAAAAABMI1AAAAAAMI1AAQAAAMA0AgUA\nAAAA0wgUAAAAAEwjUAAAAAAwjUABAAAAwDQCBQAAAADTCBQAAAAATCNQAAAAADCNQAEAAADANAIF\nAAAAANMIFAAAAABMI1AAAAAAMI1AAQAAAMA0AgUAAAAA0wgUAAAAAEwjUAAAAAAwjUABAAAAwDQC\nBQAAAADTCBQAAAAATCNQAAAAADCNQAEAAADANAIFAAAAANMIFAAAAABMI1AAAAAAMI1AAQAAAMA0\nAgUAAAAA0wgUAAAAAEwjUAAAAAAwjUABAAAAwDQCBQAAAADTCBQAAAAATCNQAAAAADCNQAEAAADA\nNAIFAAAAANMIFAAAAABMI1AAAAAAMI1AAQAAAMA0AgUAAAAA0wgUAAAAAEwjUAAAAAAwjUABAAAA\nwDQCBQAAAADTCBQAAAAATCNQAAAAADCNQAEAAADANAIFAAAAANMIFAAAAABMI1AAAAAAMI1AAQAA\nAMA0AgUAAAAA0wgUAAAAAEwjUAAAAAAwjUABAAAAwDQCBQAAAADTCBQAAAAATCNQAAAAADCNQAEA\nAADANJcOFIZhaO7cuWrWrJlq1qyp559/Xjt37nR2WQAAAAD+n0sHinnz5mnChAlq3769Pv30U5Uu\nXVqhoaGKiopydmkAAAAA5OKB4uuvv1ZQUJD69OkjX19fTZgwQUWKFNHSpUudXRoAAAAAuXiguHnz\npvLkyWP7283NTXnz5tW1a9ecWBUAAACAJC4dKFq3bq2VK1cqMjJSN27c0Lx583T06FG1atXK2aUB\nAAAAkJTN2QU8yKuvvqojR46oZ8+etraBAwfK39/fiVUBAAAASOLSgWLIkCHat2+f3nvvPVWoUEHb\nt2/X1KlTlTdvXoWEhKRpWlzInfXExsZKYt1mRaxb11alShVnl5Ah+Lw9Gr63WRfrNutKWrePymUD\nxaFDh7R27VpNmTJFzZo1kyT5+PgoISFBEydOVLt27ZQrVy4nVwkAAAA83lw2UERHR0uSvL297dpr\n1aqlsLAwnT17VhUrVnR4eln1iNrjLOlICes262Hdwhn4vD0avrdZF+s264qKilJMTMwjT8dlL8ou\nXbq0JGnv3r127QcOHFC2bNlUvHhxZ5QFAAAA4B4ue4bCy8tLDRo00KhRo3T16lWVL19eu3fv1mef\nfaZu3bopb968zi4RAAAAeOy5bKCQpOnTp2v69OmaN2+eLl68qDJlymjkyJEKDg52dmkAAAAA5OKB\nImfOnHr99df1+uuvO7sUAAAAAClw2WsoAAAAALg+AgUAAAAA0wgUAAAAAEwjUAAAAAAwjUABAAAA\nwDQCBQAAAADTCBQAAAAATCNQAAAAADCNQAEAAADANAIFAAAAANMIFAAAAABMI1AAAAAAMI1AAQAA\nAMA0AgUAAAAA0wgUAAAAAEwjUAAAAAAwjUABAAAAwDQCBQAAAADTCBQAAAAATCNQAAAAADCNQAEA\nAADANAIFAAAAANMIFAAAAABMI1AAAAAAMI1AAQAAAMA0AgUAAAAA0wgUAAAAAEwjUAAAAAAwjUAB\nAAAAwDQCBQAAAADTCBQAAAAATCNQAAAAADCNQAEAAADANAIFAAAAANMIFAAAAABMI1AAAAAAMC2b\nswsAAMAVBA1a6ewS0t3qSW2cXQKAx4DDZyiaNWumKVOm6OjRoxlZDwAAAIBMxOFAUbFiRc2ZM0fP\nPvusgoKCNH36dEVHR2dkbQAAAABcnMNdnqZNm6abN29qw4YN+uabbzR9+nRNmTJFVatWVcuWLdWy\nZUuVLFkyI2sFAAAA4GLSdA1F3rx51aZNG7Vp00Y3btzQhg0btGnTJoWFhWnSpEny8vLSs88+q1at\nWqlgwYIZVTMAAAAAF2H6Lk/58uVT7dq1Vbt2bVWrVk2GYeinn37Shx9+qMaNG+udd97RzZs307NW\nAAAAAC4mzXd5OnnypNatW6dvv/1WUVFRcnNzk4+Pj8aMGaNnnnlGkrR8+XJNnDhRFy9e1IwZM9K9\naAAAAACuIU3XUHz77bf69ddfJUleXl4aPny4mjdvrmLFitmN26NHD+3evVuRkZHpWy0AAAAAl+Jw\noJg6daoqVaqkgQMHqmXLlipduvQDx69Zs6Y8PT0fuUAAAAAArsvhQLFq1SpVrlw5Wfv169eVP3/+\nZO29e/d+tMoAAAAAuDyHL8quXLmylixZoqZNm+r06dO29nHjxqlJkyZas2ZNhhQIAAAAwHU5HCi+\n/vprvfvuuypRooTc3d1t7c2aNVP58uU1aNAgfffddxlSJAAAAADX5HCgmDt3rpo2baoFCxbYPcDO\nz89Pn3/+ufz9/bmjEwAAAPCYcThQnD59Wo0bN051eOPGjXX8+PF0KQoAAABA5uBwoHjyySd16NCh\nVIcfPXpUBQoUSJeiAAAAAGQODgeKoKAgLV26VAsWLFBcXJytPS4uTuHh4Vq8eLFatmyZIUUCAAAA\ncE0O3zb25Zdf1qFDh/TBBx9o/PjxKl68uAzD0IULFxQfH6/69evrtddey8haAQAAALgYhwNFzpw5\nNWfOHG3atElbtmzRuXPnlJCQoPr168vPz0+BgYGyWCwZWSsAAAAAF+NwoEji7+8vf3//jKgFAAAA\nQCaTpkCRmJioH3/8UX/++acSEhJSHIfrKAAAAIDHh8OB4pdfflHfvn114cKFVMexWCwECgAAAOAx\n4nCgGDdunK5fv65BgwbJw8NDOXLkyMi6AAAAAGQCDgeK/fv3q2/fvurdu3dG1gMAAAAgE3H4ORR5\n8+ZV/vz5M7IWAAAAAJlMmh5st2zZMsXHx2dkPQAAAAAyEYe7PNWoUUPffvutnn32Wfn7++vJJ59M\n8bkTdIkCAAAAHh8OB4o33njD9v+5c+emOh6BAgAAAHh8OBwoIiIiMrIOAAAAAJmQw4GiVKlSdn/H\nxcXJ3d1d7u7u6V4UAAAAgMzB4YuyJen333/XsGHDVL9+fXl5eWn37t3as2ePevbsqUOHDmVUjQAA\nAABclMOB4vTp02rfvr0iIiLk7e0twzAkSYZh6ODBg3rhhRd08ODBDCsUAAAAgOtxOFBMmDBB7u7u\nWrt2rcaOHWtr9/Hx0dq1a1WoUCF9/PHHGVIkAAAAANfkcKDYuXOnOnfurKJFiyYbVqxYMYWEhNDt\nCQAAAHjMOBwo4uPjVaBAgVSHWywWxcXFpUtR94qMjFTHjh3l5eWlgIAATZ06VYmJiek+HwAAAABp\n53CgqFatmr755psUh/31119atmyZqlSpkm6FSdLevXvVu3dvVaxYUbNmzVJISIjCwsL06aefput8\nAAAAAJjj8G1jX3nlFfXq1UsvvviiAgICJEk///yzTp06pfnz5+v48eOaOXNmuhY3adIkNWrUSOPG\njZMk1atXT1evXtXu3bvTdT4AAAAAzHE4UNSrV0+ffvqpRo0apTFjxki6e6G2JBUqVEgTJkxQ48aN\n062wy5cva9++fcnORgwaNCjd5gEAAADg0TgcKCSpSZMmWr9+vaKionTq1CklJiaqRIkS8vT0VI4c\nOdK1sCNHjsgwDD3xxBPq16+fduzYobx586pLly7q37+/LBZLus4PAAAAQNqlKVBIkru7u6pXr67q\n1atnRD02V65ckSS9+eabCgoKUq9evbR7925Nnz5dOXPmVO/evTN0/gAAAAAezuFAERoa+sCzAoZh\nyGKxKCwsLF0Ki4+PlyQ9/fTTGjJkiCSpbt26unLliqZPn/7Qeu4XFRWVLnXBdcTGxkpi3WZFrFvX\nlt434EDG+ru+R3xvsy7WbdaVtG4flcOB4vjx48naEhISdPXqVf31118qWbKkKleunC5FSVKePHkk\n3Q0U9/L19dXChQt15swZlS5dOt3mBwAAACDtHA4UGzduTLE9ISFBmzdv1ltvvaUePXqkV10qU6aM\npP+dqUhy584dSUrzNRQcUct6ko6UsG6zHtYtkH7+ru8R39usi3WbdUVFRSkmJuaRp+PwcyhS4+7u\nrsDAQAUHB2vixImPXFCSSpUqqVixYsmefbFlyxYVK1ZMpUqVSrd5AQAAADDnkQNFkpIlS+rXX39N\nr8nJYrFo4MCB2rhxo9577z1FRkZq0qRJWrFihfr3759u8wEAAABgXprv8pSSK1euaOnSpSpWrFh6\nTM7mueeeU/bs2TVjxgwtW7ZMJUqU0OjRo9WxY8d0nQ8AAAAAcxwOFC1atEjxuoW4uDidP39ed+7c\n0YgRI9K1OElq1aqVWrVqle7TBQAAAPDoHA4UhQsXTrHdzc1NXl5eCgoKkp+fX3rVBQAAACATcDhQ\nLFiwICPrAAAAAJAJpdtF2QAAAAAePw6foQgICLC7hsIwDElK1pb0d9L/N2zYkF61AgAAAHAxDgeK\ntm3bauXKlTp79qx8fX1Vrlw55ciRQ2fOnNHmzZtlsVgUGBho95q0PnwOAAAAQObicKDIlSuXbty4\noaVLl6patWp2w86cOaOQkBBVqFBBAwYMSPciAQAAALgmh6+hmD9/vrp3754sTEhSqVKl1K1bN335\n5ZfpWhwAAAAA1+ZwoLh165ayZUv9hEZMTIxu376dLkUBAAAAyBwcDhQ+Pj6aO3eufvnll2TD9uzZ\no7lz58rf3z9diwMAAADg2hy+hmLo0KHq3Lmz2rZtK29vb5UqVUqGYejkyZP66aefVLZsWQ0bNiwj\nawUAAADgYhwOFOXLl9fq1asVFhamrVu36vDhw7JYLCpTpoxefvllvfjii8qTJ09G1goAAADAxTgc\nKCSpaNGievvtt/X2229nVD0AAAAAMpE0BQpJ2rVrl7Zs2aLz58+rX79+ypUrl/bt26cWLVooe/bs\nGVEjAAAAABflcKBISEjQkCFDtHbtWtsD6zp27Khr165p6NCh+vLLLzVr1izly5cvw4oFAAAA4Foc\nvsvTjBkz9M0332jkyJFav369DMOQJAUGBmrEiBE6dOiQPvnkkwwrFAAAAIDrcThQLF++XO3bt1dI\nSIhy585ta8+ePbu6du2qTp06KSIiIkOKBAAAAOCaHA4UFy5ckKenZ6rDK1asqIsXL6ZLUQAAAAAy\nB4cDRfHixXXkyJFUh+/Zs0fFixdPl6IAAAAAZA4OB4p27dopPDxcq1atUmJioq39r7/+0ieffKL/\n/ve/CgoKypAiAQAAALgmh+/y1Lt3bx09elRDhw5Vtmx3X/bGG2/o+vXrSkhIUOPGjdWvX78MKxQA\nAACA63E4UGTLlk2TJk1Shw4dFBERoVOnTikxMVElSpSQv7+/AgMDM7JOAAAAAC7I4UAxePBgNW/e\nXE2bNpWvr29G1gQAAAAgk3D4GorvvvtOFy5cyMhaAAAAAGQyDgeKypUr6/DhwxlZCwAAAIBMxuEu\nT88995wmTZqk3377TbVr19aTTz4pi8WSbLzevXuna4EAAAAAXJfDgeL999+XJB06dEiHDh1KdTwC\nBQAAAPD4cDhQREREZGQdAAAAADKhVAOFh4eHJkyYYHtYXalSpSRJN2/eVK5cueTu7v73VAgAAADA\nZTl8UbYkXb58WXXq1NHu3bszqh4AAAAAmUiaAgUAAAAA3ItAAQAAAMA0AgUAAAAA0wgUAAAAAEx7\n4G1jr1y5onPnztn+vnbtmiTpzz//tGu/V8mSJdOxPAAAAACu7IGBYuzYsRo7dmyy9sGDB6c4vsVi\nUVRUVPpUBgAAAMDlpRoo+vfvn+aJWSyWRyoGAAAAQOaSaqB45ZVX/s46AAAAAGRCXJQNAAAAwDQC\nBQAAAADTCBQAAAAATCNQAAAAADCNQAEAAADANAIFAAAAANMIFAAAAABMI1AAAAAAMI1AAQAAAMA0\nAgUAAAAA0wgUAAAAAEwjUAAAAAAwjUABAAAAwDQCBQAAAADTCBQAAAAATCNQAAAAADCNQAEAAADA\nNAIFAAAAANMIFAAAAABMI1AAAAAAMI1AAQAAAMA0AgUAAAAA0wgUAAAAAEwjUAAAAAAwjUABAAAA\nwDQCBQAAAADTCBQAAAAATCNQAAAAADCNQAEAAADANAIFAAAAANMIFAAAAABMI1AAAAAAMC3TBIq4\nuDi1aNFCw4cPd3YpAAAAAP5fpgkUn3zyiU6cOOHsMgAAAADcI1MEip9//lkLFixQwYIFnV0KAAAA\ngHu4fKC4c+eO3nrrLYWGhqpYsWLOLgcAAADAPVw+UISFhSkhIUF9+vSRYRjOLgcAAADAPbI5u4AH\nOXbsmGbOnKl58+Ype/bszi4HAAAAwH1cNlAkJibq7bffVocOHeTl5SVJslgspqcXFRWVXqXBRcTG\nxkpi3WZFrFvXVqVKFWeXgDT4u75HfG+zLtZt1pW0bh+VywaKBQsW6Pz58woLC9OdO3ckSYZhyDAM\nJSQkyN3d3ckVAgDg+rJiAGTHFnAtLhsoIiIidP78efn4+Ni1HzlyRCtWrNDGjRtVsmRJh6eXFTeo\nj7ukHxTWbdbDugXST9Cglc4uIV2tntSGbcPfjG1y1hUVFaWYmJhHno7LBorRo0fbvUHDMDR48GCV\nK1dOAwYMUJEiRZxYHQAAAADJhQNFuXLlkrXlzJlT//jHP1StWjUnVAQAAADgfi5/29h7PcpF2QAA\nAADSn8ueoUjJihUrnF0CAAAAgHtkqjMUAAAAAFwLgQIAAACAaQQKAAAAAKYRKAAAAACYRqAAAAAA\nYBqBAgAAAIBpBAoAAAAAphEoAAAAAJhGoAAAAABgGoECAAAAgGkECgAAAACmESgAAAAAmEagAAAA\nAGAagQIAAACAaQQKAAAAAKYRKAAAAACYRqAAAAAAYBqBAgAAAIBpBAoAAAAAphEoAAAAAJhGoAAA\nAABgGoECAAAAgGkECgAAAACmESgAAAAAmEagAAAAAGAagQIAAACAaQQKAAAAAKYRKAAAAACYRqAA\nAAAAYBqBAgAAAIBpBAoAAAAAphEoAAAAAJhGoAAAAABgGoECAAAAgGkECgAAAACmZXN2AQBwvypV\nqji7BAAA4CACBZBJBQ1a6ewS0tXqSW2y3HuS7r4vAOmLbQXgWujyBAAAAMA0AgUAAAAA0wgUAAAA\nAEwjUAAAAAAwjUABAAAAwDQCBQAAAADTCBQAAAAATCNQAAAAADCNQAEAAADANAIFAAAAANMIFAAA\nAABMI1AAAAAAMI1AAQAAAMA0AgUAAAAA0wgUAAAAAEwjUAAAAAAwjUABAAAAwDQCBQAAAADTCBQA\nAAAATCNQAAAAADCNQAEAAADANAIFAAAAANMIFAAAAABMI1AAAAAAMI1AAQAAAMA0AgUAAAAA0wgU\nAAAAAEwjUAAAAAAwjUABAAAAwDQCBQAAAADTCBQAAAAATCNQAAAAADDNpQNFYmKiPv/8c7Vo0UI1\na9ZUq1attHDhQmeXBQAAAOD/ZXN2AQ8ybdo0hYWFqX///vLy8tKePXs0duxYxcbGKjQ01NnlAQAA\nAI89lw0UCQkJmjt3rkJDQ9W3b19JUv369XX58mXNmTOHQAEAAAC4AJft8nTr1i21bdtWzzzzjF17\n2bJldfnyZd2+fdtJlQEAAABI4rJnKPLnz68RI0Yka9+0aZNKlCihJ554wglVAQAAALiXy56hSMlX\nX32lyMhIujsBAAAALsJlz1Dcb9WqVXrvvffUvHlzhYSEpPn1UVFRGVAVnCk2NlbS47luq1Sp4uwS\nkAZZ7TPK5w/IGK66rXicf2+zuqR1+6gyRaD4/PPPNX78eAUGBmrixInOLgeZCDs+AIDMIiv+ZhFC\nHg8uHygmT56sWbNmqW3btvrggw/k5maul1ZW/JI+7pI2Ug9bt0GDVv4d5fytVk9q4+wSkAZsfwA4\nIqv9Xq2e1Ibtn4uLiopSTEzMI0/HpQPFvHnzNGvWLHXv3l3Dhw93djkAAAAA7uOygeLixYuaOHGi\nKleurJYtW2r//v12wz09PeXu7u6k6gAAAABILhwotm3bpvj4eP32228KDg62G2axWBQZGal//OMf\nTqoOAAAAgOTCgaJdu3Zq166ds8sAAAAA8ACZ6jkUAAAAAFwLgQIAAACAaQQKAAAAAKYRKAAAAACY\nRqAAAAAAYBqBAgAAAIBpBAoAAAAAphEoAAAAAJhGoAAAAABgGoECAAAAgGkECgAAAACmESgAAAAA\nmEagAAAAAGAagQIAAACAaQQKAAD+r717D4ui3v8A/l5d8IJQZpaY92PuysUFRDEVRFC8IKVBLpRW\ndijTfmmlBnhJjcdCyDLFOmJKKsejJJIpGoGGolmSinQ8aGpxldRC8QICC9/fH56d4wgoOwoL9X49\nD0/t15nZ987Hxf3szHeGiIgUY0NBRERERESKsaEgIiIiIiLF2FAQEREREZFibCiIiIiIiEgxNhRE\nRERERKQYGwoiIiIiIlKMDQURERERESnGhoKIiIiIiBRjQ0FERERERIqxoSAiIiIiIsXYUBARERER\nkWJsKIiIiIiISDE2FEREREREpBgbCiIiIiIiUowNBRERERERKcaGgoiIiIiIFGNDQUREREREirGh\nICIiIiIixdhQEBERERGRYmpzB6CmoaKyCpWGanPHMIntY90BANfLKutcxqqNRWPFISIiIvpLYkNB\nAACVSoWFaw6hvKLK3FHuG+8BXTF+WG9zxyAiIiL6U2NDQZK8366irNxg7hj3TfGVcnNHIAIA+M3a\nbu4I99WOZU+ZOwIRETUhnENBRERERESKsaEgIiIiIiLF2FAQEREREZFibCiIiIiIiEgxNhRERERE\nRKQYGwoiIiIiIlKMDQURERERESnGhoKIiIiIiBRjQ0FERERERIqxoSAiIiIiIsXYUBARERERkWJs\nKIiIiIiISDE2FEREREREpBgbCiIiIiIiUowNBRERERERKcaGgoiIiIiIFGNDQUREREREirGhICIi\nIiIixdhQEBERERGRYmwoiIiIiIhIMTYURERERESkGBsKIiIiIiJSjA0FEREREREpxoaCiIiIiIgU\nY0NBRERERESKsaEgIiIiIiLF2FAQEREREZFibCiIiIiIiEgxNhRERERERKQYGwoiIiIiIlKMDQUR\nERERESnGhoKIiIiIiBRr8g1FfHw8fHx8oNPpEBgYiMzMTHNHIiIiIiKi/2rSDUViYiIWLVqEp556\nCitXroS1tTX+/ve/o6CgwNzRiIiIiIgITbihEEJg5cqV0Ov1eO211+Dh4YFPP/0U7du3x+eff27u\neEREREREhCbcUOTm5uLcuXPw8vKSxtRqNTw9PZGenm7GZEREREREZNRkG4qcnBwAQPfu3WXjXbp0\nQX5+PoQQZkhFRERERES3arINxbVr1wAAVlZWsnErKytUV1ejtLTUHLGIiIiIiOgWanMHqIvxCIRK\npar1z1u0MK0Xys7OvudMf2aPP94H6+aP/FMd+bG0aGnuCERERH9p/PzVtJWVld2X7ahEE/0EmZaW\nhldffRUpKSno2rWrNP75558jKioKJ06cqPe2jhw50hARiYiIiIiavf79+9/T+k32CIVx7kR+fr6s\nocjPz0fPnj1N2ta97iQiIiIiIqpdk51D0aNHD9ja2iIlJUUaq6ysRFpaGgYNGmTGZEREREREZNRk\njxSobEUAABqcSURBVFCoVCq8/PLLCA8Ph42NDVxcXBAXF4eSkhK8+OKL5o5HRERERERownMojGJj\nY7FhwwZcunQJffv2RWhoKHQ6nbljERERERERmkFDQURERERETVeTnUNBRERERERNHxsKIiIiIiJS\njA0FEREREREpxoaCiIiIiIgUY0NBRERERESKNfuGIj4+Hj4+PtDpdAgMDERmZuYdl9+/fz/8/f3h\n7OyMUaNGIS4urpGSkqlMre2toqOjodVqGzAd3QtTa/vqq69Cq9XW+CkrK2ukxFRfpta2uLgYb7/9\nNtzc3DBgwABMmzYN+fn5jZSWTGFKbb28vGp9z2q1WqxataoRU1N9mPq+zcrKwqRJk9C/f3+MGDEC\n0dHRMBgMjZSWTGFqbXft2gU/Pz/069cPo0aNwsaNG+v3RKIZ27Ztm+jbt6+Ijo4W+/btE8HBwcLF\nxUXk5+fXuvzRo0eFnZ2dCAsLE999951Ys2aNsLe3F7GxsY0bnO7K1Nre6tSpU8Le3l5otdpGSEqm\nUlJbT09P8d5774njx4/LfqqrqxsxOd2NqbWtqKgQTz75pBgzZoz45ptvREpKivD19RWjRo0SFRUV\njZye7sTU2mZnZ8veq5mZmWLmzJnCxcVF/Prrr40bnu7I1NoWFhYKZ2dnERwcLA4ePCg2btwodDqd\niIiIaOTkdDem1jYpKUloNBrx+uuvi/T0dBEfHy+eeOIJERkZedfnarYNRXV1tRg+fLhYtGiRNFZZ\nWSm8vb1FeHh4revMmDFDjB8/XjYWGhoqRo4c2aBZyTRKamtkMBiEv7+/8PDwYEPRBCmpbUlJidBo\nNCI9Pb2xYpICSmobHx8vdDqdKCoqksays7OFu7u7OHHiRINnpvq5l9/JRllZWcLe3l5s27atoWKS\nAkpqu3btWtGvXz9RVlYmjX344YfCxcWlwfNS/Smp7bhx44Rer5eNpaamCjs7u7t+odtsT3nKzc3F\nuXPn4OXlJY2p1Wp4enoiPT291nXCwsKwbNky2ZiFhQUqKysbNCuZRkltjT7//HOUlZVh0qRJELxn\nY5OjpLanTp0CAPTp06dRMpIySmqbmpoKDw8PdOrUSRrTarXYv38/7OzsGjwz1c+9/E42WrJkCfr1\n64cJEyY0VExSQEltr169CrVajVatWkljDzzwAEpLS1FRUdHgmal+lNQ2JycHQ4cOlY25uLigqqoK\nhw4duuPzNduGIicnBwDQvXt32XiXLl2Qn59f64fJTp06oVevXgCAK1eu4Msvv8T27dsRGBjY4Hmp\n/pTUFrj55omOjkZ4eDgsLCwaOiYpoKS2p06dgqWlJZYvXw43Nzc4OTlh5syZ+P333xsjMtWTktr+\n/PPP6NmzJ6KjozFkyBA4Ojpi6tSpKCoqaozIVE9KfycbpaamIjMzEyEhIQ0VkRRSUtvRo0ejsrIS\ny5YtQ0lJCbKysrB+/XqMHDkSlpaWjRGb6kFJbW1tbVFYWCgbKygokP23Ls22obh27RoAwMrKSjZu\nZWWF6upqlJaW1rluYWEhBg4ciNDQUPTp04cNRROjpLZCCMyfPx/jx4+Hi4tLo+Qk0ymp7alTp1BR\nUQFra2usWrUKCxcuRGZmJl544QV+G9aEKKntH3/8gYSEBBw4cADvvfceIiMjcebMGbzyyiuoqqpq\nlNx0d/fy7y0ArF+/Hq6urtDpdA2WkZRRUluNRoPw8HDExsbCzc0NEydOxMMPP4z33nuvUTJT/Sip\n7VNPPYWvvvoK8fHxKCkpwcmTJ7F48WJYWFjc9SIozbahMHZWKpWq1j9v0aLul2ZtbY0NGzZI3bVe\nr8eNGzcaJCeZTkltN2/ejPz8fMyePbtBs9G9UVLbKVOmIC4uDmFhYXB1dcWECROwcuVKnD17Frt3\n727QvFR/SmprMBhgMBjw2WefYdiwYRgzZgw+/vhjnD59Gt98802D5qX6u5d/b3/55RdkZGTg+eef\nb5BsdG+U1Pbbb7/FvHnzEBAQgPXr1yMyMhIlJSWYOnUqv+RpQpTUdurUqQgMDMSiRYvg5uaGyZMn\nIzAwEG3btkWbNm3u+HzNtqGwtrYGAFy/fl02fv36dbRs2fKOL9zGxgYDBw6Er68voqOjkZOTg6+/\n/rpB81L9mVrboqIiREVFYe7cuWjVqhUMBoP0RqqqquJciiZEyfu2V69ecHV1lY3169cPNjY20vwK\nMj8ltbWysoJOp0O7du2kMQcHB9jY2OD06dMNG5jq7V7+vd2zZw+srKzg6enZkBFJISW1XbZsGYYO\nHYrFixfDzc0NTz75JGJiYnDkyBHs2LGjUXLT3SmprVqtxoIFC3DkyBEkJSXh4MGD8PX1RUlJCR54\n4IE7Pl+zbSiM54Tdfr3y/Px89OzZs9Z1UlNT8dNPP8nGHn/8cajValy8eLFhgpLJTK3toUOHUFpa\nihkzZsDBwQEODg5YunQpAMDe3p7XPG9ClLxvk5KS8OOPP8rGhBCoqKhA+/btGyYomUxJbbt161br\nN5oGg6HOb9Wo8SmprVF6ejo8PDx4bn0TpaS2ubm5NU5f69WrFx588EGcPXu2YYKSyZTUNiMjA4cP\nH0abNm3wt7/9DZaWljh58iQAoG/fvnd8vmbbUPTo0QO2trZISUmRxiorK5GWloZBgwbVuk5MTAwi\nIyNlY99//z0MBgOvINOEmFpbLy8vJCQkyH6mTJkCAEhISMDEiRMbLTvdmZL37aZNm7BkyRLZkaZ9\n+/bhxo0bGDBgQINnpvpRUtuhQ4fi6NGjuHDhgjR2+PBhlJaWwtnZucEzU/0oqS1ws/E/ceIE5040\nYUpq26VLFxw9elQ2lpubi8uXL6NLly4NmpfqT0ltd+7cifDwcNlYXFwcHnzwwbv+Tm65aNGiRfec\n2gxUKhUsLS3xySefoLKyEhUVFXj//feRk5ODiIgI2NjYIC8vD7/++qt0ScKHH34YMTExuHDhAlq3\nbo309HS8++670Ol0eOONN8z8isjI1Nq2bt0ajzzyiOznzJkzOHDgAN59990aE5LIfJS8bzt27IjY\n2Fjk5OSgXbt2SE9Px5IlS+Dp6Sk1jmR+Smqr0Wiwbds2pKamomPHjjhx4gQWLlwIrVaLN99808yv\niIyU1Ba4eQGUtWvXYvLkyejRo4f5XgDVSUltbWxssHbtWvz2229o06YNjh07hgULFsDa2lqawEvm\np6S2jz76KGJiYlBcXAxLS0usX78eCQkJmDdv3t2/GDDpLhlN0Lp164Snp6fQ6XQiMDBQZGZmSn8W\nEhJS4+Zme/bsEf7+/kKn0wl3d3cREREhbty40dixqR5Mre2tYmNjeWO7Jkzp+9bJyUm4u7uLpUuX\nivLy8saOTfVgam3z8vLE9OnThbOzsxg4cKAIDQ0VV69ebezYVA+m1vb48eNCq9WKo0ePNnZUMpGp\ntU1LSxN6vV64uLgIT09PMW/ePPHHH380dmyqByX/3vr5+QmdTif8/PzEV199Va/nUQnBGatERERE\nRKRMs51DQURERERE5seGgoiIiIiIFGNDQUREREREirGhICIiIiIixdhQEBERERGRYmwoiIiIiIhI\nMTYURERERESkGBsKIqLbJCUlQavVYsKECeaO0uzl5+fLHmu1WixatMg8YZqJ0NBQ9OvXTzZ24cIF\nlJeXS4+9vLwQHBzc2NGIiGrFhoKI6DY7d+5EmzZtkJ2djdOnT5s7TrO1detWPP3007KxqKgo+Pv7\nmylR8xAYGIiIiAjp8b59+zB27Fhcu3ZNGps7dy5efvllc8QjIqqBDQUR0S2uXLmCAwcOICgoCCqV\nComJieaO1Gz9+OOPqKiokI35+fnB0dHRTImaBycnJ4wdO1Z6nJWVJWsmAGDEiBFwc3Nr7GhERLVi\nQ0FEdIvk5GRUVlbCx8cHDg4O2LFjB6qrq80dq9kSQpg7wp8G9yURNVVsKIiIbpGUlAQrKys4ODjA\ny8sLFy9exMGDBwEAR44cgVarRXx8fI319Hq9bM5Ffn4+3nzzTbi5ucHJyQlBQUE4dOiQbB0vLy+8\n++67mDVrFhwdHTFq1ChUVlaioqIC0dHR8PX1hU6ng7OzM/R6PdLS0mTrV1dXY/Xq1fD29oZOp8Nz\nzz2H7Oxs2NnZITo6WlrOYDDg008/xciRI+Ho6IgRI0Zg1apVqKqquuO+CA0Nxfjx47Fu3Tq4uLhg\n0KBB+M9//gMA2LFjBwIDA9G/f384Ojpi9OjR+Oyzz6R1J0+ejC+//BIVFRXQarVSHq1Wi4ULFwIA\nCgoKoNVqsWvXLkRERGDIkCHQ6XR44YUXcPLkSVmWy5cvY/78+Rg8eDBcXFwwa9YspKamQqvVIiMj\n446v4dlnn0VaWhrGjh0LnU6HCRMmIDU1tcayP/zwAyZNmgRnZ2cMHDgQM2bMkM0BMeaNi4tDQEAA\n+vXrh9mzZ9f53OXl5fjoo4/g5eUFJycn+Pn5ISEhQfrzlStXYsCAAdixYwfc3NwwYMAA7N27VzaH\nIjQ0FKtWrQIADB06FGFhYQBqn0Oxd+9eBAYGwtnZGR4eHnjnnXdw+fLlOvMREd0vbCiIiP7r4sWL\nOHz4MNzd3aFWq+Ht7Q0A+PLLLwEA/fv3R+fOnZGcnCxbr6ioCFlZWRg3bpz0WK/XIysrC8HBwXjr\nrbdgMBgQHBxcoylITEzEb7/9hgULFiAoKAgWFhYIDQ3F6tWrpQ+FwcHBKCwsxGuvvYZff/1VWvf9\n99/HRx99BCcnJ4SEhMDa2hrPP/98jW+yQ0JCsGrVKri7u2P+/PkYNGgQoqOjMWfOnLvuk9zcXMTF\nxWHWrFl45plnoNFosHnzZsyZMwe2trYIDQ3F7Nmz0bZtW3zwwQf44osvAADTpk2Dq6sr1Go1oqKi\n4OPjI21TpVLJniMqKgqHDx/GtGnTMHXqVGRlZWHq1KnSkSGDwYCXXnoJ27dvx4QJEzBz5kycPn0a\n8+bNq7Gt26lUKuTl5WHGjBkYMGAA5syZgxYtWuD111+X1XHfvn146aWXAACzZ8/Giy++iGPHjkGv\n16OoqEi2zWXLlqFPnz4ICQnB6NGj63zuadOmISYmBkOGDMHcuXPRvXt3zJs3D1u2bJGWKSsrQ0RE\nBKZNm4Znn30Wzs7Osn0UGBiIkSNHAgDeeecdBAYG1roft2/fjunTp6OqqgpvvfUWAgICsGPHDkyf\nPp1HNoio4QkiIhJCCLF+/Xqh0WjEzp07pbGRI0cKnU4nrl69KoQQIioqStjb24vLly9Ly8TGxgqt\nViuKioqEEELMnj1bDBkyRFy6dElaprKyUuj1euHt7S2NDR8+XDg6OoqSkhJp7Pz580Kr1YpPP/1U\nlu3AgQNCo9GITZs2CSGEyM3NFX379hULFy6ULTdz5kyh0WjEypUrhRBCfPfdd0Kj0Yjt27fLlouL\nixMajUZ8//33de6PkJAQodFoRFpammx8zJgxYsqUKbKxa9euCUdHR/HGG2/I1nd0dJQtp9FopMz5\n+flCo9EIHx8fUVFRIS0TExMjNBqNyMjIEEIIsXXrVqHRaERSUpK0zPXr14WXl5fQaDTi8OHDd30N\n69atk8Zu3LghfHx8pFoYDAYxfPhw8dJLL8nWPX/+vOjfv78ICQmR5Q0ICKjz+Yz27t0rNBqN2LBh\ng2x80qRJ0vOuWLFCaDQaERcXVyPzrfvNuNzvv/8ujQ0fPlwEBwdL+Z944gmh1+tFZWWltMzWrVuF\nVqu94/4hIrofeISCiOi/du3aBQsLCwwbNkwaGzFiBG7cuIGvv/4aADBu3DgYDAbZKTO7d++Gi4sL\nOnXqhOrqauzduxdubm4QQqC4uBjFxcW4cuUKvLy8UFBQgDNnzkjr9u7dGzY2NtLjRx55BEeOHMGU\nKVOksaqqKumSoaWlpQBunt5SXV2NF154QfYajN+yG6WmpkKtVmPw4MFSluLiYgwbNgwqlarGEZPa\n9O/fX/b4q6++wooVK2RjFy9eRLt27aR8phg+fDgsLCykx1qtFgDwxx9/AAD27NmDjh07yiYqt23b\nFkFBQfXavpWVFZ577jnpcatWrRAUFISCggKcPn0a2dnZOHfuHLy8vGT7SK1Ww9XVtcY+un1/1Gbf\nvn2wsLCAXq+XjS9duhSxsbGyMVdX13q9jrqcOHECxcXFeOaZZ6BWq6VxPz8/bNu2rcYlaImI7jf1\n3RchIvrzKygoQGZmJpycnFBSUiKde+7g4ADg5iklAQEB0Gq16NWrF5KTk+Hv7y+d7jR//nwAwKVL\nl3D9+nUkJSUhKSmpxvOoVCoUFRWhd+/eAID27dvXWEatVmP79u04cOAAfvnlF+Tl5UkNhfE0oLy8\nPKhUKnTt2lW2bs+ePWWP8/LyYDAYMHTo0FqznD9//o77xcLCAu3atauR79ixY9i1axfOnj2LnJwc\nXLlyRZbPFA899JDssaWlJQBIczzy8vLQrVu3Guv16NGjXtvv2rWrtE0j4/YKCwulJig8PBzh4eE1\n1lepVLKrVd2etzbnzp1Dp06dajxv586dayxbn+3dSWFhIQCge/fusnFLS0v07dv3nrZNRFQfbCiI\niHDz6AQAZGZmSnMnbvXjjz+isLAQjz32GMaNG4dPPvkEV69eRXJyMlq0aIExY8YA+N+HYD8/vxr3\nYDDSaDTS/7doIT9QfOPGDQQFBeH06dMYPHgwvLy8oNVq8dhjj2HixInScgaDASqVSvaNNHDz2/db\nVVdXo3379vjwww9rzdKhQ4dax41qm6OwcOFCbNmyBTqdDjqdDhMnTsSAAQNkR1VMcbd5EAaDQXYE\nw+j211qX2/cR8L/Gp0WLFtL/z5kzB3Z2drVuo2XLlvXOC9z8eyDqOXfh9r8DpuJVyIjI3NhQEBHh\n5s3s1Go1PvjggxofXlNTU5GYmChNfPX19cWKFSuwf/9+fP311xg0aJD0LfNDDz2E1q1bo7q6Gk88\n8YRsO2fPnkVhYSHatGlTZ47du3cjOzsbH374oewUn8zMTNlyXbt2RXV1NfLz82VHKXJycmTL2dra\n4vvvv4eLi4vsA3hlZSX27NmDLl261G8H/VdBQQG2bNkCvV6PxYsXS+NVVVW4dOmSSduqr65du9Z6\ng8Hc3Nx6rV9QUAAhhKwRMO6nHj164OLFiwCAdu3a1ahZRkYGVCqVrKGoD1tbWxw+fBgVFRWyoxRp\naWlITk6WrtZ0P3Tq1AnAzSuL3Xr6VHl5Od5++234+/vDw8Pjvj0fEdHtOIeCiP7yzpw5g59//hnD\nhg3D6NGj4e3tLfv5v//7P6hUKmzfvh3AzVNLHBwckJiYiOPHj0tXdwJufhs+dOhQpKSkyD7cGwwG\nzJ07F2+99dYdv+E2nmrVq1cvaUwIgX/+858A/ncExNvbGyqVCps2bZKtb1zOaPjw4aiqqsKaNWtk\n41u2bMEbb7yBY8eO3XHf3J61pKSkRj4ASEhIQFlZmexStLd++38vRowYgd9++002l6GiogJbt26t\n1/qXL1/Gzp07pcdlZWX417/+hT59+qBbt25wdHREhw4dsGHDBunUMgA4f/48Xn31Vemyrabw9PRE\nZWWldIUwo/Xr1+O7776TzZupza373XgEo67L/Do6OqJ9+/ZISEiQ7e/k5GQkJyfXeoSGiOh+4m8Z\nIvrLM37YrOsUpcceewyDBw/GwYMHcezYMTg7O2PcuHGIiIhAq1atpMt6Gs2aNQs//PAD9Ho9Jk+e\njIceegi7d+/G8ePHsWDBArRu3brOLIMHD4Zarcbs2bMRFBQEIQR2796N33//HRYWFtIdk3v16gW9\nXo/Y2FhcvHgRzs7O+OGHH7Bv3z4A//tA6u3tDQ8PD0RHRyMnJweurq44c+YMNm/eDGdnZ+lUrbrc\nftrO448/DltbW3zyyScoLS1Fhw4dkJGRgb1796Jz586yOzp36NBBugfGkCFDFE8Ofvrpp7Fp0ybM\nnDkTkydPxqOPPorExETpErp3OwVJrVZjwYIFyM7OxqOPPopt27bhwoUL0n0zLC0tERYWhjlz5iAg\nIABPP/201MQZL8NqKm9vbwwaNAiLFy/GqVOn0Lt3b+zfvx+HDh3CsmXL7rr+rfvdeFramjVrpO3e\nytLSEm+//TbCwsIwefJkjBkzBhcuXMDGjRvh7u6OwYMHm5yfiMgUPEJBRH95u3fvRocOHeDp6Vnn\nMsar9RiPUowZMwYtWrSAu7t7jUnLPXv2xJYtW+Dm5oaNGzciKioKpaWl+OCDD2RXG6qNRqPB8uXL\n0bJlS0RGRmLNmjWwt7fHF198ATs7O9lN3BYsWIBp06YhIyMDS5cuxaVLl6S5EreethUdHY3p06fj\n+PHjWLJkCb799ls899xziImJqXVugpFKparxYd3S0hKrV6+GnZ0d1q5di6ioKJSXl2Pr1q3w9fXF\nyZMnpaZCr9fDzs4Oq1atQmJi4h1fd23PbWRhYYHY2Fj4+PggPj4ey5cvR58+fTBz5swar7U2HTp0\nwPLly7Fnzx589NFHsLGxQWxsLNzc3KRlxo0bh9WrV8Pa2horVqzA6tWr0bNnT2zYsAGOjo4mZTfm\n/8c//oHnn38eqampWLp0Kc6fP48VK1bA19dXWqa2Zuj28bFjx2LgwIHYvHlzjStEGU2YMAErVqxA\nWVkZIiMjsXPnTgQGBuLjjz82OTsRkalUor6zxoiIqMkoKyuDEAJt27aVjf/73/9GQEAAlixZAn9/\nfzOlu79KSkrQtm3bGo3DunXrEBkZiZSUlBpXuzIKDQ3FoUOHpCM3RER0//EIBRFRM5SVlQUXFxfZ\n/TAASPfLsLe3N0esBrFhwwa4uLiguLhYGquursY333yDBx98sM5mwqg+V2UiIiLlOIeCiKgZcnZ2\nRrdu3fDOO+/g559/RseOHZGVlYWEhAT4+vpKN4f7Mxg7dizWrFmDKVOmICAgAC1btkRKSgoyMzNl\nV5qqCw/EExE1LJ7yRETUTBnPyT948CCKi4vRuXNnjB8/Hq+88so939ugqcnKysLKlSvx008/oby8\nHH369MGUKVMwevToO64XFhaGQ4cO1euO4EREpAwbCiIiIiIiUuzP9RUWERERERE1KjYURERERESk\nGBsKIiIiIiJSjA0FEREREREpxoaCiIiIiIgUY0NBRERERESK/T8Y0wHzIZc4MwAAAABJRU5ErkJg\ngg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10a637a90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = critics.copy()\n",
    "df['fresh'] = df.fresh == 'fresh' # select all rowns where column \"fresh\" equals fresh\n",
    "# .groupby is offered by pandas: http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.groupby.html\n",
    "# returns a GroupBy object http://pandas.pydata.org/pandas-docs/stable/api.html#groupby\n",
    "grp = df.groupby('critic')\n",
    "counts = grp.critic.count()  # number of reviews by each critic\n",
    "means = grp.fresh.mean()     # average freshness for each critic: http://pandas.pydata.org/pandas-docs/stable/api.html#series\n",
    "#print counts\n",
    "#print means\n",
    "\n",
    "#print len(df['fresh'])\n",
    "\n",
    "# counts > 100 selects all critics with more than 100 reviews\n",
    "means[counts > 100].hist(bins=10, edgecolor='w', lw=1) # Draw histogram of the input series using matplotlib\n",
    "plt.xlabel(\"Average rating per critic\")\n",
    "plt.ylabel(\"Frequency\")\n",
    "plt.yticks([0, 2, 4, 6, 8, 10])\n",
    "plt.title(\"Average rating per critic for critics with > 100 reviews\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building and Testing The Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we create a [function](https://docs.python.org/2.7/tutorial/controlflow.html#defining-functions) that we will use below. Please note that the function returns two values $X$ and $y$ where $X$ contains the vector space representations of the critics' quotes (the *training vectors*) and $y$ indicates whether a quote is positive (encoded by 1) or negative (encoded by 0) (the *target values* or *classes*). This encoding trick is essential because you are defining the classes that the classifier will use later with the help of these numbers. Eventually, the classifier to be defined will take a look at the vector space representation of a review and predict whether it is positive or negative.\n",
    "\n",
    "In principle, you can define more than two classes. But in this tutorial, we are only interested in a classifier that can differentiate between positive and negative reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# beginning of the function\n",
    "def make_xy(critics, vectorizer=None):\n",
    "    # create a vectorizer as we did before if none was passed to the function  \n",
    "    if vectorizer is None:\n",
    "        vectorizer = CountVectorizer()\n",
    "    # fit and transform in a combined function call\n",
    "    X = vectorizer.fit_transform(critics.quote)\n",
    "    X = X.tocsc()  # some versions of sklearn return COO format, basically a way to tidy up your array and to bring it to the format you are expecting\n",
    "    y = (critics.fresh == 'fresh').values.astype(np.int) \n",
    "    #.values converts the return to an array, which is the converted to an integer with the effect that each TRUE will be stored as 1\n",
    "    return X, y\n",
    "# end of the function\n",
    "\n",
    "# here, we call our function\n",
    "X, y = make_xy(critics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next step, we need to choose an appropriate Naive Bayes classifier. In this case, we choose a [multinomial Naive Bayes classifier](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html) because we assume that our training data set follows a [multinomial distribution](https://en.wikipedia.org/wiki/Multinomial_distribution), i.e., a particular statistical property.\n",
    "\n",
    "In ML, you are typically also interested in how well your trained classifier will work with real data. To check whether your classifier works the way intended, you often use a technique called [cross-validation](https://en.wikipedia.org/wiki/Cross-validation_(statistics)). The idea behind cross-validation is pretty straight-forward. You take a subset from your data of which you already know the desired property (e.g., that it is a positive review), the so-called *training set*, and train your classifier on it. The classifier is then tested against a so-called *test set* (containing elements with a known property) in order to find out if your classifier predicts the desired property correctly. Typically, you repeat cross-validation multiple times with varying training and test set combination in order to avoid an effect called [overfitting](https://en.wikipedia.org/wiki/Overfitting). \n",
    "\n",
    "Roughly speaking, overfitting means that you \"over\"-train your classifier in a way that it will perfectly predict properties of your known training data. Unfortunately, you want to use your classifier with real-world data you have not seen before - otherwise you would already know that you are dealing with positive reviews or not. An overfitted classifier will hardly be able to predict the desired property in an unknown data set as it has been perfectly adjusted to a really small extract of reality and not the general case.\n",
    "\n",
    "The imported function *[train_test_split](http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.train_test_split.html)* basically creates random training and test sets from the data given. As random is involved in the process, the accuracy, i.e., how well your classifier works, will change everytime you run the next cell. For instance, for one run it may be 73.86% while it will be 74.07% for another."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MN Accuracy: 77.13%\n"
     ]
    }
   ],
   "source": [
    "# import the classifier\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "# import the training and test functionality\n",
    "from sklearn.cross_validation import train_test_split\n",
    "xtrain, xtest, ytrain, ytest = train_test_split(X, y)\n",
    "# the actual training happens here\n",
    "# we take the training features (our critics' quotes in the vector space model) and the target values (positive review or not)\n",
    "# and train our classifier\n",
    "classifier = MultinomialNB().fit(xtrain, ytrain) \n",
    "print \"MN Accuracy: %0.2f%%\" % (100 * classifier.score(xtest, ytest))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on training data: 0.9173\n",
      "Accuracy on test data:     0.7713\n"
     ]
    }
   ],
   "source": [
    "training_accuracy = classifier.score(xtrain, ytrain)\n",
    "test_accuracy = classifier.score(xtest, ytest)\n",
    "\n",
    "print \"Accuracy on training data: %0.4f\" % (training_accuracy)\n",
    "print \"Accuracy on test data:     %0.4f\" % (test_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-Validation and Hyper-Parameter Fitting\n",
    "\n",
    "Throughout the next cells, we will further tune our classifier. In priciple, we will make use of cross-validation with multiple iterations. *[KFold](http://scikit-learn.org/stable/modules/generated/sklearn.cross_validation.KFold.html)* offers this functionality right out of the box. In order to discover the best parameters for the classifier, we will run some tests. Please note that it is not necessary to understand every detail of the following cells to follow the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.765118166871\n"
     ]
    }
   ],
   "source": [
    "from sklearn.cross_validation import KFold # import KFold for cross-validation\n",
    "result = 0\n",
    "nfold = 5\n",
    "for train, test in KFold(y.size, nfold): # split data into train/test groups, 5 times\n",
    "    classifier.fit(X[train], y[train]) # fit\n",
    "    result += classifier.score(X[test], y[test]) # evaluate score function on held-out data\n",
    "print result / nfold # average"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Scoring TODO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def cv_score(classifier, X, y, scorefunc):\n",
    "    result = 0.\n",
    "    nfold = 5\n",
    "    for train, test in KFold(y.size, nfold): # split data into train/test groups, 5 times\n",
    "        classifier.fit(X[train], y[train]) # fit\n",
    "        result += scorefunc(classifier, X[test], y[test]) # evaluate score function on held-out data, scorefunc will be passed to the function as a parameter\n",
    "    return result / nfold # average"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the [log-likelihood](https://en.wikipedia.org/wiki/Likelihood_function) as the score here. We'll go into this later... [predict_log_proba](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html#sklearn.naive_bayes.MultinomialNB.predict_log_proba)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def log_likelihood(classifier, x, y):\n",
    "    prob = classifier.predict_log_proba(x)\n",
    "    rotten = y == 0\n",
    "    fresh = ~rotten\n",
    "    return prob[rotten, 0].sum() + prob[fresh, 1].sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\alpha$ : Additive (Laplace/Lidstone) smoothing parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#the grid of parameters to search over\n",
    "alphas = [0, .1, 1, 5, 10, 50] # .1 is a shortcut for 0.1\n",
    "min_dfs = [1e-5, 1e-4, 1e-3, 1e-2, 1e-1] # 1e-5 etc. is scientific notation, see https://en.wikipedia.org/wiki/Scientific_notation\n",
    "\n",
    "#Find the best value for alpha and min_df, and the best classifier\n",
    "best_alpha = None\n",
    "best_min_df = None\n",
    "maxscore=-np.inf\n",
    "for alpha in alphas:\n",
    "    for min_df in min_dfs:         \n",
    "        vectorizer = CountVectorizer(min_df = min_df)       \n",
    "        Xthis, ythis = make_xy(critics, vectorizer)\n",
    "        \n",
    "        #your code here\n",
    "        classifier = MultinomialNB(alpha=alpha)\n",
    "        cvscore = cv_score(classifier, Xthis, ythis, log_likelihood)\n",
    "\n",
    "        if cvscore > maxscore:\n",
    "            maxscore = cvscore\n",
    "            best_alpha, best_min_df = alpha, min_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "alpha: 5.000000\n",
      "min_df: 0.001000\n"
     ]
    }
   ],
   "source": [
    "print \"alpha: %f\" % best_alpha\n",
    "print \"min_df: %f\" % best_min_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with the Best Parameters\n",
    "\n",
    "After running the tests, we have finally discovered the best parameters for our data. Please note that we now call CountVectorizer with a mimimum document frequency, i.e., we restrict our index vocabulary to terms that occur at least *best_min_df* times in our input documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy on training data: 0.78\n",
      "Accuracy on test data:     0.74\n"
     ]
    }
   ],
   "source": [
    "vectorizer = CountVectorizer(min_df=best_min_df)\n",
    "X, Y = make_xy(critics, vectorizer)\n",
    "xtrain, xtest, ytrain, ytest = train_test_split(X, Y)\n",
    "\n",
    "classifier = MultinomialNB(alpha=best_alpha).fit(xtrain, ytrain)\n",
    "\n",
    "# Your code here. Print the accuracy on the test and training dataset\n",
    "training_accuracy = classifier.score(xtrain, ytrain)\n",
    "test_accuracy = classifier.score(xtest, ytest)\n",
    "\n",
    "print \"Accuracy on training data: %0.2f\" % (training_accuracy)\n",
    "print \"Accuracy on test data:     %0.2f\" % (test_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Confusion matrix](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html) is a means to visualize the performance of your algorithm. Each column of the matrix displays the elements in a predicted class while each row shows how many elements are in an actual class. This may sound confusing but can be made clear by an example.\n",
    "\n",
    "As said above, we have two classes (positive or negative review). Hence, we have two rows and two columns. The first column represents the first class (negative reviews) and the second column represents the second class. The same pattern works for the rows. This will give us the following matrix.\n",
    "\n",
    "$$\n",
    "\\begin{matrix}\n",
    "& negative & positive \\\\\n",
    "negative & a & b \\\\\n",
    "positive & c & d\\\\\n",
    "\\end{matrix}\n",
    "$$\n",
    " \n",
    "If we now sum up the values $a$ and $b$ in the $negative$ row, we will find out how many actual negative reviews are in our data. If we inspect $a$, we can see how many of these were classfied as negative reviews while $b$ shows us how many were predicted (wrongly) as positive. If $a/b$ and $c/d$ are fairly equal, we can infer that the classifier has problems to differentiate between the tow classes. In an optimal world, there should only be values on the diagonal, i.e., $a$ and $d$, as the diagonal contains correct classification.\n",
    "\n",
    "For a more detailed description, refer to this [article](https://en.wikipedia.org/wiki/Confusion_matrix)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 902  553]\n",
      " [ 446 1990]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "cmatrix=confusion_matrix(ytest, classifier.predict(xtest))\n",
    "print cmatrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpreting the Resulting Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Good words\t     P(fresh | word), i.e., the probability a review is positive given the word...\n",
      "         masterpiece 0.93\n",
      "             delight 0.89\n",
      "         intelligent 0.88\n",
      "               witty 0.88\n",
      "              superb 0.87\n",
      "            touching 0.87\n",
      "        entertaining 0.86\n",
      "          remarkable 0.86\n",
      "             kubrick 0.86\n",
      "             perfect 0.85\n",
      "Bad words\t     P(fresh | word), i.e., the probability a review is positive given the word...\n",
      "                save 0.23\n",
      "      disappointment 0.21\n",
      "          uninspired 0.20\n",
      "               fails 0.20\n",
      "             unfunny 0.19\n",
      "           pointless 0.18\n",
      "               bland 0.17\n",
      "                dull 0.17\n",
      "       unfortunately 0.17\n",
      "                lame 0.16\n"
     ]
    }
   ],
   "source": [
    "words = np.array(vectorizer.get_feature_names())\n",
    "\n",
    "x = np.eye(xtest.shape[1])\n",
    "probs = classifier.predict_log_proba(x)[:, 0]\n",
    "ind = np.argsort(probs)\n",
    "\n",
    "good_words = words[ind[:10]] # the 10 best words (starting from the beginning of the array)\n",
    "bad_words = words[ind[-10:]] # the 10 worst words (counting backwards from the end of the array)\n",
    "\n",
    "good_prob = probs[ind[:10]] # the probabilites for the words chosen above\n",
    "bad_prob = probs[ind[-10:]]\n",
    "\n",
    "print \"Good words\\t     P(fresh | word), i.e., the probability a review is positive given the word...\"\n",
    "# the zip function aggregates the elements fro good_words and good_prob\n",
    "for w, p in zip(good_words, good_prob):\n",
    "    print \"%20s\" % w, \"%0.2f\" % (1 - np.exp(p)) # the 1-np.exp(p) becomes necessary because we used predict_log_proba() above, in other words: we execute a mathematical transformation to get readable results\n",
    "    \n",
    "print \"Bad words\\t     P(fresh | word), i.e., the probability a review is positive given the word...\"\n",
    "for w, p in zip(bad_words, bad_prob):\n",
    "    print \"%20s\" % w, \"%0.2f\" % (1 - np.exp(p))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.1453868   0.0920968   0.06990596 ...,  0.54297715  0.09994146\n",
      "  0.74958823]\n",
      "15561\n",
      "Mis-predicted Rotten quotes\n",
      "---------------------------\n",
      "The Waterboy is arguably Sandler's most enjoyable motion picture to date, but it's still far from a masterpiece.\n",
      "\n",
      "This pacifist spirit of brotherhood echoes the heroics in Princess Mononoke and other anime titles, but the artistic gap between the Miyazaki masterpiece and this project is huge.\n",
      "\n",
      "The plot of The Glimmer Man involves not only the Family Man but Our Evil Secret Government, the Russian Mafia and Rich Powerful Politicians -- the three stooges of action cinema in the '90s.\n",
      "\n",
      "Walken is one of the few undeniably charismatic male villains of recent years; he can generate a snakelike charm that makes his worst characters the most memorable, and here he operates on pure style.\n",
      "\n",
      "It survives today only as an unusually pure example of a typical 50s art-film strategy: the attempt to make the most modern and most popular of art forms acceptable to the intelligentsia by forcing it into an arcane, antique mold.\n",
      "\n",
      "Mis-predicted Fresh quotes\n",
      "--------------------------\n",
      "Next Friday is an extremely funny movie, and this is coming from someone who barely cracked a smile during Friday, the first installment of this franchise.\n",
      "\n",
      "There's too much talent and too strong a story to mess it up. There was potential for more here, but this incarnation is nothing to be ashamed of, and some of the actors answer the bell.\n",
      "\n",
      "The gangland plot is flimsy (bad guy Peter Greene wears too much eyeliner), and the jokes are erratic, but it's a far better showcase for Carrey's comic-from-Uranus talent than Ace Ventura.\n",
      "\n",
      "Though it's a good half hour too long, this overblown 1993 spin-off of the 60s TV show otherwise adds up to a pretty good suspense thriller.\n",
      "\n",
      "Some of the gags don't work, but fewer than in any previous Brooks film that I've seen, and when the jokes are meant to be bad, they are riotously poor. What more can one ask of Mel Brooks?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "x, y = make_xy(critics, vectorizer)\n",
    "\n",
    "#Returns the probability of the samples for each class in the model. The columns correspond to the classes in sorted order, as they appear in the attribute classes\n",
    "prob = classifier.predict_proba(x)[:, 0]\n",
    "print prob\n",
    "print len(prob)\n",
    "predict = classifier.predict(x)\n",
    "\n",
    "bad_rotten = np.argsort(prob[y == 0])[:5]\n",
    "bad_fresh = np.argsort(prob[y == 1])[-5:]\n",
    "\n",
    "print \"Mis-predicted Rotten quotes\"\n",
    "print '---------------------------'\n",
    "for row in bad_rotten:\n",
    "    print critics[y == 0].quote.irow(row)\n",
    "    print\n",
    "\n",
    "print \"Mis-predicted Fresh quotes\"\n",
    "print '--------------------------'\n",
    "for row in bad_fresh:\n",
    "    print critics[y == 1].quote.irow(row)\n",
    "    print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To find out whether a review is considered positive or not by the classifier, we will have to check the returned array of the function *[predict_proba](http://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html#sklearn.naive_bayes.MultinomialNB.predict_proba)*. The returned array contains the probabilities an element belongs to a certain class. The array is sorted by the class IDs, i.e., the value at index 0 displays the probability that a review is negative. The higher a probability score is, the more likely the element belongs to the considered class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is a good review! [ 0.01615328  0.98384672]\n"
     ]
    }
   ],
   "source": [
    "# check if the classifier will predict right\n",
    "result=classifier.predict_proba(vectorizer.transform(['This movie is not remarkable, touching, or superb in any way']))\n",
    "if result[0][1]>result[0][0]:\n",
    "    print \"This is a good review! \"+ str(result[0])\n",
    "else:\n",
    "    print \"This is a bad review! \"+ str(result[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obviously, the classification above returned a false positive. This is due to the vector space model considering all words separately. As a result, the negation via the word \"not\" was not recognized. Hence, \"remarkable\", \"touching\", and \"superb\" persuaded the classifier that it must be a positive review as it had no information about the semantic shift because of the present \"not\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is a bad review! [ 0.93298369  0.06701631]\n"
     ]
    }
   ],
   "source": [
    "# check if the classifier will predict right\n",
    "result=classifier.predict_proba(vectorizer.transform(['lame and inept.']))\n",
    "if result[0][1]>result[0][0]:\n",
    "    print \"This is a good review! \"+ str(result[0])\n",
    "else:\n",
    "    print \"This is a bad review! \"+ str(result[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What to Do Next?\n",
    "\n",
    "* Instead of the CountVectorizer, you might want to try [$tf * idf$](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html#sklearn.feature_extraction.text.TfidfVectorizer) on your textual data as it might give you better results. Roughly speaking, it acknowledges the fact that some words occur more often than others in your document collection.\n",
    "*  For visual learners, there a video called [\"An End-to-End Machine Learning Ecosystem in a Quarter\" by Chris Hausler](https://www.youtube.com/watch?v=of6pHdSbtOM) which gives you a good overview on building your own ML architecture.\n",
    "* To conclude, the lecture on ML by [Andrew Ng](https://itunes.apple.com/de/itunes-u/machine-learning/id384233048?mt=10) is probably one of the best lectures that you can find on ML on the internet.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}