Experiment.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:6f3f596951c6b868679ede9c16083eab2e7a6c88de79be3c0d741da1010e550c"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Multilabel Classification in the Dreambank"
     ]
    },
    {
     "cell_type": "heading",
     "level": 4,
     "metadata": {},
     "source": [
      "Folgert Karsdorp"
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Setup"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The Dreambank is a collection of transcribed dreams, that have been annotated using the Hall van de Castle System. In a set of 991 dreams, the dreambank has been annotated for the following information:\n",
      "\n",
      "- activities ($n=4814$);\n",
      "- characters ($n=2597$);\n",
      "- emotions ($n=696$);\n",
      "- social interactions ($n=1370$):\n",
      "    - aggression ($n=729$);\n",
      "    - friendliness ($n=550$);\n",
      "    - sexuality ($n=91$);\n",
      "- striving ($n=243$):\n",
      "    - succes ($n=113$);\n",
      "    - failure ($n=130$);\n",
      "- good fortune and misfortune:\n",
      "    - good fortune ($n=61$);\n",
      "    - bad fortune ($n=407$);\n",
      "- objects ($n=5044$);\n",
      "- settings (outdoor, etc.) ($n=1289$);\n",
      "\n",
      "In this notebook I give a description of the experiments I have done on automatically assigning some of these annotations to the Dreambank. All source files and scripts can be found in the same directory as this notebook. \n",
      "\n",
      "The purpose of automatic annotation is to assist dream researchers in annotating large collections of dreams. Therefore, I chose to use models from *Information Retrieval* that produce rankings of labels where the goal is to assign to the highest ranks, the most appropriate labels for a particular dream. \n",
      "\n",
      "I experimented with the following three models.\n",
      "\n",
      "#### Small Document IR model\n",
      "\n",
      "In the small document setup, queries are formed by all words of a dream. Queries are then matched against all individual dreams in the training data and ranked using Okapi BM25. After removing all duplicate labels, we evaluate the performance using mean average precision.\n",
      "\n",
      "#### Big Document IR model\n",
      "\n",
      "In the big document setup, again queries are formed by all words of a dream. Different from the small document setup is that for each unique label, we merge all documents into one big document that contain that label. These labels then form the class label of the big documents. We match each query against each big document and rank the big documents using BM25. Performance is meassured with mean average precision.\n",
      "\n",
      "#### Multi-Label IR model\n",
      "\n",
      "The Multi-Label IR model follows the same query mechanism as the other two models. The model is based on a proposal by Zhang & Zou (2007) ([click here](http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=7A0E2422C713672E81928AF14CAD3F82?doi=10.1.1.89.7148&rep=rep1&type=pdf) for the paper) that for each unseen dream, first identifies its $k$ nearest neighbors using the small document IR model. After that we apply the maximum a posteriori principle on the basis of statistical information of the label sets of these $k$ neighbors, to determine the label set of the unseen dream (see the complete algorithm below). \n",
      "\n",
      "![](ML-knn.png)\n",
      "\n",
      "We produce a ranking of labels on the basis the their posterior probability and again evaluate using mean average precision. "
     ]
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Experiments"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "First we import all required models, functions and classes:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import numpy as np\n",
      "from IRSystem import IRSystem, ML_IRSystem\n",
      "from metrics import mean_average_precision\n",
      "from dream_experiment import read_dreams, read_labels, match_labels_documents"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Next we read all dreams that have been annotated for lemmata and part of speech using the Stanford Core NLP toolkit:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "documents = {doc_id: text for doc_id, text in read_dreams(\"data/dreambank.en.stanford.out\", \n",
      "                                                          lemmata=True)}"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 2
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We'll store the results of all experiments in the following table:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import pandas as pd\n",
      "\n",
      "score_df = pd.DataFrame(columns=['fortune', 'actions', 'objects', 'emotions', \n",
      "                                 'settings', 'combined'],\n",
      "                        index=['small_doc', 'big_doc', 'ml'])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 152
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Good fortune / Misfortune"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In the first experiment we will look at the classifications of good versus misfortune. We read the label file and match the labels against the documents:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "labels_fortune = list(read_labels(\"data/labels.hcnorms_misgoodfortune.all.txt\"))\n",
      "labels_fortune, texts = zip(*match_labels_documents(documents, labels_fortune))\n",
      "labels_fortune, texts = np.array(labels_fortune), np.array(texts)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 250
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In this and all following experiments we make use of 10-fold cross-validation. We set the parameters of BM25 to $k1=1.2$ and $b=0.75$. \n",
      "\n",
      "Before we run the different models, let's first have a look at some of the statistics of the labels sets:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def flatten_labels(labels):\n",
      "    for i in labels:\n",
      "        if isinstance(i, list) or isinstance(i, tuple):\n",
      "            for j in flatten_labels(i):\n",
      "                yield j\n",
      "        else:\n",
      "            yield i"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print len(set(flatten_labels(list(labels_fortune))))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "7\n"
       ]
      }
     ],
     "prompt_number": 214
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "There are 7 different labels in the labels_fortune dataset. They follow the following distribution:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "label_counts = pd.Series(Counter(flatten_labels(list(labels_fortune))))\n",
      "label_counts.plot(kind='bar')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 216,
       "text": [
        "<matplotlib.axes._subplots.AxesSubplot at 0x119513a10>"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAAEICAYAAACtXxSQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEUlJREFUeJzt3X2QJHV9x/H3sstx4VzuwZqoSUhSKcM3arQuECOi3oMS\n9FALYxWoicH4GAs0iA8YLxYGg1GDQKSCeThiMPkjCnjRWBbGKkTOXKkkxNNcoV/RFKmUJXGT3ZXj\n7lZgb/PHzIXlvJ2dnp25nv7d+1W1tTs9092fmur9TPeve2bGFhYWkCQ12wl1B5AkrZxlLkkFsMwl\nqQCWuSQVwDKXpAJY5pJUgInlHhARzwA+kJlbI2IjcB0wD/wIuDAzfxARrwfeADwMXJmZnx1maEnS\no3XdM4+Iy4AdwEmdSX8KvCkztwI7gXdGxOOANwNnAc8H3h8Rq4YXWZJ0pOWGWb4DvBQY69x+eWZ+\no/P3icBB4NeA3Zn5UGbe35nnacMIK0k6uq5lnpk7aQ+dHL59H0BEnAVcDFwLnAL8cNFs+4C1A08q\nSVrSsmPmR4qIlwHbgXMz838j4n5gctFDJoGZbst4+OH5hYmJ8aqrlqTj3dhSd1Qq84h4Je0TnVsy\n83Bh3wm8LyJOAlYDTwL2dlvOzMyBKqutrNWaZGpq31DXMUzmr5f569Pk7DD8/K3W5JL39VrmCxFx\nAvBh4D+BnREB8MXMvCIirgO+RHvYZntmPriyyJKkKpYt88y8l/aVKgCPXeIxNwA3DC6WJKkK3zQk\nSQWwzCWpAJa5JBXAMpekAljmklQAy1ySClD5HaCqqvoXZs/Pz/cx35JvDJN0HLDMj4Frb9rD1Ozc\nUJbdWreaSy/YOJRlS2oOy/wYmJqd477pg3XHkFQwx8wlqQCWuSQVwDKXpAJY5pJUAMtckgpgmUtS\nASxzSSqAZS5JBbDMJakAlrkkFcAyl6QCWOaSVADLXJIKYJlLUgEsc0kqgGUuSQWwzCWpAJa5JBXA\nMpekAljmklSAZb/QOSKeAXwgM7dGxBOBG4FDwF7g4sxciIjXA28AHgauzMzPDjGzJOkIXffMI+Iy\nYAdwUmfSNcD2zNwEjAHnRcTjgTcDZwHPB94fEauGF1mSdKTlhlm+A7yUdnEDnJ6Zuzp/3wqcDTwd\n2J2ZD2Xm/Z15njaMsJKko+ta5pm5k/bQyWFji/7eB6wFTgF+eJTpkqRjZNkx8yMcWvT3KcAscD8w\nuWj6JDDTbSHr15/MxMR4xVVX02pNLv+gY2B+fn7o69iwYQ3j48N9Pqsalee/X+avT5OzQ335q5b5\n1yJic2beAWwDbgPuBN4XEScBq4En0T45uqSZmQP9ZO1ZqzXJ1NS+oa6jdwtDX8P09H4efdBUr9F6\n/qszf32anB2Gn7/bC0WvZX64kd4G7Oic4LwbuKVzNct1wJdoD9tsz8wHV5BXklTRsmWemffSvlKF\nzLwH2HKUx9wA3DDgbJKkHvmmIUkqgGUuSQWwzCWpAJa5JBXAMpekAljmklQAy1ySCmCZS1IBLHNJ\nKoBlLkkFsMwlqQCWuSQVwDKXpAJY5pJUAMtckgpgmUtSASxzSSqAZS5JBbDMJakAlrkkFcAyl6QC\nWOaSVADLXJIKYJlLUgEsc0kqwETdASSVaqHyHPPz833MN1Z5PSWyzCUNzbU37WFqdm4oy26tW82l\nF2wcyrKbyDKXNDRTs3PcN32w7hjHBctcGmkOVag3lrk04hyqUC8ql3lEnADcAJwGHAJeD8wDN3Zu\n7wUuzszquxSSfoxDFepFP5cmngOsycxnA+8F/hi4GtiemZtoH6+dN7iIkqTl9FPmB4G1ETEGrAUe\nBM7IzF2d+28Fzh5QPklSD/oZM98NrAa+BTwWeDGwadH9D9Au+SWtX38yExPjfay6d63W5FCX36v2\nyajh2rBhDePjw30+qxqV579fo5K/ydtPk7OvRF3bTj9lfhmwOzP/ICJ+BrgdOHHR/ZPAbLcFzMwc\n6GO1vWu1Jpma2jfUdfRu+KcOpqf3M0pXI4zW81/daOVv8vbT5Oz9Gfa20+2Fop9hljXA/Z2/Z2i/\nIHwtIjZ3pm0Ddh1tRknScPSzZ34V8DcR8SXae+TvAu4CdkTEKuBu4JbBRZQkLadymWfmLPAbR7lr\ny4rTSJL64qcmSlIBLHNJKoBlLkkFsMwlqQCWuSQVwDKXpAJY5pJUAMtckgpgmUtSASxzSSqAZS5J\nBbDMJakAlrkkFcAyl6QCWOaSVADLXJIKYJlLUgEsc0kqgGUuSQWwzCWpAJa5JBXAMpekAljmklQA\ny1ySCmCZS1IBLHNJKoBlLkkFsMwlqQCWuSQVYKKfmSLiXcCLgROBPwN2AzcCh4C9wMWZuTCgjJKk\nZVTeM4+ILcAzM/MsYAvwC8DVwPbM3ASMAecNLuJC5Z/5+fk+5pOk5upnz/wc4N8j4lPAKcA7gNdm\n5q7O/bd2HvOpwUSEa2/aw9Ts3KAW9yitdau59IKNQ1m2JB0r/ZR5CzgVeBHtvfLP0N4bP+wBYG23\nBaxffzITE+M9rWx+fp6p2Tnumz7YR9TebNiwhvHx3vJU1T5KGK5h5u9XqzVZd4QVGZX8Td5+mpx9\nJeradvop8/8BvpmZDwPfjog54KcX3T8JzHZbwMzMgQqrG/4QyPT0fh79ejRITc9fXas1ydTUvrpj\n9G208jd5+2ly9v4Me9vp9kLRz9Us/wy8ACAifgo4GbgtIjZ37t8G7FpiXknSEFTeM8/Mz0bEpoi4\nk/aLwUXAvcCOiFgF3A3cMtCUkqSu+ro0MTPfeZTJW1YWRZLUL980JEkFsMwlqQCWuSQVwDKXpAJY\n5pJUAMtckgpgmUtSASxzSSqAZS5JBbDMJakAlrkkFcAyl6QCWOaSVIC+PjVRao7qX5DwyHfIVjE6\nX5Cg45NlruL5HbI6HljmKt6wv0NWGgWOmUtSASxzSSqAZS5JBbDMJakAlrkkFcAyl6QCWOaSVADL\nXJIKYJlLUgEsc0kqgGUuSQWwzCWpAJa5JBWg709NjIifBO4CngccAm7s/N4LXJyZ1T9IWpLUl772\nzCPiROAvgf20P5X/GmB7Zm7q3D5vYAklScvqd5jlKuDPge93bp+embs6f98KnL3SYJKk3lUu84j4\nHWAqMz/fmTTGo78z6wFg7cqjSZJ61c+Y+auBhYg4G9gIfAxoLbp/EpjttoD1609mYmK8p5W1v49x\nuDZsWMP4eG95qmp6/n61WpN1RwCa//w3OX+Ts69EXdt+5TLPzM2H/46I24E3AldFxObMvAPYBtzW\nbRkzMwcqrHH451Gnpw8P/Q9D0/NX12pNMjW1r+4YHU1//pucv8nZ+zPsbb/bC8UgvgN0AXgbsCMi\nVgF3A7cMYLmSpB6tqMwzc+uim1tWFkWS1C/fNCRJBbDMJakAgxgzl6QCVT+B276Cp+p8gzmBa5lL\n0hKuvWkPU7NzQ1l2a91qLr1g48CWZ5lL0hKmZue4b/pg3TF64pi5JBXAMpekAljmklQAy1ySCmCZ\nS1IBLHNJKoBlLkkFsMwlqQCWuSQVwDKXpAJY5pJUAMtckgpgmUtSASxzSSqAZS5JBbDMJakAlrkk\nFcAyl6QCWOaSVADLXJIKYJlLUgEsc0kqgGUuSQWwzCWpABNVZ4iIE4GPAj8HnARcCXwTuBE4BOwF\nLs7MhcHFlCR108+e+W8BU5m5CXgBcD1wNbC9M20MOG9wESVJy+mnzG8GLl80/0PA6Zm5qzPtVuDs\nAWSTJPWo8jBLZu4HiIhJ2sX+buBDix7yALB2IOkkST2pXOYAEXEqsBO4PjP/PiL+ZNHdk8Bst/nX\nrz+ZiYnxntY1Pz/fT8RKNmxYw/h4b3mqanr+frVak3VHAJr//Dc5f5OzQ/Py93MC9HHA54GLMvP2\nzuSvRcTmzLwD2Abc1m0ZMzMHKqxx+OdRp6f30x7qH4am56+u1Zpkampf3TE6mv78Nzl/k7PDKObv\ntpPUz575dtrDKJdHxOGx80uA6yJiFXA3cEsfy5Uk9amfMfNLaJf3kbasOI0kqS++aUiSCmCZS1IB\n+rqaRceT6ieB2lcBVJ1vdE7gSk1kmWtZ1960h6nZuaEsu7VuNZdesHEoy5aOJ5a5ljU1O8d90wfr\njiGpC8fMJakAlrkkFcAyl6QCWOaSVADLXJIKYJlLUgEsc0kqgGUuSQWwzCWpAJa5JBXAMpekAljm\nklQAy1ySCmCZS1IBLHNJKoBlLkkFsMwlqQCWuSQVwDKXpAJY5pJUAMtckgpgmUtSASxzSSqAZS5J\nBZgY1IIi4gTgI8DTgB8Br8vM7w5q+ZKkpQ1yz/wlwKrMPAv4feDqAS5bktTFIMv8WcDnADLzq8Cv\nDnDZkqQuBjbMApwC3L/o9nxEnJCZh4584Bln/PJRF3DXXXuPOv3ma17F/KGFH5v+8rf/7VEf//EP\nXXjU6Ud7fGvdas4446mV8ph/6Txg/uXyHE/5W+tWV8oDvedvrVtdOU/Vx49i/qWMLSz8+EbSj4i4\nGvhKZt7cuf1fmXnqQBYuSepqkMMsu4FzASLiTOAbA1y2JKmLQQ6z/APw6xGxu3P71QNctiSpi4EN\ns0iS6uObhiSpAJa5JBXAMpekAljmklSAQV7Noj5ExKql7svMB49lFknN1fg984h4X+f3S+rO0qe9\nwA+APOLnW3WG6lVEPDUinnjEtDPryrMSEbElIp5Td45+RcQ5dWeoIiJOWfT3UyPilRHxpDozVRUR\nrc7vX4yI8yPiyXVlafyliRFxD/Bh4M20P9xrrHPXQmb+VW3BetTZGD4PPC8zp+vOU0VEXA6cA5wI\n/BtwUWYuRMTtmbm13nTLi4jzaW8zc8DfAZtpf+LnlzPzyjqz9SIifhdY4JFt/q10PuCuIdv+FzLz\nuRHxauAi4AvAs4EbM3NHvemWFxHXAd8Dvg+8BdgFnAl8MjOvOtZ5Gr9nDrwS2ACsAh5/xM/Iy8wp\n2p8yeXrdWfpwbmY+OzOfAeyn/RHITfJ24MnAC4FLgG3Ai+i8k7kBXkL7zXmPB55A+3/gCZ2fJjj8\nIvRaYGtmvhPYAryqtkTVPD0zPwi8DnhOZr6F9ovRy+oI0/gyz8yvZuZ7gccBlwNvBN4NvCMi7hn1\nQ8+IOD8z/wn4Tt1Z+hERh/8h3wGsjYjLaO8tNsEYcDAz7wHek5kPZebiPd1R90Lgdtrnvq4A7s3M\nKzLzinpj9WwyIjYA/w083Jk2T/tIrxE6+f8DWNOZtLauLI0v80U+DTwlM58A/BLtjxc4F/ijWlMt\n74qIeArw8Yg4LR5xWt3BevAJ4M6I2NApwdcAzwOeWW+snn0M2NP5dM/rASJiJ3BrvbF6k5mHMnM7\n8HXgFuAnao5U1W7a/7fPAt4aEY+hPVz3iVpT9e69wB20j4i+HhH/CPwLcMyHWKCsMj81MxOg8w1H\nP9fZ43qo3ljL+ghwHRDAXxzxM+q+B1xAe5iLzJwDXgD8Zp2hKpiifVj/s4umvatzpDfyOmP+AP8K\nvId2sTTJ7sx8DnAW7f+BA8ArMvOaemP17DG0s3+R9pDd9cCZmVnLi1HjT4AeFhE3A98Fvkx7z/Dn\ngb+m/c/53Bqj9SQifg/YCqymfZi/kJnb6k3VXUTcDZxP+3m+kE5uYOzwC+soWyI/tJ/7b9cWrEfm\nr9ei/B8FfptHtn/qyF/SdeYXAm+gvWe4F/hD4FeAV9SYqYo30c4/27ndhFfZI48q/v+fERj5F1DM\nX7dS8p/GCOQvZs+86SJiZ2a+tO4c/WjiUcVi5q9XIfmfC5xEjflL2jNvuk9HxFeAb3ZuL2Tma+oM\nVEETjyoWM3+9zD8AlvnouAT4IPDDzu0mbdB7M/OLdYdYAfPXy/wDYJmPju/XdRZ8AJp8VAHmr5v5\nB8AyHx1zEfE5YA/tvfKFzjXETdDkowowf93MPwCW+ej4TOd30zZkaPZRBZi/buYfAK9m0YpFxCdp\nv525iUcV5q+Z+QfDPXMNQpOPKsD8dTP/ALhnLkkFKOmzWSTpuGWZS1IBLHNJKoBlLkkF+D+s5n6X\nrEUB0wAAAABJRU5ErkJggg==\n",
       "text": [
        "<matplotlib.figure.Figure at 0x119576a50>"
       ]
      }
     ],
     "prompt_number": 216
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from collections import Counter\n",
      "from sklearn.cross_validation import KFold\n",
      "from utils import flatten, unique_everseen\n",
      "\n",
      "def small_doc_experiment(X, y):\n",
      "    kf = KFold(len(y), n_folds=10, shuffle=True, random_state=1)\n",
      "    rank_scores = np.zeros(10)\n",
      "    for i, (train, test) in enumerate(kf):\n",
      "        # separate dataset into training (or index) and test set\n",
      "        X_train, X_test, y_train, y_test = X[train], X[test], y[train], y[test]\n",
      "        # make a set of all unique labels\n",
      "        labels = set(flatten(list(y_train)))\n",
      "        # initialize the IR System\n",
      "        model = IRSystem(k1=1.2, b=0.85, cutoff=0)\n",
      "        # Fit the model with unigrams, remove all stopwords and hapaxes.\n",
      "        model.fit_raw(X_train, y_train, ngram_range=(1, 1), stop_words='english', min_df=2)\n",
      "        # compute the ranking\n",
      "        ranking = model.rank_labels(X_test, raw=True).tolist()\n",
      "        # remove all duplicate labels\n",
      "        ranking = map(lambda r: list(unique_everseen(r)), map(flatten, ranking))\n",
      "        ranking, y_test = zip(*[(r, y_) for r, y_ in zip(ranking, y_test) if any(l in labels for l in y_)])\n",
      "        # compute the average precision\n",
      "        rank_scores[i] = mean_average_precision(ranking, y_test)\n",
      "    return rank_scores"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = small_doc_experiment(texts, labels_fortune)\n",
      "score_df['fortune']['small_doc'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.54 (+/- 0.08)\n"
       ]
      }
     ],
     "prompt_number": 260
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Next we run the same experiment, using the big document setup:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from collections import defaultdict\n",
      "\n",
      "def big_doc_experiment(X, y):\n",
      "    kf = KFold(len(y), n_folds=10, shuffle=True, random_state=1)\n",
      "    rank_scores = np.zeros(10)\n",
      "    for i, (train, test) in enumerate(kf):\n",
      "        # separate dataset into training (or index) and test set\n",
      "        X_train, X_test, y_train, y_test = X[train], X[test], y[train], y[test]\n",
      "        big_docs = defaultdict(str)\n",
      "        for (labels, doc) in zip(y_train, X_train):\n",
      "            for label in labels:\n",
      "                big_docs[label] += \" \" + doc\n",
      "        y_train, X_train = zip(*big_docs.items())\n",
      "        # make a set of all unique labels\n",
      "        labels = set(flatten(list(y_train)))\n",
      "        # initialize the IR System\n",
      "        model = IRSystem(k1=1.2, b=0.75, cutoff=0)\n",
      "        # Fit the model with unigrams, remove all stopwords and hapaxes.\n",
      "        model.fit_raw(X_train, y_train, ngram_range=(1, 1), stop_words='english', min_df=2)\n",
      "        # compute the ranking\n",
      "        ranking = model.rank_labels(X_test, raw=True)\n",
      "        ranking, y_test = zip(*[(r, y_) for r, y_ in zip(ranking, y_test) if any(l in labels for l in y_)])\n",
      "        # compute the average precision\n",
      "        rank_scores[i] = mean_average_precision(ranking, y_test)\n",
      "    return rank_scores"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = big_doc_experiment(texts, labels_fortune)\n",
      "score_df['fortune']['big_doc'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.52 (+/- 0.03)\n"
       ]
      }
     ],
     "prompt_number": 261
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Finally we run an experiment using the Multi-Label IR system:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def ml_experiment(X, y, num_neighbors=101):\n",
      "    kf = KFold(len(y), n_folds=10, shuffle=True, random_state=1)\n",
      "    rank_scores = np.zeros(10)\n",
      "    for i, (train, test) in enumerate(kf):\n",
      "        # separate dataset into training (or index) and test set\n",
      "        X_train, X_test, y_train, y_test = X[train], X[test], y[train], y[test]\n",
      "        # make a set of all unique labels\n",
      "        labels = set(flatten(list(y_train)))\n",
      "        # initialize the ML-IR System\n",
      "        model = ML_IRSystem(num_neighbors=num_neighbors, smooth=1., k1=1.2, b=0.75, cutoff=0)\n",
      "        # Fit the model with unigrams, remove all stopwords and hapaxes.\n",
      "        model.fit_raw(X_train, y_train, ngram_range=(1, 1), stop_words='english', min_df=2)\n",
      "        # compute the ranking\n",
      "        ranking = model.rank(X_test, raw=True)\n",
      "        ranking = [sorted(ranks, key=ranks.__getitem__, reverse=True) for ranks in ranking]\n",
      "        ranking, y_test = zip(*[(r, y_) for r, y_ in zip(ranking, y_test) if any(l in labels for l in y_)])\n",
      "        # compute the average precision\n",
      "        rank_scores[i] = mean_average_precision(ranking, y_test)\n",
      "    return rank_scores"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "For the Multi Label IR system, it is important to set the right number of neighbors. Here we experiment with a number of different settings:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = np.zeros(11)\n",
      "for i, k in enumerate(range(1, 102, 10)):\n",
      "    scores[i] = ml_experiment(texts, labels_fortune, num_neighbors=k).mean()\n",
      "sb.plt.plot(range(1, 102, 10), scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 198,
       "text": [
        "[<matplotlib.lines.Line2D at 0x1177ffd10>]"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAECCAYAAAD5OrxGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl03OV97/H3jHZrsyxLshbLC8Zf2xjj3WAbDAYMgZDS\nJKdNbkNuCGRvm0PvuSmhy2lv0ybntrSXLLQpoYGkqW8PTZyE68QQCKtCbPCCbGw/8obkVZJl7cto\nmbl/zMgehK2x5JFm+7zO0bF+v9/8NM9jSZ959J3n9/w8gUAAERFJHd5YN0BERCaXgl9EJMUo+EVE\nUoyCX0QkxSj4RURSjIJfRCTFpI920My8wOPAEsAHPOicOxJ2fBXwKOABTgKfBAZHO0dERGIr0oj/\nXiDTObcWeJhgyANgZh7gX4FPOeduBF4E5oTOybrYOSIiEnuRgn8dsA3AObcdWBl2bD7QAvyJmb0M\nTHXOudA5v7zEOSIiEmORgr8A6AjbHgqVfwCmA2uBbwG3Abea2S0RzhERkRgbtcZPMMDzw7a9zjl/\n6PMW4HBolI+ZbSM4uh/tHBERibFII/Ea4C4AM7seqA07dhTIM7OrQts3AvsinHNRgeCCQfrQhz70\noY+xfYyLZ7RF2kJv4A7P0AG4H1gB5DnnngiVdr5BcFZPjXPuoYud45yri9COQHNz53j7EPdKSvJR\n/xJTMvcN1L9EV1KS7xnPeaMG/yRS8CewZO5fMvcN1L9EN97g15uuIiIpRsEvIpJiFPwiIilGwS8i\nkmIU/CIiKUbBn6T2v3uOP/ynV6k73hbrpohInFHwJ6mX95yixzfIszXHYt0UEYkzCv4k1D8wxN4j\nLQC8824rDY3JO49ZRMZOwZ+E9h07h29giDnlwSWTnttxPMYtEpF4ouBPQjtdEwCf2GRUTM9lx4FG\nznX0xbhVIhIvFPxJZmDQz57DZykuyGb2jHzuWDWTIX+AF946EeumiUicUPAnmQP15+j1DbHCSvB4\nPFx/zQwKcjN5ec9JevoGY908EYkDCv4k85ZrBmCllQKQke7lthVV9PUP8erbp2LZNBGJEwr+JDLk\n97Pn0Fmm5mUyt7Lg/P6bl1WSmeHlV28dZ3BI98QRSXUK/iTiGtro6h1g+fwSvJ4Lq7Xm5WRw45IK\nWjt9vHmwKYYtFJF4oOBPIjtDZZ4VoTJPuE2rZuLxwHPbG4iTezCISIwo+JOE3x9gZ10zeTkZzJ9Z\n+L7jJVNzWGmlNDR1caC+NQYtFJF4oeBPEodPttPR3c/y+dNJ817823rnmmoAtu1omMymiUicUfAn\nibdCF22tvEiZZ9ic8gLmz5zKvqPnONHcNVlNE5E4o+BPAoFAgF11zUzJSmfBrKJRH3vn6uCo/zmN\n+kVSloI/CRw73cm5Dh9Lr55Oetro39Il84qZMW0Kv32nkdZO3yS1UETiiYI/CQyXeVZYScTHej0e\n7lgdXMbhxZ1axkEkFSn4E1wgEGCnayIrM43Fc6Zd1jlrF8+gYEoGL+8+Sa9PyziIpBoFf4I73tRF\nc1sf111VTEZ62mWdk5GexsYVVfT4Bnm99vQEt1BE4k36aAfNzAs8DiwBfMCDzrkjYccfAh4AmkO7\nPgvUA98D5gEDwB87596OftMF3r82z+W6ZVklv3ijnuffPM7GFZWXnAIqIskn0m/7vUCmc24t8DDw\n6Ijjy4H7nHO3hD4OAZ8BekLnfAb4t2g3Wi7Y6ZrITPdy7dziMZ2XPyWTdUvKaenoO3/Fr4ikhkjB\nvw7YBuCc2w6sHHF8BfCImb1mZg+H9i0KO6cOqDSzAiTqTp7t5nRLD4vnFpOVeXllnnCbVs3EA/xS\nyziIpJRIwV8AdIRtD4XKP8M2A58DNgLrzexuYA/wQQAzux4oAXKj1mI5b+cYZvNcTFnRFJbPL6H+\nTCd1x9ui2TQRiWOj1vgJhn5+2LbXORe+ru9jzrkOADPbCiwDvg4sNLPXgBqgDjgXqSElJfmRHpLQ\nJqJ/bx9pIT3Nw61rZpObkzGur/GxOxaws66ZX+85xfoV1eNuSzJ//5K5b6D+paJIwV8D3AM8Exq9\n1w4fMLNCoNbMFgE9BEf9TwKrgV875/7EzFYCq51zEa8Uam7uHGcX4l9JSX7U+9fU2sOxUx0suaqY\nnq4+errGd0/d4twM5lUW8ub+Rt4+cIaK6WP/42wi+hcvkrlvoP4luvG+qEUq9WwB+syshuAbuw+Z\n2cfN7DPOuXaCb/i+BLwK7HPObQMc8GUz+w3wvwm+wStRdmEJ5vGVecLdoWUcRFLKqCN+51wA+MKI\n3XVhxzcTrPOHn3MOuD1aDZSLe8s14/V4WHb1lQf/squnU1qUwxvvnOHDN82lMC8rCi0UkXilydsJ\nqKW9j2OnO1gwayp546zth/N6PdyxaiaDQwFe3KVlHESSnYI/Ae2su/SdtsZr7bXl5OVk8NKuk/j6\nh6L2dUUk/ij4E9BO14QHWH719Kh9zayMNDYur6S7b5DX92oZB5FkpuBPMO1dPg6faOfqqsKo1+I3\nLq8iI93Lczsa8Pt1QZdIslLwJ5hddc0EgBULolfmGVaQm8m6xTM4297Hrjot4yCSrBT8CWZ4UbYV\n8698Ns/F3K5lHESSnoI/gXT29OMa2phbUcC0guwJeY7y4lyWXj2dY6c7OHSifUKeQ0RiS8GfQHYf\nOos/EIjKRVuj0QVdIslNwZ9ALlytG/36frirqwqZW1HAnkNnOd3SPaHPJSKTT8GfIHr6Btj/7jmq\nS/MonZozoc/l8Xi4c3U1AeBXbx6f0OcSkcmn4E8Qbx9uYcgfmJDZPBezfH4J0wuzqdl3ho7u/kl5\nThGZHAr+BPFWaO39lRNc3x/m9Xq4Y3U1A4N+fq1lHESSioI/AfT1D7Lv2DkqpudSXjx597RZf205\nudnp/HrXSXwDWsZBJFko+BNA7ZEWBgb9EzZ3/1KyMtO4ZXklXb0D/GbfmUl97lTX0dNPW1fE21iI\njEukG7FIHIjm2vtjdevyKrZtb+D5HQ1suK4Cr9cz6W1IFf5AgAPvtvLKnpPsPnSW9DQvD35wUUy+\n75LcFPxxrn9giNojLZROzWFmad6kP39hXhY3XDOD12pPs+fwWZZP8l8dqaC9u5/Xa0/x6tunaG4L\n3kmtsiSX5rZevrNlL79701w+eMMsPB696Ep0KPjj3DvHzuEbGGLFgpKY/eJvWl3Na7Wn2bajQcEf\nJf5AgAP1rbyyOzi6H/IHyEz3sv7acjYsq2BueQHHm7r41o9r2fLqUU42d/HpuxaSmZEW66ZLElDw\nx7nhtXlWTvBFW6OpnJ7LkquKqT3SwuGT7cyrLIxZWxJde3c/NXtP8+qeUzS19QJQVZLLhqWV3HBN\nGVOyL9xYp7osn7/476v49pa97DjQRFNrL3/0kSUU5esOaXJlFPxxbHDIz57DZykuyGL2jPHdVDla\n7lxdTe2RFp7b3sC8D18b07YkmlFH90srmFtRcMm/5gpyM/mfH1vGD59zvL73NP/r6Tf5ow8vYW5F\nwST3QpKJgj+O7X+3lV7fIOuvLY95fdeqpzJ7Rj676pppbO2hrGhKTNuTCMYyuh9NRrqX++9aQFVJ\nLv/50mG+8aNdfPquBVx/zYyJbL4kMQV/HNsZumgrHmZ1eDwe7lxTzb/87B2ef/M4922yWDcpLp0f\n3e85xe665vOj+3XXzuDmpZWjju5H4/F42LS6mhnFuXz35/v412f3c/JsN79701y8etNXxkjBH6eG\n/H52HzpLYW4m86rio6a+wkooLsimpvY0966fQ/6UzFg3KW5cbHRfWZLLzWMc3Uey5Kpi/uy+lXzz\nx7VsfaOeU2e7efCDi8jJ0q+yXD79tMSpuoY2unoHuGV5ZdyM6NK8XjatmsnmFw/x0u6TfGjdnFg3\nKaZGG91vWFrJVeMc3UdSMT2XP//kSv75p/vYfegsf/fvO/njjyyhZIIX75PkoeCPU2+Fbn24Ms6m\nT65fUs7PXj/GiztP8IE11WSkT/70Qt/AEG8eaOLoqXYyM9LIyUoPfmQGP8/OCu3LDO3PSiMrIy1q\nIdzR3c/rkzC6H01eTgYP/d51/OeLh3lx1wn+5um3+NLvLsaqiyb8uSXxjRr8ZuYFHgeWAD7gQefc\nkbDjDwEPAMM3aP0scAT4HjAf8AOfcc656Dc9efkDAXa5ZvJyMphfPTXWzXmPnKx0bl5WyS9+W89v\n9p1hw9LKSXvu401dvLLnJG+800ivb3BM53o8kJ0ZfBEYflHIzkp7z4tDcN+Fz3Oy0invGcDX2092\nZjqnW7p5OWx0n5HuZd3iGWxYNnGj+9Gkp3n5g03zqSzJ5Ue/quMf/u8e7rvDuOm6iklthySeSCP+\ne4FM59xaM1sDPBraN2w5cJ9zbvfwDjO7E8h1zq03s9uAvwU+GuV2J7XDJ9pp7+7nxiXlpHnjbzml\nW1dU8dyOBp7bcZwbJzhkfP1D7DjQyCtvn+LoqQ4ApuZlctuK2SyfX4I/EKDPN0iPb4i+/kF6fYP0\n9g/R6xukL+zz4EfwMW2dPk77evCP857CldNz2bC0ghsWzyB3Ekb3kdy8rJIZ06bwnS17eeqXBznR\n3MXvb5wXlz87Eh8iBf86YBuAc267ma0ccXwF8IiZzQC2Oue+AfQChWbmAQoBLeY+RpN1p63xKsrP\n4vpryqjZe4bawy3cXhr9OeUNjZ288vYpfvvOGXp9Q3gIvrG5YWkFS64qvuJQCwQC9A/6L/riEHzx\nCL5weNLSaGnrodc3SE5WOusWl3NV5eSP7iNZMKuIv/jUKr71X7W88NYJTp/t5vP3Lo6LFyaJP5GC\nvwDoCNseMjOvc84f2t4MfAfoBLaY2d0EXyiygYNAMXBPdJuc3AKBALvqmsjJSmfR7Pit196xupqa\nvWfYtqOB29dG503evv5Bdhxo4pU9pzh2OvhjV5Sfxe0rZ3LjkgqKC6N3g3mPx0NWRrD2P9qcqZKS\nfJqbO6P2vBOpdGoOj9y3gn/9+Tu8faSFr/1gJ3/8kWsndSlvSQyRgr8DCL9kNDz0AR5zznUAmNlW\nYBlwHVDjnPszM6sCfm1mi51zo478S0pie2XqRLvc/tU1tNLS4eOWFVWUz4iPaZwXU1KSz/IFpew6\n2ERdQyvzr+BNxSMn2njut/W8vOsEvb5BvB5YtaiMO6+fzYoFpaSlxbZkkWg/m3/9+XX88Bf7+fFL\nh/m7H+7kK59cxfJR/npMtP6NVbL3bzwiBX8NwRH7M2Z2PVA7fMDMCoFaM1sE9AAbgSeBG7nwV0Ir\nkAFEnPqRKKOq8RjLqPGF7e8CsHhWUdz/n2xcWsGug01sefkwn/7AgjGde2F0f5Jjp4P9LMrPYtOq\nmdy4pJxpBcHR/blzsb3ZeyKN+MPdvaaaotwMnvrlQf7qiTf42MaruW1l1ftKVInav8uVCv0bj0jB\nvwW43cxqQtv3m9nHgTzn3BNm9jDwEsEZPy8457aZ2W+B75vZawRD/6vOud5xtS7FBAIBdh5sJisj\njWvmTIt1cyJaOKuI6tI8flN7intumHVZ88jrz3QGZ+bsb8TXP4THA0vnTWfD0gqunVus9f6jaO3i\ncsqKpvCtn+xl84uHOHm2i09sMtJj/BeUxJ4nMM6ZDVEWSPZX5cvpX0NjJ3/1/TdZtaCUL9y7eBJa\nduXeeOcMTzy7n9tWVPHfbp9/0cf0+gbZfqCRV/acov5M8P9hWkEWNy2pYH3Y6D4eJcOI8VxHH9/6\n8V7qGzuZX1XIFz98LQWhq66ToX+jSYH+jWukpAu44kgs77Q1XqsWlLLl1aO8VnuaD62fQ17OhVkk\n757p4JU9p/htaHTv9XhYdnVwdL94jkb3k2VaQTYPf2I5T249wFsHm/ibp97iyx9dQlUMbuwj8UHB\nH0d21jWTke5lyVXFsW7KZUtP8/Khm67i3559h1f2nGTj8iq27w+N7huDI63igizuWlPN+iUVWks+\nRrIy0vjC71zDs9Nz+enrx/jbf9/JZz+4iE0xeuNzYNBPWponbpYjSTUK/jhxuqWbU2e7WXb1dLIz\nE+vbcsf1s9j8/EG2vlHP//tNPb6B8NF9JYvnTNPoPg54PB4+tH4OFdNz+d7W/Xz7J3tp7xtkw7Uz\nonZdgt8foKOnn9ZOH22dPtq6fLR2+Wjr7A/+2xXc3903SFF+FqsXlrJmURmzyvLj7tqIZJZYCZPE\n4uFOW+M1JTuDjcur2PpGPcUF2dx1wyzWX1uu0X2cWrmglNKiHL7541p+8IsD1L17jk99YMGot3UM\nBAL0+gZp7eqnrdMXDPbzoR4K9K5+2rv6R70iOicrnaL8LGaW5tHQ2MVzO47z3I7jlBXlsGZRGWsW\nlem6g0mg4I8TOw82keb1cN28xCnzhLv3xjmsWlBKVUmeRvcJYPi2jt/9+Tv8dn8jja09fGTDVXT1\nDoSCvP98qA+P1PsH/Jf8eulpHqbmZTG3soCpeVkU5WVRlJ/F1LzM0L/Bj6zMCy8uA4N+9h1tYfuB\nRvYcOsvPa97l5zXvUl2ax5pryli9oCyqF+3JBZrVMwkizSxoauvl4X95g2vnFvPQ7103iS2LjmSe\nOZHMfQOYWjSFR3/4FjX7zlz0uAfIz82kKG9EiIf+HQ73vJyMKyrV9PUPsufQWbbvb2TfsXMM+YO5\ndHVVIWsWlbFyQen5mUhjkezfP83qSWDxdKctSS0Z6Wl8+u6FLJo9jRNnu8JG6sF/C3IzJ2Xef3Zm\nOtdfM4Prr5lBV+8Ab7kmduxvxDW0cehEO//xq0Msml3EmkVlLJ9fohvPXCH978WBna75/JuhIpPN\n4/Fww+L4uX9vXk4GNy+t5OallbR2+njzQCPbDwT/Eth37BxPb3NcN6+YNQvLWHJV8ajvTcjFKfhj\n7FxHH0dPdbBwVpFuZSgyQlF+FptWV7NpdTWNrT1s39/I9v2N7HTN7HTNZGemsXx+CWsWlbFodpGW\nor5MCv4Y23l+No/KPCKjKSuawofWzeGetbM53tTF9gON7NjfxG/2neE3+86QPyWDlQtKWbOwjHlV\nhbpGYBQK/hjb6ZrwAMvj7BaLIvHK4/FQXZZPdVk+H91wFUdOdrB9fyNvHmzkpV0neWnXSaYVZLF6\nYRl3rp1DfqZX1wiMoOCPofYuH4dOtDOvqpDCPM15Fxkrj8fDvKpC5lUV8rHb5nGgvpXt+xvZVdfM\ntu0NbNvewIxpU1izqIxbV1S9Z0mRVKbgj6Fdh84SIH7vtCWSSNK8XhbPKWbxnGI+eccQtUfOsedo\nCzveOcPPXj9Gr2+Qj916daybGRcU/DF0fhqnyjwiUZWRnsYKK+HO9XNpONHKwfpW5lRE/xahiUrB\nHyNdvQMcrG9jTnmBrk4UmUA5Weks0+DqPTT3KUZ21zXjDwQ0m0dEJp2CP0Z21iXe2vsikhwU/DHQ\n0zfIO8fOMbM0j9KiKbFujoikGAV/DLx95CxD/oBG+yISEwr+GLhwi0VN4xSRyafgn2R9/YPsPdpC\nefEUKqfrhhMiMvkU/JNs79FzDAz6NdoXkZhR8E+y4Yu2NI1TRGJFwT+JBgaHePtICyVTs5lZmhfr\n5ohIihr1yl0z8wKPA0sAH/Cgc+5I2PGHgAeA5tCuzwE3AJ8KbecA1wFlzrmOqLY8Ae07dg5f/xAr\nllVqtUARiZlISzbcC2Q659aa2Rrg0dC+YcuB+5xzu8P21QFPA5jZt4HvKfSDLszmUZlHRGInUqln\nHbANwDm3HVg54vgK4BEze83MHg4/YGYrgWucc9+LVmMTVSAQoKt3gD2HzjKtIIu55VosSkRiJ9KI\nvwAIH60PmZnXOecPbW8GvgN0AlvM7G7n3NbQsUeAv4pmY+OF3x8M8s6efjp7BugM/zz0b/jxrt4B\nhvwBANZeO0NlHhGJqUjB3wHkh22Hhz7AY8NlHDPbCiwDtprZVGC+c+6Vy21ISUl+5AdNkIFBPx3d\nPtq7+mnv8tHe3U9Ht4+Orn7au4P7OkL/tnf109XbTyAQ+evmZqdTkJdF+fRcCvOymFaQze/dNp/p\nU3MmvlOTLJbfv4mWzH0D9S8VRQr+GuAe4Bkzux6oHT5gZoVArZktAnqAjcCTocM3AS+OpSHNzZ1j\nefgV6+4b4B//cw9nzvXQ6xuK+HgPkJuTQf6UDMqn5ZA/JZP8KRnkhf4NfmSSn5Nx/lh6WrCSVlKS\nf75/gYHBSe/rRAvvX7JJ5r6B+pfoxvuiFin4twC3m1lNaPt+M/s4kOeceyJU13+J4IyfF5xz20KP\nmw8cef+Xix+1h1s4drqT4oJsZs8oCAZ3zogQDwv2vOwMvF6VaEQk8Y0a/M65APCFEbvrwo5vJljn\nH3neP0SldRPoYEMrAF/68GJmz9CbrSKSOlL2Aq6DDa1MyUqnulT1PxFJLSkZ/C3tfTS39TF/5lSV\nb0Qk5aRk8A+XeRbMKopxS0REJl9qBn99KPirp8a4JSIiky/lgj8QCHCwoZXc7HSqtFCaiKSglAv+\ns+19tHT4sOoivLqCVkRSUMoFv8o8IpLqUi/49cauiKS4lAr+YH2/jfwpGbrfrYikrJQK/qbWXlo7\ng/V9rZApIqkqpYL/QKjMs1D1fRFJYSkV/K6hDQCrVn1fRFJXygR/IBDgYH0rhbmZlBdPiXVzRERi\nJmWC/8y5Htq7+7Hqqarvi0hKS5ngPz9/X9M4RSTFpUzwHwjV9xeqvi8iKS4lgj8QCOAaWinKz6K0\nKPnudysiMhYpEfwnz3bT2TOg+r6ICCkS/MPTOBeozCMikhrBrzd2RUQuSPrg94fW3y8uyKKkMDvW\nzRERibmkD/4TTV109w2yQOvziIgAKRD8B4fr+yrziIgAqRD8ofq+aWE2EREA0kc7aGZe4HFgCeAD\nHnTOHQk7/hDwANAc2vVZ59whM/sqcA+QAXzbOff0RDQ+Er8/QN3xNqYXZjO9UPP3RUQgQvAD9wKZ\nzrm1ZrYGeDS0b9hy4D7n3O7hHWZ2M3BD6Jxc4CtRbvNlO97URY9vkOVWEqsmiIjEnUilnnXANgDn\n3HZg5YjjK4BHzOw1M3s4tO8OYK+Z/RR4Fvh5FNs7Jgfqh9ffV31fRGRYpOAvADrCtodC5Z9hm4HP\nARuB9WZ2N1BM8AXho8DngR9Fr7ljo/vrioi8X6RSTweQH7btdc75w7Yfc851AJjZVmAZ0AIcdM4N\nAnVm1mdm051zZ0d7opKS/NEOj9nQkJ9DJ9qpmJ7L/LnTo/q1xyPa/Ys3ydy/ZO4bqH+pKFLw1xB8\nk/YZM7seqB0+YGaFQK2ZLQJ6CI76nwTSgC8D/2hmFUAuwReDUTU3d46rA5dy9FQHvb5BVi8sjfrX\nHquSkvyYt2EiJXP/krlvoP4luvG+qEUK/i3A7WZWE9q+38w+DuQ5554I1fVfIjjj5wXn3DYAM7vJ\nzHYQLCV90TkXGFfrrsBwmUfTOEVE3mvU4A8F9hdG7K4LO76ZYJ1/5Hl/GpXWXYHz9X29sSsi8h5J\neQHX4JCfQ8fbKS+ewtS8rFg3R0QkriRl8L97phPfwJBG+yIiF5GUwa9lmEVELi05g3/4jd2ZemNX\nRGSkpAv+gUE/h0+0Uzk9l4LczFg3R0Qk7iRd8B873UH/oF/1fRGRS0i64L+wTIPKPCIiF5N8wV/f\nigcwjfhFRC4qqYJ/YHCIwyc7qCrNIy8nI9bNERGJS0kV/EdOdjA4pPq+iMhokir4LyzToPq+iMil\nJFfwh+r78xX8IiKXlDTB7xsY4ujpDqrL8snNVn1fRORSkib4j5xsZ3AooGmcIiIRJE3waxlmEZHL\nkzzBX9+GxwPztT6PiMiokiL4+/oHOXa6g9kz8snJinRTMRGR1JYUwX/4RDtD/oDKPCIilyEpgv9g\nQxug9fdFRC5HkgR/K2leD1dXFca6KSIicS/hg7/XN8i7pzuZXZ5Pdqbq+yIikSR88B860YY/oPq+\niMjlSvjgP1gfqu8r+EVELkvCB/+BUH1/nur7IiKXZdSiuJl5gceBJYAPeNA5dyTs+EPAA0BzaNdn\nnXOHzGwX0B7ad9Q590DUWw709A3Q0NjJvMpCsjLSJuIpRESSTqR3Q+8FMp1za81sDfBoaN+w5cB9\nzrndwzvMLBvAOXdLtBs7Ut3xdgIBlXlERMYiUqlnHbANwDm3HVg54vgK4BEze83MHg7tuw6YYmbP\nmdmLoReMCXHh/roKfhGRyxUp+AuAjrDtoVD5Z9hm4HPARmC9md0NdAN/75y7A/g88KMR50TNwfpW\n0tO8zKssmIgvLyKSlCKVejqA/LBtr3POH7b9mHOuA8DMtgLLgF8BhwFC9f4WoBw4OdoTlZTkj3b4\nfTp7+jne3MU1c4upKI//hdnG2r9Ek8z9S+a+gfqXiiIFfw1wD/CMmV0P1A4fMLNCoNbMFgE9BEf9\nTwL3E3wz+EtmVkHwr4bTkRrS3Nw5pobvdM0EAnBVecGYz51sJSX5cd/GK5HM/UvmvoH6l+jG+6IW\nKfi3ALebWU1o+34z+ziQ55x7IlTXf4ngjJ8XnHPbzCwd+L6ZvTp8zoi/EqJC99cVERmfUYPfORcA\nvjBid13Y8c0E6/zh5wwC90WrgZdsW0MrGele5lZo/r6IyFgk5AVcHT39nGjuZl5lIRnpCdkFEZGY\nScjUrNMyzCIi45aQwX8gVN9fqAu3RETGLCGD/2B9K5kZXmaXa5qWiMhYJVzwt3f5ON3Sw9VVU0lP\nS7jmi4jEXMIlpzs+vAyzpnGKiIxHwgX/wXqtzyMiciUSLvgPNLSRnZnG7Bmq74uIjEdCBX9rp4/G\ncz3MnzmVNG9CNV1EJG4kVHoOL9Ngqu+LiIxbYgX/cH1f8/dFRMYtsYK/oZWcrHRmlam+LyIyXgkT\n/C3tfTS39WEzp+L1emLdHBGRhJUwwa9lmEVEoiPxgl/z90VErkjiBH99G7nZ6VSV5sW6KSIiCS0h\ngr+5rZeWjj7mz5yK16P6vojIlUiI4NcyDSIi0ZMYwR+68YrW3xcRuXJxH/yBQICDDa3k5WRQUZIb\n6+aIiCS8uA/+prZeWjt9LKhWfV9EJBriPviH6/umMo+ISFTEf/DrxuoiIlEV18EfCAQ4WN9KQW4m\nFcVTYt1uJI9lAAAHoklEQVQcEZGkkD7aQTPzAo8DSwAf8KBz7kjY8YeAB4Dm0K7POefqQsdKgZ3A\nrcP7xurMuR7au/tZvbAUj+r7IiJRMWrwA/cCmc65tWa2Bng0tG/YcuA+59zu8JPMLAP4LtB9JY07\nX+ZRfV9EJGoilXrWAdsAnHPbgZUjjq8AHjGz18zs4bD9fw/8M3D6ShqnC7dERKIvUvAXAB1h20Oh\n8s+wzcDngI3AejO728w+BTQ7554PPWZcNZpAIIBraKUwL5OyopzxfAkREbmISKWeDiD8ride55w/\nbPsx51wHgJltBZYBtwMBM7sNWAo8bWa/45xrHO2JSkree3OV+jMddPQMsGFZFaWlBZfXmzg2sn/J\nJpn7l8x9A/UvFUUK/hrgHuAZM7seqB0+YGaFQK2ZLQJ6CI76n3TOfS3sMS8RfMN31NAHaG7ufM/2\nG3tOAjBnRt77jiWakpL8hO/DaJK5f8ncN1D/Et14X9QiBf8W4HYzqwlt329mHwfynHNPhOr6LxGc\n8fOCc27buFpxERfur6sbr4iIRNOowe+cCwBfGLG7Luz4ZoJ1/kudf8t4GuUPBHDH25hWkEXJVNX3\nRUSiKS4v4DrZ3E1X7wALqos0f19EJMriMvgvrM+jMo+ISLTFZ/CH7q+r9fdFRKIv7oLf7w/gGtqY\nXpjNdNX3RUSiLu6C/3hTFz2+QS3TICIyQeIu+IfLPAtmqb4vIjIR4i/4z8/f14hfRGQixFXwD/n9\n1J1oo3RqDtMKsmPdHBGRpBRXwd/Q2EWvb0hlHhGRCRRXwa8yj4jIxIur4D/QoPX3RUQmWtwE/+CQ\nn0Mn2pkxbQpT87Ji3RwRkaQVN8EPMCUrnRuuKYt1M0REklqkZZknTXqal3/44lotyiYiMsHiasSv\n0BcRmXhxFfwiIjLxFPwiIilGwS8ikmIU/CIiKUbBLyKSYhT8IiIpRsEvIpJiFPwiIilm1Ct3zcwL\nPA4sAXzAg865I2HHHwIeAJpDuz4LHAWeAOYDAeDzzrl3ot90EREZj0gj/nuBTOfcWuBh4NERx5cD\n9znnbgl9HAI+CPidc+uBPwf+NtqNFhGR8YsU/OuAbQDOue3AyhHHVwCPmNlrZvZw6HE/Az4XOj4b\naI1aa0VE5IpFCv4CoCNseyhU/hm2mWDIbwTWm9ndAM65ITN7Cvgm8B/Ra66IiFypSMHfAeSHP945\n5w/bfsw5d845NwBsBZYNH3DOfYpgnf8JM8uJUntFROQKRVqWuQa4B3jGzK4HaocPmFkhUGtmi4Ae\ngqP+J83sPqDKOfd1oBfwhz5G4ykpyY/wkMSm/iWuZO4bqH+pyBMIBC550Mw8XJjVA3A/wbp+nnPu\nCTP7OPAQwRk/Lzjn/jo0un8KmAFkAF93zj07cV0QEZGxGDX4RUQk+egCLhGRFKPgFxFJMQp+EZEU\no+AXEUkxkaZzTqhIawElIjPLAP4NmAVkAV8DDhCc6eQH9gFfcs4l7LvqZlYK7ARuJdinp0ievn2V\n4BTmDODbBKc0P0US9C/0+/Y9gtfX+IHPAEMkeP/MbA3wDefcLWY2j4v0x8w+Q3AtsUHga865rTFr\n8BiN6N9SghfGDhHMzE8655rG2r9Yj/gjrQWUiP4AaHbO3QTcCXyHYL8eCe3zAL8Tw/ZdkdAL23eB\nboJ9+UeSp283AzeEfh5vBuaSRN87YBOQG1pH638Bf0eC98/MvkJwUcis0K73/Tya2Qzgj4C1wB3A\n180sMxbtHauL9O//AH/onLsF+Anwp2ZWxhj7F+vgj7QWUCJ6BvjL0OdeYABY7px7NbTvl8BtsWhY\nlPw98M/A6dB2MvVtE7DXzH4KPAv8HFiRRP3rBQpD1+cUAv0kfv8OAx8mGPJw8Z/HVUCNc27AOdcR\nOmfJ+75SfBrZv48554YvpM0g+D1dzRj7F+vgj7QWUMJxznU757rMLJ/gi8Cf897/5y6Cv3QJx8w+\nRfCvmedDuzxc+IGEBO5bSAnBCxQ/Cnye4DpTydS/GiAbOEjwr7ZvkuD9c879hGB5Y1h4fzoJ9qcA\naL/I/rg3sn/OuTMAZrYW+BLwT4yjf7EO2UhrASUkM5sJ/Br4gXNuM+9dsiIfaItJw67c/cDtZvYS\nsBR4mmBYDkvkvgGcBZ53zg065+qAPt77C5To/fsKwZGhEfz+/YDgqHFYovcP3vu7VkCwPyNzJp8E\nXjXYzH6f4F/ddznnWhhH/2Id/DXAXQAj1wJKVKF62/PAV5xzT4V27zazDaHPPwC8erFz451zboNz\n7uZQfXEP8ElgWzL0LeR1gu/LYGYVwBTgxSTqXy4X/sJuJTi5Iyl+NsNcrD87gBvNLCu0xthCgm/8\nJhwz+wTBkf7Nzrl3Q7vH3L+YzuoBthAcQdaEtu+PZWOi5BGCo8S/NLPhWv+XgW+G3nDZD/xXrBoX\nZQHgfxBcgTXh++ac22pmN5nZDoKDoi8C75Ik/SP4/sz3zew1giP9rxKcnZUM/RueifS+n8fQrJ5v\nAq8R/L4+4pzrj1E7xysQKoM/BtQDPzEzgJdDa6SNqX9aq0dEJMXEutQjIiKTTMEvIpJiFPwiIilG\nwS8ikmIU/CIiKUbBLyKSYhT8IiIpRsEvIpJi/j9dZWZrxKF2dAAAAABJRU5ErkJggg==\n",
       "text": [
        "<matplotlib.figure.Figure at 0x1172fea90>"
       ]
      }
     ],
     "prompt_number": 198
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = ml_experiment(texts, labels_fortune)\n",
      "score_df['fortune']['ml'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.57 (+/- 0.03)\n"
       ]
      }
     ],
     "prompt_number": 262
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The fortune label set can be divided into a good fortune versus misfortune label set instead of the more finegrained original label set. We evaluate the performance of the models on the simplified label set:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "labels_fortune = [(doc, list(set(l[:2] for l in labels))) for \n",
      "     doc, labels in list(read_labels(\"data/labels.hcnorms_misgoodfortune.all.txt\"))]\n",
      "labels_fortune, texts = zip(*match_labels_documents(documents, labels_fortune))\n",
      "labels_fortune, texts = np.array(labels_fortune), np.array(texts)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 241
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The label distribution is rather skewed:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "Counter(flatten_labels(labels_fortune))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 243,
       "text": [
        "Counter({'mf': 344, 'gf': 61})"
       ]
      }
     ],
     "prompt_number": 243
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = small_doc_experiment(texts, labels_fortune)\n",
      "score_df['fortune']['small_doc'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.90 (+/- 0.03)\n"
       ]
      }
     ],
     "prompt_number": 244
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = big_doc_experiment(texts, labels_fortune)\n",
      "score_df['fortune']['big_doc'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.96 (+/- 0.02)\n"
       ]
      }
     ],
     "prompt_number": 245
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = ml_experiment(texts, labels_fortune)\n",
      "score_df['fortune']['ml'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.96 (+/- 0.02)\n"
       ]
      }
     ],
     "prompt_number": 246
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Actions"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The next experiment focusses on the action annotations. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "labels_actions = list(read_labels(\"data/dream_acts.txt\"))\n",
      "labels_actions, texts = zip(*match_labels_documents(documents, labels_actions))\n",
      "labels_actions, texts = np.array(labels_actions), np.array(texts)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 199
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The label set follow the following distribution:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "label_counts = pd.Series(Counter(flatten_labels(list(labels_actions))))\n",
      "label_counts.plot(kind='bar', figsize=(14, 6))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 219,
       "text": [
        "<matplotlib.axes._subplots.AxesSubplot at 0x123086110>"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAzIAAAGDCAYAAADwPCXUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X+wpXddH/D3ZjfJmribH3pLwNIyVfkOVkMkUEtAEjSC\nURhsa4Nt/QEdE6lphoZQbbeMP8ZAHCFkCCrVrBqcsdCQwSqNAZwISYi1KZi0xOgXYot2ihlXdjeb\nSbIkuXv7xzmX3Gzuj3M35+xzPruv18zOnnvuc599z9nn+Z7v+z4/zpalpaUAAABUcsLQAQAAADZL\nkQEAAMpRZAAAgHIUGQAAoBxFBgAAKEeRAQAAytm20QKttR9J8obxl1+V5IVJXp7kPUkOJbk3yWW9\n96XW2iVJLk3yRJKreu83zyI0AABwfNuymc+Raa39YpJ7krw2yTW999tba+9L8rEkf5Tk40nOzajw\nfCrJi3vvj009NQAAcFyb+NSy1tqLk3xT7313knN777ePv3VLkguTvCTJnb33x3vvB5Lcn+TsaQcG\nAADYzDUyu5L87PjxlhXPP5TktCQ7kzy4yvMAAABTteE1MknSWjs9yfN777eNnzq04ts7k+xPciDJ\njhXP70iyb611PvHE4tK2bVs3lxYAADjebFntyYmKTJJXJLl1xdd3t9bOHxebi8bfuyvJ21trJyfZ\nnuQFGd0IYFX79j0y4T+9OQsLO7Jnz0MzWfe0Vcqa1MpbKWtSK6+ss1Mpb6WsSa28ss5OpbyVsia1\n8so6O7PKu7CwY9XnJy0yz0/y5yu+vjLJ9a21k5Lcl+Sm8V3LrktyR0anrO1yoT8AADALExWZ3vu7\nDvv680kuWGW53Ul2TyUZAADAGnwgJgAAUI4iAwAAlKPIAAAA5SgyAABAOYoMAABQjiIDAACUo8gA\nAADlKDIAAEA5igwAAFCOIgMAAJSjyAAAAOUoMgAAQDmKDAAAUI4iAwAAlKPIAAAA5SgyAABAOYoM\nAABQjiIDAACUo8gAAADlbBs6ADC0pU0tvbi4uMmf2bKp9QMATEKRAXLtjfdkz/6DU13nwunbc8XF\n50x1nQAAyxQZIHv2H8wDex8dOgYAwMRcIwMAAJSjyAAAAOUoMgAAQDmKDAAAUI4iAwAAlKPIAAAA\n5SgyAABAOYoMAABQjiIDAACUo8gAAADlKDIAAEA5igwAAFCOIgMAAJSjyAAAAOUoMgAAQDmKDAAA\nUI4iAwAAlKPIAAAA5WzbaIHW2r9P8tokJyb5xSR3JrkhyaEk9ya5rPe+1Fq7JMmlSZ5IclXv/eZZ\nhQYAAI5v6x6Raa1dkOSlvffzklyQ5O8luSbJrt77K5JsSfK61tpZSS5Pcl6SVye5urV20gxzAwAA\nx7GNTi17VZLPttb+S5KPJPndJOf23m8ff/+WJBcmeUmSO3vvj/feDyS5P8nZM8oMAAAc5zY6tWwh\nyXOTvCajozEfyegozLKHkpyWZGeSB1d5HgAAYOo2KjJ/k+RPe+9PJPlca+1gkq9b8f2dSfYnOZBk\nx4rndyTZt96KzzjjlGzbtnXziSewsLBj44XmRKWsSa28lbImw+VdXFyc2brPPPPUbN06m/18UraD\n2amUNamVV9bZqZS3UtakVl5ZZ+do5t2oyHwqyZuTvLu19pwkpyS5tbV2fu/9tiQXJbk1yV1J3t5a\nOznJ9iQvyOhGAGvat++RZ5p9VQsLO7Jnz0MzWfe0Vcqa1MpbKWsydN6lma15796H89SDuEeX7WB2\nKmVNauWVdXYq5a2UNamVV9bZmVXetcrRukWm935za+0VrbW7Mrqe5seTfCHJ9eOL+e9LctP4rmXX\nJbljvNyu3vtjU8wPAADwFRvefrn3/pOrPH3BKsvtTrJ7CpkAAADW5QMxAQCAchQZAACgHEUGAAAo\nR5EBAADKUWQAAIByFBkAAKAcRQYAAChHkQEAAMpRZAAAgHIUGQAAoBxFBgAAKEeRAQAAylFkAACA\nchQZAACgHEUGAAAoR5EBAADKUWQAAIByFBkAAKAcRQYAAChHkQEAAMpRZAAAgHIUGQAAoBxFBgAA\nKEeRAQAAylFkAACAchQZAACgHEUGAAAoR5EBAADKUWQAAIByFBkAAKAcRQYAAChHkQEAAMpRZAAA\ngHIUGQAAoBxFBgAAKEeRAQAAylFkAACAchQZAACgHEUGAAAoR5EBAADK2TbJQq21P07y4PjL/53k\n6iQ3JDmU5N4kl/Xel1prlyS5NMkTSa7qvd889cQAAMBxb8Mi01rbniS991eueO53k+zqvd/eWntf\nkte11v4oyeVJzk3yVUk+1Vr7/d77Y7OJDgAAHK8mOSLzwiSntNY+Nl7+PyR5Ue/99vH3b0nyqiSL\nSe7svT+e5PHW2v1Jzk7y6enHBgAAjmeTXCPzcJJ39t5fneRNSX7rsO8/lOS0JDvz5OlnK58HAACY\nqkmOyHwuyf1J0nv/fGvtS0m+dcX3dybZn+RAkh0rnt+RZN9aKz3jjFOybdvWTQeexMLCjo0XmhOV\nsia18lbKmgyXd3FxcWbrPvPMU7N162z280nZDmanUtakVl5ZZ6dS3kpZk1p5ZZ2do5l3kiLzxoxO\nEbustfacjArKx1tr5/feb0tyUZJbk9yV5O2ttZOTbE/ygoxuBLCqffseeabZV7WwsCN79jw0k3VP\nW6WsSa28lbImQ+ddmtma9+59OMmWma1/I7aD2amUNamVV9bZqZS3UtakVl5ZZ2dWedcqR5MUmV9L\n8hutteVrYt6Y5EtJrm+tnZTkviQ3je9adl2SOzI6ZW2XC/0BAIBZ2LDI9N6fSPJDq3zrglWW3Z1k\n9zOPBbCWyY8gjU6b28wRp+GOHgEAmzPR58gAzJNrb7wne/YfnNr6Fk7fnisuPmdq6wMAZk+RAcrZ\ns/9gHtj76NAxAIABTXL7ZQAAgLmiyAAAAOU4tQwAgMNs7tb8bq7CEBQZAACeZto3VkncXIXpUmQA\nAHgaN1Zh3rlGBgAAKEeRAQAAylFkAACAchQZAACgHEUGAAAoR5EBAADKUWQAAIByFBkAAKAcRQYA\nAChHkQEAAMpRZAAAgHIUGQAAoBxFBgAAKEeRAQAAylFkAACAchQZAACgHEUGAAAoR5EBAADKUWQA\nAIBytg0dAACYJ0sTL7m4uLip5ZMtm04DsBZFBgB4imtvvCd79h+c2voWTt+eKy4+Z2rrA0gUGQDg\nMHv2H8wDex8dOgbAulwjAwAAlKPIAAAA5SgyAABAOYoMAABQjiIDAACUo8gAAADlKDIAAEA5igwA\nAFCOIgMAAJSjyAAAAOUoMgAAQDnbJlmotfa3knwmyXcmOZTkhvHf9ya5rPe+1Fq7JMmlSZ5IclXv\n/eaZJAYAAI57Gx6Raa2dmORXkjycZEuSdyfZ1Xt/xfjr17XWzkpyeZLzkrw6ydWttZNmlhoAADiu\nTXJq2TuTvC/JX42/flHv/fbx41uSXJjkJUnu7L0/3ns/kOT+JGdPOywAAECyQZFprb0hyZ7e+8fH\nT20Z/1n2UJLTkuxM8uAqzwMAAEzdRtfIvDHJUmvtwiTnJHl/koUV39+ZZH+SA0l2rHh+R5J96634\njDNOybZtWzcdeBILCzs2XmhOVMqa1MpbKWsyXN7FxcWZrfvMM0/N1q3T3c9nlXcWWY9Epe22Utak\nVt4hs9rH5sexuB0k87Et2A5m52jmXbfI9N7PX37cWvtEkjcleWdr7fze+21JLkpya5K7kry9tXZy\nku1JXpDRjQDWtG/fI88w+uoWFnZkz56HZrLuaauUNamVt1LWZOi8SzNb8969y5fWTdNs8s4m6+ZU\n2m4rZU1q5R0+q31sHgyftdp7w+SGf20nVylrMru8a5Wjie5atsJSkiuTXD++mP++JDeN71p2XZI7\nMjpdbVfv/bFnkBcAAGBNExeZ3vsrV3x5wSrf351k9xQyAQAArMsHYgIAAOUoMgAAQDmKDAAAUI4i\nAwAAlKPIAAAA5SgyAABAOYoMAABQjiIDAACUo8gAAADlKDIAAEA5igwAAFCOIgMAAJSjyAAAAOUo\nMgAAQDmKDAAAUI4iAwAAlKPIAAAA5SgyAABAOYoMAABQjiIDAACUo8gAAADlKDIAAEA5igwAAFCO\nIgMAAJSjyAAAAOUoMgAAQDmKDAAAUI4iAwAAlKPIAAAA5SgyAABAOYoMAABQjiIDAACUo8gAAADl\nKDIAAEA5igwAAFCOIgMAAJSjyAAAAOUoMgAAQDmKDAAAUI4iAwAAlLNtowVaa1uTXJ/k+UmWkrwp\nyZeT3JDkUJJ7k1zWe19qrV2S5NIkTyS5qvd+84xyAwAAx7FJjsi8Jsmh3vvLk7wtyTuSXJNkV+/9\nFUm2JHlda+2sJJcnOS/Jq5Nc3Vo7aTaxAQCA49mGRab3/jtJfmz85fOS7Etybu/99vFztyS5MMlL\nktzZe3+8934gyf1Jzp56YgAA4Lg30TUyvffF1toNSd6T5LcyOgqz7KEkpyXZmeTBVZ4HAACYqg2v\nkVnWe39Da+1ZSe5Ksn3Ft3Ym2Z/kQJIdK57fkdHRm1WdccYp2bZt6+bSTmhhYcfGC82JSlmTWnkr\nZU2Gy7u4uDizdZ955qnZunW6+/ms8s4i65GotN1WyprUyjtkVvvY/DgWt4NkPrYF28HsHM28k1zs\n/0NJ/nbv/eokjyZZTPLp1tr5vffbklyU5NaMCs7bW2snZ1R0XpDRjQBWtW/fI1OI/3QLCzuyZ89D\nM1n3tFXKmtTKWylrMnTepZmtee/eh/PUA7jTMJu8s8m6OZW220pZk1p5h89qH5sHw2et9t4wueFf\n28lVyprMLu9a5WiSIzI3JbmhtXZbkhOTvDnJnyW5fnwx/31Jbhrftey6JHdkdMrart77Y9MIDwAA\nsNKGRab3/miS16/yrQtWWXZ3kt3PPBYAAMDafCAmAABQjiIDAACUo8gAAADlTHz7ZWAzNne3l9Ft\nLjfzM8Pe+QcAYGiKDMzItTfekz37D051nQunb88VF58z1XUCAFSkyMCM7Nl/MA/sfXToGAAAxyTX\nyAAAAOUoMgAAQDmKDAAAUI4iAwAAlKPIAAAA5SgyAABAOYoMAABQjiIDAACUo8gAAADlKDIAAEA5\nigwAAFCOIgMAAJSjyAAAAOUoMgAAQDmKDAAAUI4iAwAAlKPIAAAA5SgyAABAOYoMAABQjiIDAACU\no8gAAADlKDIAAEA5igwAAFCOIgMAAJSjyAAAAOUoMgAAQDmKDAAAUI4iAwAAlKPIAAAA5SgyAABA\nOYoMAABQjiIDAACUo8gAAADlbFvvm621E5P8epK/m+TkJFcl+dMkNyQ5lOTeJJf13pdaa5ckuTTJ\nE0mu6r3fPMPcAADAcWyjIzL/Isme3vsrknx3kl9Kck2SXePntiR5XWvtrCSXJzkvyauTXN1aO2l2\nsQEAgOPZukdkknwoyU3jxyckeTzJi3rvt4+fuyXJq5IsJrmz9/54ksdba/cnOTvJp6cfGQAAON6t\nW2R67w8nSWttR0al5m1J3rVikYeSnJZkZ5IHV3keAABg6jY6IpPW2nOTfDjJL/XeP9Ba+4UV396Z\nZH+SA0l2rHh+R5J96633jDNOybZtWzefeAILCzs2XmhOVMqa1Mo7ZNbFxcWZrfvMM0/N1q3T23cq\nZU1ml3cWWY+EfWx2KuU9Fscv+9jmHYvbQTIf24LtYHaOZt6NLvZ/VpKPJ/nx3vsnxk/f3Vo7v/d+\nW5KLktya5K4kb2+tnZxke5IXZHQjgDXt2/fIM82+qoWFHdmz56GZrHvaKmVNauUdPuvSzNa8d+/D\nGV2eNi2VsiazyjubrJsz7Ha7udf1zDNPHb9mkzqeX9vNGT6rfWweDJ+12nvD5IZ/bSdXKWsyu7xr\nlaONjsjsyugUsZ9qrf3U+Lk3J7lufDH/fUluGt+17Lokd2R0Lc2u3vtjU0kOwFFx7Y33ZM/+g1Nd\n58Lp23PFxedMdZ0AkGx8jcybMyouh7tglWV3J9k9nVgAHG179h/MA3sfHToGAEzEB2ICAADlKDIA\nAEA5igwAAFCOIgMAAJSjyAAAAOVs+IGYAADzaXOfdTL6kMfN/Mywn3sDrE+RAQDK8vlHcPxSZACA\nsnz+ERy/XCMDAACU44jMcc25xQAA1KTIHOecWwwAQEWKzHHOucUAAFTkGhkAAKAcRQYAAChHkQEA\nAMpRZAAAgHIUGQAAoBxFBgAAKEeRAQAAylFkAACAchQZAACgHEUGAAAoR5EBAADKUWQAAIByFBkA\nAKAcRQYAAChHkQEAAMpRZAAAgHIUGQAAoBxFBgAAKEeRAQAAylFkAACAchQZAACgHEUGAAAoR5EB\nAADKUWQAAIByFBkAAKCcbUMHgMksbWrpxcXFTf7Mlk2tHwCAYSkylHHtjfdkz/6DU13nwunbc8XF\n50x1nQAAzJ4iQxl79h/MA3sfHToGAABzYKIi01r7tiQ/33t/ZWvtG5LckORQknuTXNZ7X2qtXZLk\n0iRPJLmq937zjDIDAADHuQ0v9m+t/USS65OcPH7q3Ul29d5fkdGFBa9rrZ2V5PIk5yV5dZKrW2sn\nzSYyAABwvJvkrmX3J/nHefJq6Bf13m8fP74lyYVJXpLkzt774733A+OfOXvaYQEAAJIJikzv/cMZ\nnS62bOXtnR5KclqSnUkeXOV5AACAqTuSi/0PrXi8M8n+JAeS7Fjx/I4k+9ZbyRlnnJJt27YewT+/\nsYWFHRsvNCeGzDq6RfFsnHnmqdm6dXr/v5WyJrXyVsqazC7vLLIeiaHGhGrbwZHw3jCZSvvYsb7d\nHovbQeK13axKWZOjm/dIiszdrbXze++3Jbkoya1J7kry9tbayUm2J3lBRjcCWNO+fY8cwT+9sYWF\nHdmz56GZrHvahs+6uc9m2Yy9ex/OdD+bpVLWpFbeSlmTWeWdTdbNGXZMqLYdbM7w4+3khs9aaR87\ndrfbY3U7SLy2m1EpazK7vGuVo80UmeUt+sok148v5r8vyU3ju5Zdl+SOjE5X29V7f+wZ5AUAAFjT\nREWm9/6FjO5Ilt7755NcsMoyu5PsnmI2AACAVU1y1zIAAIC5osgAAADlKDIAAEA5R3LXMgAAmBOb\nu8Pa6NbSm/mZYe+6yNoUGQAASrv2xnuyZ//Bqa5z4fTtueLic6a6TqZLkQEAoLQ9+w/mgb2PDh2D\no8w1MgAAQDmKDAAAUI4iAwAAlKPIAAAA5SgyAABAOYoMAABQjtsvT93kH7DkA5kAAODIKDIzMO0P\nZfKBTAAA8FSKzAz4UCYAAFbn7J1pUWQAAOAocvbOdCgyAABwFDl7ZzrctQwAACjHERkAgJnbzHUO\nro2ASSgyAABHwbSvi0iO32sjIFFkAACOCtdFwHS5RgYAAChHkQEAAMpRZAAAgHIUGQAAoBxFBgAA\nKEeRAQAAylFkAACAchQZAACgHEUGAAAoZ9vQAQCOXUubWnpxcXGTP7NlU+sHgGOJIgMwQ9feeE/2\n7D841XUunL49V1x8zlTXCQDVKDIAM7Rn/8E8sPfRoWMAwDHHNTIAAEA5igwAAFCOIgMAAJSjyAAA\nAOUoMgAAQDmKDAAAUI7bLwPATPlgVIBZmGqRaa2dkOSXk5yd5MtJfrT3/ufT/DcAoBofjAowfdM+\nIvN9SU7qvZ/XWvu2JNeMnwOA45YPRgWYvmlfI/OyJB9Nkt77f0/y4imvHwAyOvVq8j9Pnq416R8A\n5t20j8jsTHJgxdeLrbUTeu+HDl/w3HO/edUVfOYz967y7FLOPfdb1lj+s0/5evnNarLlnzyveHN5\n1lv+s1k4ffvTnv/gu3541eV/4K2/uerzK5ffesKWfOxXTjrCPOsvf+Dhx7J46Mk37UnyrLTa8gun\nb1/n9T/y/Ctf12fyeq50+VU3HnGejZY/8PBj+advef+m8myUf+W2sNk86y3/oXf/yFO2g0nzrLf8\nyv+v6e1f9z5t3ZPmWW/55dd12vtX8tR9bBr7VzL6/1q5HWwmz3rLL7+u09q/fuCtv7nqeDit7eHf\nXP2h7Dvw5ac9/963Xbzq8mvt7yuXP+GELfnqrzrxiPKsvfy3PG2sTZ759rD82k57/1q57s3kWW/5\nrSdsyTsu7UecZ73ll1/bae1fH3zXDz9trN1MnvWWXzh9+1T3r+V1Hmme9Zef/nwmmd14W2k+s7zu\nzeZZb/lZzmdGy3921ecPf31OOGFLDh1amnj5SfP85V/+xarf37K0NL3fPLXWrknyR733D42//r+9\n9+dO7R8AAADI9E8tuzPJ9yRJa+0fJvlfU14/AADA1E8t++0k39Vau3P89RunvH4AAIDpnloGAABw\nNEz71DIAAICZU2QAAIByFBkAAKAcRQYAAChn2nctY5Naa9t77weHznG41trW3vvi+PGOJI/23p8Y\nOBZHWWvt74wfbslhH3fee//Lo5+IobXWvjfJJ3rvjwydBRiW8WA2Wmun9973D52jgmPmiExr7YOt\ntW8YOsekWmtfP/4A0bmbDLbWvjnJ51prZ4yfunD89d8fMNaaWmvfs+Lx1654/KZhEm1ea+0Xhs6w\nhi8k+WSSDyb5z4f9mSuttWe31q5prf1Ma+3UofNMorX2mvHfrx06yyRaa2cluSHJPx84yppaa6e3\n1q5srb2xtXbC+LmzW2v/behs62mt/eDQGTbSWtvSWnvd+PFprbV3tdaumsf9rbX2+dba21przx46\ny2ZUGhMqjAdJ2THh5qEDbMaQc/Bjosi01s5L8twkPzZ0lvWM3wS+t7X2e0n+OMmXkrxw4FiruS7J\n63vv+5Kk9/7bSX4wyXsGTbW2f7vi8YdWPH790Q7yDLxy6ABr+P4kn07y10l+OcmFvfeX9t5fOmys\nVb0/yeeTPJ5kXovh4a4c//2WQVNM7o1Jrsh8T1w+lORrknxHkreNf6Hx0STvHTTVxi4dOsAErk7y\nI621bUl+McmpSfYmed+gqVb3siSPJrmltfbbrbWLhg40oUpjQoXxIKk5Jnyptfbm1tp3t9Ze3Vp7\n1dCB1jL0HPxYObXskoxewF9urZ3Ye3986ECHa629NckbkvzPJO9OckLv/R2Dhlrblt77p1c+0Xv/\nw9baSUMFYhi99w8n+XBr7fSMSs0HW2t7k3yg9/7RYdM9zbbe+39MktbarUOHOUa9Nsm3J3l5a+1b\ne+93Dx1oFV/de9/VWtuSpCf5iyTn9N7/euBcGzm5tXZPRpkPJVnqvc/bBPH83vtLW2snJvneJM/t\nvT/cWvvDoYMdbvz/fU2Sa1pr/yDJv2ytvSPJh3vvPzdsumNGhfEgqTkm7E1yzvjPso8PlGUjg87B\nyxeZ1tppSf5O7/3e1trvJPknGZ0GM2/emuQDSX699/7Z1tqVG/3AgLau8fyJRzXFMai11nLYtSYZ\nXX+yfYA4Exufq7u7tfYnGf3G8IYkZw0aan3HxNHmedJa+64kn+q9L7bWrk/ypsznUfCDSdJ7X2qt\nPZrktfN4HeIqfiKjsWDZPH5a9YHx3y9J8tne+8Pjr+f6vaH3ftf4lKKlJD+cRJF5hgqNB0mhMaG1\n9vmMzi7Y1Xv/4tB5NjIPc/DyRSajU54+MH78/iS/lfksMs/L6D/4PePziU+d44u5Ptpae1eSn+u9\nPzi+2P+nk/zBwLnWckpr7fkZTQKe8njYWKv6law+Qfmbox1kUq21Fyb5Z0m+J8ndSa5P8gODhlrd\nqWtsB0u9988NG+2YsDfJtUnSe/9Ma+0jA+eZxN55nbAsG1/D8dYkDyV554pyMI8eH5/i8sYkH06S\n1tq3J9k3aKo1tNael+SHMjrN+M8yGrsuGzLTsWB8ZKPieJDM/5jwsoy22d9rrf2fJL/ae79l4Ezr\nGXwOvmVpaR5/6bM5rbWX9N7/x4qvz++93zZkpvW01r4xyY9mNLh+uvf+/QNHeorxb65+IqPDhadk\n9Cb1/ozeZA8NmW01rbVP5sly8JS7a/Xe5/Lak1W22Qt6758cMNKqWmv3ZfR6fiDJf83onPNkDstB\n0e3gZb33O5f/HjrPWipNtltrB5L8yfjLb0py3/jxUu/9vGFSra219vGMSsHXJHlO731uJ9rji3nf\nkeSBjLaHV2Z0Pdrre+9/NmS2w7XWbsvoqPGvJblhzk8j+ooVY8LLe++fGjrPalYcNfi13vtfDZ1n\nI9XGhGXLp0Qm+bbM8SmRrbVzeu/3DPXvlz4iM/5N0DcluaK19u6MJi8nJPnXSeb1DlsvHl9/8pOt\ntbdldI7pXBmXlZ9vrd16eEFMMncFsfd+QbJ6ORgq01pW2WaT0al887rNLr/5f+f4z0pzVQ6KbQdf\nKQattXvmucSMvT9PTrZ/IfP9W+2z89RCO+/KXNvVe78/ycXjfeyxJB9L8rHxe8NcFZkkP917/+Q4\n61dKzLz+onNlOUiSeS0xY8tHDW4pctTghRldd1ZhPPiKQqdEXtVa+5okv57kPx3tX3SVLjIZHSl4\ndkbXFyzfYvFQnnoXq7mwYgL7lja67XLy5AT2w4MFW0W1yXaxvGW22aRcOai0HVQqBkmhyXaSL6fI\n0aNVzPW1XcV+ebjYWvux1MiaFCoHBW+k8PHUOoL0vBQ6JbL3/prxL+d+MMnvt9bu673/6NH690sX\nmd77vUnuba39apJn9d7vbq19X5LfHzjaapYnsCdn/iewpSbbKZS32DZbrRyU2Q5Sqxgcbq4n26lX\nEitd21VpH6uUtWI5SFLmqEGZknjYKZHfUeWUyIxu+HFyRvODxaP5D5cuMiu8N6Pz9+9O8o0ZvZHN\n1W0rK01gK2VN6uUdm/ttdqzMZKDodpDMfzFIak22q5XERzO6Ccjhj5P5O32zzD5WKevhKpSDSkcN\nipXEUqdEJklr7Q8ymiMsly+nlh2Br+u9/0aS9N7fOb7od15VmcAmtbImtfKW2GaLTgYqbAeVikFS\naLJ9mLkviZVO31yhwj62rEzWKuWg8FGDCiWx2imRyah83bH8xdEuXcdKkTnUWmu99z6+q8o8v3mV\nmMCOVcqa1MpbaZtNCk0GUmM7KFUMik22S5XEYqdvLquwjy0rkbVYOah41OB5KVASU+gsiHm5Zu5Y\nKTJXZPQlP4mNAAADmElEQVSJ489K8sUk/2rgPOupNIGtlDWplbfSNpsUmQyMzf12UKwYVJtslyqJ\nKTRxWWHu97EVqmStVA5KHTWoVBKLnQUxF2PXsVJkvjXJV2d0t5qFjG4B9y2DJlpbpQlspaxJrbyV\nttmkzmQgKbAdFCsGyZy8YU2iWkksNnFZNvf72ApVslYqB2XGg7FKJXHZ3J8FMS9j17FSZC5L8ook\nb0tyU+bws1lWqDSBrZQ1qZW30jab1JkMJDW2g1ITgXl5w5pEwZK4bO4nLitU2MeWVclaZkyoNB6M\nVSqJyyqdBTHo2DXPv1XdjC+O7w2+s/f+iczvhpk8OYH9vSRvyPzu+EmtrEmtvJW22WT1ycC8mvvt\noPd+b+/9Z5Kcl+R3e+8/m+SzmcOsh3lvknPGj5ffsObN4RPCZyf52szhhPAwT5m4JHnOwHnWM/f7\n2AolshYdEyqMB8nTx4SzMv9jwqHWWkuSAmdBDDp2zfMLsxn7W2v/KKP/+Ddlvt8AKk1gK2VNauWt\ntM0mRSYDY5W2gyoTgWVzP9kuOiFMak1cKu1jlbImtcaEuR8PkrJjwvJZEF9M8sEkVw6cZz2Djl3z\nPFBuxiVJvpBkV0Y7/uWDpllfpQlspaxJrbyVttmk1mSg0nZQYiKwQqXJdqUJYVJr4lJpH6uUNak1\nJlQaD5JaY0KlsyAGHbuOiWtkeu8HMjo3L5nvwT8ZTWC/PqMJ7Fsy3xPYSlmTQnmLbbNJrclAme0g\ntW6ikNS6VqrSOeZJnWs5klr7WKWsSa0xodJ4kNQaEypdRzvo2HVMFJlKKk1gK2VN6uUtpsxkoNh2\nUG0iUGmyXWlCmBSauFTaxyplHas0JlQaD5JaY8IXe+9/1Vrb2Xv/RGvt3w0daB2Djl3z/J8IzIne\n+4He+9299//Xe7+y9/7JoTMdIyqdPpDUulaq0qlaSa3TN5mdSmNCpfEgqTUmVDoLYtCxS5EBGE61\niUClyXalCWFSa+LC7FQaEyqNB0mtMaHSdbSDjl2KDMBwqk0EKk22K00Ik1oTF2an0phQaTxICo0J\nxc6CGHTsco0MwHCqTQTKXCuVWueYV7yWg9moNCZUGg+SYmNCFUOPXYoMwHBKTQSGfsPapEoTQlhW\nZkwoNh4kxoRj0palpaWhMwDAVLXWdmY0IfzrjCaEH5nz0zOAGTImHJsUGQAAoBwX+wMAAOUoMgAA\nQDmKDAAAUI4iAwAAlKPIAAAA5fx/vJk/yk9e5u4AAAAASUVORK5CYII=\n",
       "text": [
        "<matplotlib.figure.Figure at 0x123091750>"
       ]
      }
     ],
     "prompt_number": 219
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = small_doc_experiment(texts, labels_actions)\n",
      "score_df['actions']['small_doc'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.60 (+/- 0.01)\n"
       ]
      }
     ],
     "prompt_number": 161
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = big_doc_experiment(texts, labels_actions)\n",
      "score_df['actions']['big_doc'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.56 (+/- 0.02)\n"
       ]
      }
     ],
     "prompt_number": 162
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = ml_experiment(texts, labels_actions)\n",
      "score_df['actions']['ml'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.73 (+/- 0.03)\n"
       ]
      }
     ],
     "prompt_number": 163
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = np.zeros(11)\n",
      "for i, k in enumerate(range(1, 102, 10)):\n",
      "    scores[i] = ml_experiment(texts, labels_actions, num_neighbors=k).mean()\n",
      "sb.plt.plot(range(1, 102, 10), scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 200,
       "text": [
        "[<matplotlib.lines.Line2D at 0x1159efad0>]"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAECCAYAAAD5OrxGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl81fWd7/HXOdlDVkhYwr7lyxpAUNkXEasIinbaitpW\nr9rRdpY77Z1Wnbl93NvOtDPTq9NxprYD2trWWhQr7lIUQSECKovs34SwBJKQhC0LZD3n3D/OiUYM\nOVlOcrb38/Hg8cg5v9/v8Plywvv8zvf7+35/Do/Hg4iIRA9nsAsQEZHepeAXEYkyCn4RkSij4BcR\niTIKfhGRKKPgFxGJMrHtbTTGOIEngTygAbjfWlvk2zYAWNNq96nAD6y1q3zb+wM7gcXW2oIeqF1E\nRLqg3eAHVgDx1trZxphrgcd8z2GtLQcWARhjZgE/Blb7HscB/w1c7KG6RUSki/x19cwB1gNYa3cA\nMy7fwRjjAJ4AHrLWtswG+xnwS6AscKWKiEgg+Av+NKC61WOXr/unteXAfmttIYAx5h6g0lq7wbfd\nEYhCRUQkMPwFfzWQ2np/a637sn3uAla1enwvsMQYswlvv/9vfeMBIiISAvz18efjPaNfa4yZCext\nY58Z1tptLQ+stQtafvaF/1/6xgOuyOPxeBwOfTEQEemkLgWnv+Bfh/fsPd/3+F5jzEogxVq72hiT\nDVR15S9uzeFwUFlZ092XCVnZ2alqX5iK5LaB2hfusrNT/e/UhnaD3zdY+9BlTxe02l4JXNXO8Yu6\nVJWIiPQYTeASEYkyCn4RkSij4BcRiTIKfhGRKKPgFxGJMgp+EZEoo+AXEYkyCn4RkSij4BcRiTIK\nfhGRKKPgFxGJMgp+EZEoo+AXEYkyCn4RkSij4BcRiTIKfhGRKKPgFxGJMgp+EZEoo+AXEYkyCn4R\nkSij4BfpgJ22khc2HaGqtiHYpYh0W2ywCxAJdeeq61n9+gEam9xs2l3CslnDueHqocTFxgS7NJEu\n0Rm/iB/Pv3uExiY3syYOJC7GyZ/eO8qjq3bw4aFyPB5PsMsT6TSd8Yu045OCSj46XMGonDTuWzae\n+gYXr287ztsfneRXrxzgnZ2nWLl4LCMHpQW7VJEOazf4jTFO4EkgD2gA7rfWFvm2DQDWtNp9KvAD\n4Cnfn1zAAzxorT0Q+NJFelazy81/v7wXB3D3Dbk4HQ6SE2P56qIxLJiaw9pNRewqqOTHv/2YWRMH\n8OUFo+mblhjsskX88tfVswKIt9bOBh4GHmvZYK0tt9YustYuAh4FdgKrgVsAt7V2LvCPwD/3SOUi\nPeydj09xsryWBVNzGDHw82f0AzKT+avbJ/P9ldMYNiCFbQfKeXTVdl7ecpSGRleQKhbpGH/BPwdY\nD2Ct3QHMuHwHY4wDeAJ4yFrrsda+DPylb/MI4HzAqhXpJRdqG3gl/xipyXHcvmD0FfcbNzyTH37z\nau5dOo6khFhezT/Oo6u388H+Mtzq/5cQ5S/404DqVo9dvu6f1pYD+621hS1PWGtdxphn8H4gPBeI\nQkV60wubjtDQ6OLrSyeQkhTX7r5Op4N5eTn85FszWTZ7OLV1TTz1+iH++XcfU3jqQi9VLNJxjvau\nSjDGPAZst9au9T0+aa0detk+zwM/t9Zua+P4AcAOYLy1tq6dOnRqJCFjf9EZHnkynzFD0vl/f7uA\nGKejU8dXnLvEb984yPt7SgCYOyWHe5ZNZEDf5J4oV6Jb5345ffxd1ZOP94x+rTFmJrC3jX1mtA59\nY8zXgSHW2p8CdYDb96ddlZU1HS463GRnp6p9YcLldvOLtXsA+Np1Y4hxOjrdNgdwz42GuZMHsmZj\nIVs/KWX7/tN86ZqhLJ05nKSE0LmYLpLeu7ZEQ/u6wl9Xzzqg3hiTj3dg9++MMSuNMQ8AGGOygarL\njnkRmGqMeQ/v+MDfWms13VHCwru7SjhVeZG5eYMYnZPerdcaMzidR78+nQeWTyA1OY43tp3gkVXb\nef+TUtxufcntTW6PR3MuWmm3q6cXeSL9U1ntC31VFxt5dNU2HDj4yV/OJC05PmBta2hy8ecPi3lz\n+wkam9wM7Z/CHYvHMn54ZgAq77pIee+uJD4pnmffOMi7u09x+/zR3HD1UP8HhZHs7NQe6eoRiRov\nbj5CXYOLu5bkkpYcH9DXToiL4ZY5I5mXl8NL7xWRv/80P/vjbqaNzeKri8ao/z/ALtY3sX5HMRt3\nnqK+0UXftARGDupat0gkUvCLAEdKqsjfd5qh/VNYOC2nx/6ezNQE7ls2geumD2HNxkJ2F55hb9FZ\nFk8fwi1zRpCc2P4VRNK+uoZm3v7oJH/+6CR1Dc1kpibw5QWjmT8lh7hYrVDTQsEvUc/t9vCHDQUA\n3LUklxhnzwfEyEFpPHzXVZ+u+rnho5N8sP80K+aNZMHUnF6pIZI0NLp4d9cp3tx+gov1zaQkxfHV\nRWP4yg2Gmqr2LiiMTgp+iXrvfVLKifIaZk0cSO7QjF77ex0OBzPG9WfKmH68/fEpXv/gOM9uKODd\nXSXccd0YJo3q12u1hKumZhebd5fyxrbjVF9qIjkhltvnj2Lx9CEkJcSSGB9L5I5gdJ2CX6JazaVG\nXnqviMT4GL6y6MozdHtSXGwMS2cOZ87kQax7/yhb9pby+AufMDirDxkp8aQkx5OSFEdqUhwpyXHe\nn5PjP/c4Nia6viE0u9xs2VvG6x8c53xNA4nxMdwyZwQ3XD1U3WUdoOCXqPbS+0e5WN/MHdeNISMl\nIai1pPeJ556bxrF4+hDWbj7CkVNVlJy52KFjkxJiSE2K/+yDoZ0PidTkeJITY3E6unRBSFC53G4+\n2H+a1/KPc6aqnvhYJzfNHMZN1w73O8NaPqPgl6h1rKya9/eUMjirD9dNHxLscj41tH8K3/3qVACa\nmt3U1jVRc6mR2rom389ffNyyT3F5Pc0u/5doOxyQkhRHRmoCSXExpCTHM7BvMmZYBmMGp4fUJDPw\njsN8eKicV7Yeo/x8HbExTq6fMYSbZw4nPcgf2OEotN5dkV7i9nh4dkMBHrwDuqHaVRIX6yQzNYHM\n1I6Fm8fjob7R1epDofHTD4a2PjSqahs5dbHx0zVT3tx+AqfDwfCBqZhhGZihGYwdkkFyYnCiwu3x\nsMtW8vLWY5SeuUiM08HCaYNZNmu4lsDuBgW/RKWte8s4VlbNNeP7My7Ik6gCyeFwkJQQS1JCLNkZ\nSX73z85Opby8mtr6JorLa7DFF7DFFzhWVs2xsmrW7yjG4YBhA1IxQzMYNyyT3KHpPd6P7vF4+KTo\nLC+/f5TiilqcDgdzJw9i+ZwRHWqXtE/BLwFxrKyabftPs2zOiIBPfgq0i/VNvLi5iIS4GL66aEyw\nywk6p9NBWnI8k0b2Y9JI75VEDY0ujpRWYYsvUFB8nqNl1Zw4XcOGj07iwNsdZYZlYoZlkDs0I2D9\n6x6Ph4PHz7Nuy1GOllbjAGZOGMAtc0cyUJPcAkbBL912rrqe/3hxL9UXG9l/7Bzf+9pU+qWH7tfw\nde8fpbauia8s1B2zriQhPoaJI/oycURfABqbXBSVVmOLz2OLL1BUWk1xRS1vf3wSgCHZKZ92DeUO\ny+jSh78tPs+6949ScMq7/Nd0k82KuSMZnJ0SuIYJoOCXbnK53Tz2+4+pvthI7pB0Ck5V8ZNnd/K/\n7pjKoH59gl3eFxSX17BpdwkD+yazJMLWbelJ8XExjB+e+enaQk3NLo6WVnu7hk5e4EhJFacqa9m4\n8xQAg7P6kDuspWsog/Q+V/4gKCqpYt2Woxw87r1n05TR/VgxbxTDB2qJhZ6i4Jdueen9oxw4epYZ\nJpuHVkxi/Y5i1m4u4qfP7uLvvjolpG5C7mkZ0PXAnUvGhuyAbjiIi43xdfW0fBC4OVZWjT15AVt8\nniMlVZTsusimXd57Egzql+zdf2gGZlgGGSkJnDhdw7otR9lbdBaAiSMyWTF/VLdXRRX/FPzSZbsL\nK3lrezE5WX24d+l4HA4HN80cTp+kOH67/jD/9sfd/M2X84K+AmWLD/af5khJFdNzsz/ty5bAiIt1\nkjvU29+/fPYIml1ujp+u+bRrqPBUFZt3l7B5t/eDoF9aAmervau15w5J57b5oz79EJGep+CXLqm4\nUMfTrx8iPtbJI/dcQ1LsZ5OB5k/JITkhllWvHeDfX9jDg7dO4qrc7CBWC5fqm1m7uYj4WCd3LB4b\n1FqiQWyMkzGD0xkzOJ2bZ3ln2p4or6GgpWvoVBWjB6exYu4oJozIxBGGk8nCmYJfOq2p2cUv1+3n\nUkMz9908nhGD0r6wpvuMcf1JSozlv/60j1+s28c9N41jXl7PrXrpzytbj1F9sZHb5o8K6YHnSBUb\n42R0Tjqjc9K5aebwYJcT9dTJKZ32x3cKOVFew/wpg5gzedAV95s4oi9/v3IayQmx/ObNw6zfUdyL\nVX6mZdCxf0YSN16jAV0RBb90ygf7y9i8p5Rh/VO48/pcv/uPyknj4bunk5mawAubjvCn94p69RZ4\nHo93yWW3x8OdS8YSFxvTa3+3SKhS8EuHnaqs5Xd/tiQlxPLt2yYRH9exEB2c1YdH7r6KAZlJvLHt\nBL/7s+21e85+eKgCe/ICU8dkkTc6q1f+TpFQp+CXDqlraObJdftpbHJz383j6Z/ZuVmUWelJPHz3\ndIb1T+G9PaX86tUDNLvcPVStV11DM8+/W0hsjJM7rteArkgLBb/45fF4eOatw5w+d4kbrxnW5St0\n0vvE8/07ryJ3SDofH67gP17cS31jc4Cr/czrHxznQm0jS2cOo7/WdxH5lIJf/Hp3VwkfHa5g7JB0\nbl8wqluvlZwYy3e/NpUpo/tx4Ng5Hluzh9q6pgBV+pmysxfZ8NFJstITWaqrSEQ+R8Ev7SoqrWLN\nxkLSkuN48NZJAZntGh8Xw3dun8ysiQMoKq3mX/+wi/M1DQGo1svj8fCHtwtwuT3csXhsh8ciRKKF\ngl+uqLauiV++vB+3x8O3bpnY4TXhOyI2xsl9yyZw/fQhlJy5yE+f3Un5+UsBee2dtpKDx88zaVRf\npo3VgK7I5dqdwGWMcQJPAnlAA3C/tbbIt20AsKbV7lOBHwC/AX4NDAcSgH+y1r4W+NKlJ7k9Hla9\ndoBz1Q3cNn8UE3yrNAaS0+Fg5fVjSUmO4+Utx/jps7v47lenMGxA1xfnamh0sebdQmJjHNx1fa5m\nhIq0wd8Z/wog3lo7G3gYeKxlg7W23Fq7yFq7CHgU2AmsBu4GKq2184Ebgf/qkcqlR73xwXH2Hz3H\n5FH9uHlWz/WROxwObpkzkruW5FJ9sZF/fW43BScvdPn13th+nHPVDXzpmmEM0PrtIm3yF/xzgPUA\n1todwIzLdzDGOIAngIestR7gBeCHrV6/5y7bkB5x4Pg5Xt5yjH5pCTywfEKv3JR78fQhfGv5BBqb\nXDz+/B72Fp3p9GuUn7vE+h3FZKYmsGzWiMAXKRIh/AV/GlDd6rHL1/3T2nJgv7W2EMBae9FaW2uM\nSQXWAv8QsGqlx52vaWDVqwdwOh08tGJywO6s1BEzJw7kr788GQ/wn3/ax/YDpzt8rMfj4bl3Cml2\neQd0E+I1oCtyJf4WaasGWne4Oq21l8+6uQv4eesnjDFDgZeAX1hr19AB2dmRfdOFcGhfs8vNz9bs\noeZSEw/eNplrpwzu8LGBat/i7FQG9k/jx09vZ/XrB3HGxnDzXP+XkO7YX8a+o2eZMjaLm+aOCmjf\nfji8d92h9kUff8Gfj/eMfq0xZiawt419Zlhrt7U88A36bgC+ba3d1NFCLl/dMZJkZ6eGRfuef7eQ\nQ8fPcc34/lydm9XhmgPdvv6p8fz9ymk8/sIn/GrdPk5X1rJ8zogrhnljk4tfvbSXGKeDrywYzZkz\ntQGrJVzeu65S+8JbVz/U/HX1rAPqjTH5eAd2/84Ys9IY8wCAMSYbqLrsmEeBdOCHxphNvj9aBzfE\n7bQV/PnDkwzql8w3bxwX9Kthhg1I5ZG7ryIrPZGXtx7jj+8U4r7C4m5v7SjmTFU9S2YMJScr9G73\nKBJqHL25UmI7PJH+qRzK7Ss/f4kfPfMRLreH//2NGZ2+uXVPtu98TQOPP7+HkjMXmTVxAPcuHf+5\nSWSVF+r4x6d2kJwYy08emElSQmBvMRHq7113qX3hLTs7tUtnaJrAFeUam1w8uW4/dQ0uvnnjuE6H\nfk/LTE3gB3ddxeicNLYdKOcXL+2jscn16fY1GwtpanbztUVjAh76IpFKwR/l/vB2AScralk4bTCz\nJg4MdjltSkmK43t3TGXiiEw+KTrL48/v4VJ9M3uLzrK78Ay5QzO4dsKAYJcpEjZ0ihTFtuwtZcve\nMoYPTGXl4jHBLqddifGx/M1fTGH16wf5+HAF//bcLuqbXDgdDu5eohm6Ip2h4I9SxeU1PLuhgOSE\nWL69YlJY3JkqLtbJg7dM5PeJsby3pxSA62cMYUj/0OqeEgl1Cv4odKm+mSdf3k9Ts5uHbp1Edhit\nVe90OvjGlwx9UxMoOHmBFXNHBrskkbCj4I8yHo+H37x1iIrzdSydOZypYbh6pcPhYPkcBb5IV2lw\nN8q8/fEpdtpKzNAMbpuv8BSJRgr+KHLkVBVrNx0hvU88D946kRin3n6RaKT/+VGi+lIjv3zFe1OV\nB2+dSHpK4G6qIiLhRcEfBdxuD6tfPcD5mga+vGA0ZlhmsEsSkSBS8EeBV/OPceD4eaaOyeLGa4cF\nuxwRCTIFf4Tbf/Qsr+UfJys9kfuWje+Vm6qISGhT8Eewc9X1rHrtIDExDr592yT6JPbeTVVEJHQp\n+CNUs8vNL1/eT21dE3den8uIgWnBLklEQoQmcEUgW3yeP7xdwKlK71LGC6bmBLskEQkhCv4IcqG2\ngRc2HWH7gXIA5uUN4k4tYCYil1HwR4Bml5uNO0/xytZj1De6GDEwlbtvMIzKUfeOiHyRgj/MHTrh\n7dYpPXORPomxfONGw/y8HJxOneWLSNsU/GHqXHU9z797hI8OV+AAFk4bzO3zR5GSpCt3RKR9Cv4w\n0+xys+Gjk7yWf5yGJhejctK4+wZdtSMiHafgDyP7j53lD28XUn7uEqnJcdy5ZCxzJg/SpCwR6RQF\nfxg4U1XH8xuPsLOgEocDFk8fwop5IzUhS0S6RMEfwpqaXazfUcwb207Q2Oxm7JB07lqSy7ABqcEu\nTUTCmII/RO0tOsNzbxdScaGO9D7xfPPGMcycOEDX5ItIt3Uo+I0xTuBJIA9oAO631hb5tg0A1rTa\nfSrwA2vtKt/2a4F/sdYuCmThkariQh1r3ilkz5EzOB0Obrh6KLfOHUlSgj6jRSQwOpomK4B4a+1s\nX5A/5nsOa205sAjAGDML+DGw2vf4+8DdQG2A6444jU0u3tx+gje3F9PscjNuWAZ3LcllcHZKsEsT\nkQjT0eCfA6wHsNbuMMbMuHwHY4wDeAK401rr8T19BLgd+H0Aao1IHo+HPYVn+OPGQs5U1ZOREs/X\nrhvLNeP7q1tHRHpER4M/Dahu9dhljHFaa92tnlsO7LfWFrY8Ya19yRgzovtlRqbyc5d47p1C9h09\nS4zTwU3XDmP5nBEkxqtbR0R6TkcTphpofSnJ5aEPcBfw864Wkp0d2VeqtG5ffUMzL2wsYN3mIppd\nbqaOzeZbt01maBhfrRPJ718ktw3UvmjU0eDPx3tGv9YYMxPY28Y+M6y127paSGVlTVcPDXnZ2alU\nVtbg8XjYaStZ824h56ob6JuWwB3XjWW6ycbhCN9/g5b2RaJIbhuofeGuqx9qHQ3+dcASY0y+7/G9\nxpiVQIq1drUxJhuoaud4TzvbokLNpUZWvXqAA8fPExvjYNns4dw8cwQJ8THBLk1EokyHgt83WPvQ\nZU8XtNpeCVx1hWOPA7O7WF9E8Hg8/ObNwxw4fp5Jo/py1/W5DOibHOyyRCRKaRSxF2zYUcyeI2cY\nPzyT//mVKVpbR0SCSvfc7WEV5y/x1Cv7SEqI5b6bxyv0RSToFPw9yOV2s/r1g9Q3uvj6Dbn0TUsM\ndkkiIgr+nvTW9mKKSqqZN3Uw104YEOxyREQABX+POXG6hle2HiMjJZ6HvpynWbgiEjIU/D2gscnF\nqtcO4HJ7uO/mCaQmxwe7JBGRTyn4e8CLm4soO3uJxdOHMHFk32CXIyLyOQr+ADtw7Bzv7DzFoH7J\nfGXh6GCXIyLyBQr+AKqta+LpNw4S43TwwPIJxMdpVq6IhB4FfwA9u8FyobaRW+aOZMTAtGCXIyLS\nJgV/gGw/eJoPD1UwenAaS2cOC3Y5IiJXpOAPgHPV9fz+zwUkxMXwwLIJxDj1zyoioUsJ1U1uj4en\n3zhEXUMzK68fS/9MLb4mIqFNwd9N73x8ikMnzjN1TBbz8gYFuxwREb8U/N1QUlnLi5uLSE2O45s3\njdPsXBEJCwr+Lmp2uVn92kGaXW7uuXEc6X00O1dEwoOCv4te3nKM4opa5uUNYlpudrDLERHpMAV/\nFxScvMBb20+QnZHIHYvHBrscEZFOUfB3Ul1DM0+9fhAccP+yCSQl6CZmIhJeFPyd9MeNhZypqmfp\nzOGMHZIR7HJERDpNwd8Juwoq2bq3jGEDUrh17shglyMi0iUK/g6qqm3gmbcOExvj5IHlE4mN0T+d\niIQnpVcHeDwefvPWYWrrmvjKwtEMzuoT7JJERLpMwd8B731Syt6is0wYkcniGUOCXY6ISLe0e0mK\nMcYJPAnkAQ3A/dbaIt+2AcCaVrtPBX4ArAZ+2dYx4aj8/CXWbCwkOSGW/7F0PE7NzhWRMOfvjH8F\nEG+tnQ08DDzWssFaW26tXWStXQQ8CuzEG/q3AQltHRNuXG43T712kMYmN1//kqFvWmKwSxIR6TZ/\nwT8HWA9grd0BzLh8B2OMA3gCeMha6/Ed81Z7x4SLN7edoKi0mmsnDODaCQOCXY6ISED4C/40oLrV\nY5ev+6e15cB+a21hJ44JecfKqnk1/ziZqQncfUNusMsREQkYf9NOq4HUVo+d1lr3ZfvcBfy8k8d8\nQXZ2qr9dek19YzO/fvpDXG4P37tzOiOG9u32a4ZS+3pCJLcvktsGal808hf8+XjP6NcaY2YCe9vY\nZ4a1dlsnj/mCysqajuzWK/6woYCSylqWzBhKTmZit2vLzk4NqfYFWiS3L5LbBmpfuOvqh5q/4F8H\nLDHG5Pse32uMWQmkWGtXG2OygSp/x3SpsiDZf/QsG3edIierD19eMCrY5YiIBFy7we8brH3osqcL\nWm2vBK7qwDFhobauiaffPESM08EDyyYQHxcT7JJERAIu7AZde4rH4+F3f7ZU1TayYt5Ihg9Uv6CI\nRCYFv8/2A+V8fLiCMYPTuena4cEuR0Skxyj4gbNV9Tz7tiUhPob7l0/A6dTsXBGJXFEf/G6Ph6ff\nOEhdg4s7F4+lf0ZSsEsSEelRUR/8b390ksPFF5g2Nou5eYOCXY6ISI+L6uA/VVnLn94rIi05jm/e\nOA6HFmATkSgQtcHf7HKz+rWDNLs83HPTeNL6xAe7JBGRXhG1wb/jYDknK2qZlzeIqWOzgl2OiEiv\nidrg37y7BAewbPaIYJciItKrojL4i8trKCqtZtKofmTrKh4RiTJRGfyb95QCsGja4CBXIiLS+6Iu\n+Osamtl24DR90xLIG90v2OWIiPS6qAv+HQfLaWh0MX9KjmboikhUiqrg93g8bNpdgtPhYF5eTrDL\nEREJiqgK/qNl1ZysqGXa2CwyUxOCXY6ISFBEVfBv3l0CwEIN6opIFIua4L9Y38SHhyron5HE+BGZ\nwS5HRCRooib4P9h3mqZmNwum5eDUmjwiEsWiIvg9Hg+b95QQG+NgzmStwCki0S0qgr/g5AXKzl5i\nhulPWrIWYxOR6BYVwb9Jg7oiIp+K+OCvvtjITltJTlYfxg5JD3Y5IiJBF/HBv3VfGS63h4VTc3Sj\nFRERIjz43R4Pm3eXEB/rZPakgcEuR0QkJMS2t9EY4wSeBPKABuB+a21Rq+1XA48BDqAE+IZv01PA\nGKAJ+Btr7SeBL92/g8fOcaaqnrl5g0hOjAtGCSIiIcffGf8KIN5aOxt4GG/IA2CMcQCrgHustfOA\njcBI4AHgku+YB4Bf90ThHdEyqKvll0VEPuMv+OcA6wGstTuAGa225QJnge8aYzYDGdZaC0xodUwB\nMNgYkxbguv06V13PJ0fOMnxAKiMGpvb2Xy8iErL8BX8aUN3qscvX/QOQBcwG/hO4HlhsjFkE7AGW\nARhjZgLZQJ9AFt0RW/aW4fZ4WDhNg7oiIq2128ePN/Rbny47rbVu389ngSO+s3yMMevxfiN4HBhv\njNkC5AMFwDl/hWRnB+6s3OVys3VfGUkJsdw8fwxJCf6a2fMC2b5QFMnti+S2gdoXjfwlYj6wHFjr\nO3vf22rbUSDFGDPaN+A7D++g7jXAu9ba7xpjZgDXWGsb/BVSWVnTpQa0ZXdBJWer6ll01WBqq+uo\nDdgrd012dmpA2xdqIrl9kdw2UPvCXVc/1PwF/zpgiTEm3/f4XmPMSiDFWrvaGHMf8JxvoDffWvuW\nMaYv8Lwx5lGgHu8Ab6/atMc3U3eqBnVFRC7XbvBbaz3AQ5c9XdBq+ybg2suOOQcsCVSBnVVxoY4D\nR88xZnA6Q/unBKsMEZGQFXETuN7fU4oHWDhNt1YUEWlLRAV/s8vNlr2l9EmMZYbpH+xyRERCUkQF\n/66CSmouNTFn8iDi42KCXY6ISEiKqOBvuafugqnq5hERuZKICf6ysxc5XHyB8cMzGdSv1+eLiYiE\njYgJ/s27SwHdbEVExJ+ICP7GJhf5+8pI6xPPtLFZwS5HRCSkRUTwf3S4gksNzczLG0RsTEQ0SUSk\nx0RESm7eXYIDWDBFg7oiIv6EffAXl9dQVFrN5NH9yMpICnY5IiIhL+yDf/Me36Cu1uUREemQsA7+\nuoZmth04Td+0BPJG9wt2OSIiYSGsg3/HwXIaGl3Mn5KD06mbrYiIdETYBr/H42HT7hKcDgfz8jSo\nKyLSUWEb/EfLqjlZUcu0sVlkpiYEuxwRkbARtsHfsi6PZuqKiHROWAb/xfomPjxUQf+MJMaPyAx2\nOSIiYSXN3fdHAAAIfElEQVQsg/+DfadpanazYFoOTocGdUVEOiPsgt/j8bB5TwmxMQ7mTh4U7HJE\nRMJO2AV/wckLlJ29xIxx/UlNjg92OSIiYSfsgn9Ty6CuZuqKiHRJWAV/9cVGdtpKBmf1YeyQ9GCX\nIyISlsIq+LfuK8Pl9rBw2mAcGtQVEemSsAl+t8fD5t0lxMc5mTVxYLDLEREJW7HtbTTGOIEngTyg\nAbjfWlvUavvVwGOAAygBvgE0A08BuYAbeMBaa7tb6MFj5zhTVc+8vEEkJ7ZbtoiItMPfGf8KIN5a\nOxt4GG/IA2CMcQCrgHustfOAjcBI4Aagj7V2LvAj4J8DUegmzdQVEQkIf8E/B1gPYK3dAcxotS0X\nOAt81xizGcjwndnXAem+D4Z0oLG7RZ6rrueTI2cZPjCVkYPSuvtyIiJRzV+fSRpQ3eqxyxjjtNa6\ngSxgNvAdoAh43RjzMbAFSAQOA/2A5d0t8v1PSnF7PCzS2b6ISLf5C/5qILXV45bQB+/Z/pGW/ntj\nzHq83whmAfnW2n8wxgwB3jXGTLLWtnvmn52d2ubzLpebrftOk5wYy9J5o0lKCM/+/Su1L1JEcvsi\nuW2g9kUjfymaj/eMfa0xZiawt9W2o0CKMWa0b8B3HvA03u6hlm8J54E4IMZfIZWVNW0+v6ugknPV\n9Vx31WBqq+uo9fdCISg7O/WK7YsEkdy+SG4bqH3hrqsfav6Cfx2wxBiT73t8rzFmJZBirV1tjLkP\neM7Xn59vrX3TGPMB8BtjzBa8of+ItbauS9XRavllzdQVEQmIdoPfWusBHrrs6YJW2zcB1152zAXg\ntkAUV3Ghjv3HzjFmSDpD+qcE4iVFRKJeSE/gem+P92x/kc72RUQCJmSDv6nZzda9ZfRJjGXGuOxg\nlyMiEjFCNvh3FVRSc6mJuXmDiIv1OzYsIiIdFLLB3zKou0DdPCIiARWSwV965iL25AXGD89kYN/k\nYJcjIhJRQjL4N7cM6mqmrohIwIVc8Dc0ufhg32nS+8QzdWxWsMsREYk4IRf8Hx2q4FJDM/OmDCI2\nJuTKExEJeyGXrJv3lOAA5k/JCXYpIiIRKaSC/8TpGo6WVjN5dD+y0pOCXY6ISEQKqeBvmamrm62I\niPSckAn+pmYX2w6W0zctgbxR/YJdjohIxAqhxe0djBqUxpzJA3E6HcEuRkQkYoVM8MfFOvn7ldOC\nXYaISMQLma4eERHpHQp+EZEoo+AXEYkyCn4RkSij4BcRiTIKfhGRKKPgFxGJMgp+EZEoo+AXEYky\n7c7cNcY4gSeBPKABuN9aW9Rq+9XAY4ADKAG+AdwJfNO3SxIwBRhgra0OePUiItJp/s74VwDx1trZ\nwMN4Qx4AY4wDWAXcY62dB2wERlprn7HWLrLWLgI+Bv5aoS8iEjr8Bf8cYD2AtXYHMKPVtlzgLPBd\nY8xmIMNaa1s2GmNmABOttU8FtGIREekWf8GfBrQ+W3f5un8AsoDZwH8C1wOLjTGLWu37KPB/AlSn\niIgEiL/grwZSW+9vrXX7fj4LHLFezXi/GcwAMMZkALnW2vcCXbCIiHSPv2WZ84HlwFpjzExgb6tt\nR4EUY8xo34DvPKClW2c+3j7/jnJkZ6f63yuMqX3hK5LbBmpfNPIX/OuAJcaYfN/je40xK4EUa+1q\nY8x9wHO+gd58a+1bvv1ygaI2Xk9ERILM4fF4gl2DiIj0Ik3gEhGJMgp+EZEoo+AXEYkyCn4RkSjj\n76qeHuVvLaBwZIyJA34NDAcSgH8CDgHPAG5gP/Ada23YjqobY/oDO4HFeNv0DJHTtkfwXsIcB/wX\n3kuanyEC2uf7//YU3qvu3MADgIswb58x5lrgX6y1i4wxY2ijPcaYB4BvAc3AP1lr3whawZ10Wfum\nAk/gfd8agG9Yays6275gn/FfcS2gMHYXUGmtnQ/cCPwCb7se9T3nAG4NYn3d4vtg+2/gIt62PE7k\ntG0hMMv3+7gQGEUEvXfADUAfa+1c4EfATwjz9hljvg+sxnuSBW38PhpjBgJ/jXelgS8BPzXGxAej\n3s5qo30/B/7KtxbaS8APjDED6GT7gh387a0FFK7WAj/0/ewEmoCrrLXv+557C+8SF+HqZ8AvgTLf\n40hq2w3APmPMy8BrwKvA9AhqXx2Q7pt3kw40Ev7tOwLcjjfkoe3fx6vxzjNq8i0YeQRvL0M4uLx9\nd1hrWybSxuF9T6+hk+0LdvC3txZQWLLWXrTW1hpjUvF+CPwjn/93rsX7ny7sGGPuwfttZoPvKQef\n/UJCGLfNJxuYDvwF8CDwHJHVvnwgETiM91vbE4R5+6y1L+Ht3mjRuj01eNuTBlS18XzIu7x91trT\nAMaY2cB3gH+nC+0Ldsi2txZQ2DLGDAXeBX5nrf0j3v7GFqnAhaAU1n334p3JvQmYCvwWb1i2COe2\nAZwBNlhrm621BUA9n/8PFO7t+z7eM0OD9/37Hd6zxhbh3j74/P+1NLztuTxnUoHzvVlUIBljvob3\nW/dSa+1ZutC+YAd/PrAUoI21gMKSr79tA/B9a+0zvqd3G2MW+H6+CXi/rWNDnbV2gbV2oa9/cQ/e\nG++sj4S2+WzFOy6DMSYHSAY2RlD7+vDZN+zzeC/uiIjfzVbaas+HwDxjTIIxJh0Yj3fgN+wYY+7G\ne6a/0Fp73Pd0p9sX1Kt6aGMtoGAWEyCP4j1L/KExpqWv/2+BJ3wDLgeBF4NVXIB5gO8BqyOhbdba\nN4wx840xH+I9Kfo2cJwIaR/e8ZnfGGO24D3TfwTv1VmR0L6WK5G+8Pvou6rnCWAL3vf1UWttY5Dq\n7CqPrxv8P4ATwEvGGIDN1tr/29n2aa0eEZEoE+yuHhER6WUKfhGRKKPgFxGJMgp+EZEoo+AXEYky\nCn4RkSij4BcRiTIKfhGRKPP/Ac5k3IzVG03WAAAAAElFTkSuQmCC\n",
       "text": [
        "<matplotlib.figure.Figure at 0x117a69390>"
       ]
      }
     ],
     "prompt_number": 200
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Settings"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now, let's turn to the settings annotations."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "labels_settings = list(read_labels(\"data/dream_sets.txt\"))\n",
      "labels_settings, texts = zip(*match_labels_documents(documents, labels_settings))\n",
      "labels_settings, texts = np.array(labels_settings), np.array(texts)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 201
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The label set follows the following distribution:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "label_counts = pd.Series(Counter(flatten_labels(list(labels_settings))))\n",
      "label_counts.plot(kind='bar', figsize=(14, 6))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 222,
       "text": [
        "<matplotlib.axes._subplots.AxesSubplot at 0x12467bf50>"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAzIAAAGECAYAAADtORVsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X2UZHdZJ/DvZIZkCA55OduC7LJmfeFZXMVABJR3XATh\ngHjEjXoEBZUIAkIU3wILrie8uApZwsKqBMzCKphEEJUFUVwIRCGiBsmiP4ig6Eq0dWYgJBlIenr/\nqGroTHpmumq6uu6v+vM5Z06qb1fd+/zy9L11v3Vfatfq6moAAAB6ctK8CwAAAJiUIAMAAHRHkAEA\nALojyAAAAN0RZAAAgO4IMgAAQHf2HOuXVbU7yWuS3CPJapKnJflckkuTHE5ybZJntNZWq+qpSc5L\ncmuSC1trb5th3QAAwA52vCMyj01yuLX2oCTPT/LiJC9LckFr7SFJdiV5fFXdNcmzkjwgyaOSvKSq\nTp5d2QAAwE52zCDTWntrkh8e/3hWkgNJzmmtXTme9vYkj0hy3yRXtdZuaa19Jsl1Se41k4oBAIAd\n77jXyLTWVqrq0iSvSPJrGR2FWXNDktOS3DnJpzeYDgAAsOWOeY3Mmtbak6vqLkmuTrJ33a/unORg\nks8k2bdu+r6Mjt4c1a23rqzu2bN7smoBAICdZNfRfnG8i/2flOTftNZekuTmJCtJPlhVD22tvSfJ\no5O8K6OA86KqOiWjoHPPjG4EcFQHDtw00QimsbS0L8vLN8x8ObPSe/2JMQyFMQxD72Povf7EGIbC\nGOav9/oTYxiKWY9haWnfUX93vCMyVyS5tKrek+QOSZ6d5K+SvGZ8Mf9HklwxvmvZxUnem9Hpahe0\n1j6/FcUDAAAc6ZhBprV2c5Lv2uBXD9vguZckuWRrygIAADg6X4gJAAB0R5ABAAC6I8gAAADdEWQA\nAIDuCDIAAEB3BBkAAKA7ggwAANAdQQYAAOiOIAMAAHRHkAEAALojyAAAAN0RZAAAgO4IMgAAQHcE\nGQAAoDuCDAAA0B1BBgAA6I4gAwAAdEeQAQAAuiPIAAAA3RFkAACA7ggyAABAdwQZAACgO4IMAADQ\nHUEGAADojiADAAB0R5ABAAC6I8gAAADdEWQAAIDuCDIAAEB3BBkAAKA7ggwAANAdQQYAAOiOIAMA\nAHRHkAEAALojyAAAAN0RZAAAgO4IMgAAQHcEGQAAoDuCDAAA0B1BBgAA6I4gAwAAdEeQAQAAuiPI\nAAAA3dkz7wIA+rM61atWVlameO2uqZYFAItOkAGYwkWXXZPlg4dmNv+l0/fm/HPPntn8AaB3ggzA\nFJYPHsr1+2+edxkAsGO5RgYAAOiOIAMAAHTnmKeWVdUdkrwuyZcnOSXJhUn+PsnvJvno+Gmvbq1d\nXlVPTXJekluTXNhae9vMqgYAAHa0410j871JlltrT6qqM5J8KMl/SfKy1trL155UVXdN8qwk5yS5\nY5L3VdXvt9Y+P6O6AQCAHex4QebyJFeMH5+U5JaMwkpV1eOTfCzJc5LcL8lVrbVbktxSVdcluVeS\nD86kagAAYEc75jUyrbUbW2ufrap9GYWa5yW5OslzW2sPTfLxJC9Msi/Jp9e99IYkp82mZAAAYKc7\n7u2Xq+ruSd6c5FWttTdV1WmttbXQ8pYkr0xyZUZhZs2+JAeONd8zzjg1e/bsnq7qCSwt7Tv+kwas\n9/oTYxgKY9g6oy+2nL0zz7xTdu+e/XZyEkPpwYkwhmEwhvnrvf7EGIZiXmM43sX+d0nyziQ/0lr7\nP+PJ76iqH22t/UmSR2R0+tjVSV5UVack2ZvknkmuPda8Dxy46URrP66lpX1ZXr5h5suZld7rT4xh\nKIxhq61uy1L2778xya5tWdZmDKsH0zGGYTCG+eu9/sQYhmLWYzhWSDreEZkLMjpF7AVV9YLxtOck\nuaiqbknyqSTnjU8/uzjJezM6Xe0CF/oDAACzcswg01p7dpJnb/CrB23w3EuSXLJFdQEAAByVL8QE\nAAC6I8gAAADdEWQAAIDuCDIAAEB3BBkAAKA7ggwAANAdQQYAAOiOIAMAAHRHkAEAALojyAAAAN0R\nZAAAgO4IMgAAQHcEGQAAoDuCDAAA0B1BBgAA6I4gAwAAdEeQAQAAuiPIAAAA3RFkAACA7ggyAABA\ndwQZAACgO4IMAADQHUEGAADojiADAAB0R5ABAAC6I8gAAADd2TPvAgAAoH+rE79iZWVlitftmng5\ni0qQAQCALXDRZddk+eChmcx76fS9Of/cs2cy714JMgAAsAWWDx7K9ftvnncZO4ZrZAAAgO4IMgAA\nQHcEGQAAoDuCDAAA0B1BBgAA6I4gAwAAdEeQAQAAuiPIAAAA3RFkAACA7ggyAABAdwQZAACgO3vm\nXQAATGd14lesrKxM8bpdEy8HgNkTZADo1kWXXZPlg4dmMu+l0/fm/HPPnsm8AThxggwA3Vo+eCjX\n77953mUAMAeukQEAALojyAAAAN0RZAAAgO4IMgAAQHcEGQAAoDvHvGtZVd0hyeuSfHmSU5JcmOQv\nk1ya5HCSa5M8o7W2WlVPTXJekluTXNhae9sM6wYAAHaw4x2R+d4ky621hyT51iSvSvKyJBeMp+1K\n8viqumuSZyV5QJJHJXlJVZ08u7IBAICd7HjfI3N5kivGj09KckuS+7TWrhxPe3uSRyZZSXJVa+2W\nJLdU1XVJ7pXkg1tfMgAAsNMdM8i01m5Mkqral1GoeX6SX1z3lBuSnJbkzkk+vcH0ozrjjFOzZ8/u\nKUqezNLSvpkvY5Z6rz8xhqEwhq2zsrKyLcs588w7Zffu2W8nJzGUHiTb04ch9iAZVh+mZQzz13v9\nybDGYJu0/Y53RCZVdfckb07yqtbaG6vqv6779Z2THEzymSTrR7AvyYFjzffAgZsmr3ZCS0v7srx8\nw8yXMyu9158Yw1AYw1Zb3Zal7N9/Y0Zn8A7DsHqQbEcfhtaDZIh9mJwxzF/v9SdDHINt0qzmfzTH\nvEamqu6S5J1JfrK1dul48p9X1UPHjx+d5MokVyd5cFWdUlWnJblnRjcCAAAA2HLHOyJzQUaniL2g\nql4wnvbsJBePL+b/SJIrxnctuzjJezMKRxe01j4/q6IBAICd7XjXyDw7o+BypIdt8NxLklyyNWUB\nAAAcnS/EBAAAuiPIAAAA3RFkAACA7ggyAABAdwQZAACgO4IMAADQHUEGAADojiADAAB0R5ABAAC6\nI8gAAADdEWQAAIDuCDIAAEB3BBkAAKA7ggwAANAdQQYAAOiOIAMAAHRHkAEAALojyAAAAN0RZAAA\ngO4IMgAAQHcEGQAAoDuCDAAA0B1BBgAA6I4gAwAAdEeQAQAAuiPIAAAA3RFkAACA7ggyAABAdwQZ\nAACgO4IMAADQHUEGAADojiADAAB0R5ABAAC6I8gAAADdEWQAAIDuCDIAAEB3BBkAAKA7ggwAANAd\nQQYAAOiOIAMAAHRHkAEAALojyAAAAN0RZAAAgO4IMgAAQHcEGQAAoDuCDAAA0B1BBgAA6I4gAwAA\ndGfPZp5UVfdP8tLW2sOr6t5JfifJx8a/fnVr7fKqemqS85LcmuTC1trbZlIxAACw4x03yFTVTyZ5\nYpLPjiedk+TlrbWXr3vOXZM8a/y7OyZ5X1X9fmvt81tfMgAAsNNt5ojMdUm+I8kbxj+fk+QeVfX4\njI7KPCfJ/ZJc1Vq7JcktVXVdknsl+eDWlwwAAOx0xw0yrbU3V9VZ6yZ9IMmvtNb+vKouSPLCJNck\n+fS659yQ5LRjzfeMM07Nnj27J694QktL+2a+jFnqvf7EGIbCGLbOysrKtiznzDPvlN27Z7+dnMRQ\nepBsTx+G2INkWH2YljHMX+/1J8Mag23S9tvUNTJHeEtrbS20vCXJK5NcmWT9CPYlOXCsmRw4cNMU\ni57M0tK+LC/fMPPlzErv9SfGMBTGsNVWt2Up+/ffmGTXtixrM4bVg2Q7+jC0HiRD7MPkjGH+eq8/\nGeIYbJNmNf+jmeauZe+oqvuOHz8io9PHrk7y4Ko6papOS3LPJNdOMW8AAIDjmuSIzFrMfFqSV1XV\nLUk+leS81tpnq+riJO/NKBxd4EJ/AABgVjYVZFprf5PkAePHH0ryoA2ec0mSS7ayOAAAgI34QkwA\nAKA7ggwAANAdQQYAAOiOIAMAAHRHkAEAALojyAAAAN0RZAAAgO4IMgAAQHcEGQAAoDuCDAAA0J09\n8y4AAIATsTrxK1ZWVqZ6XbJritfAbAgyAACdu+iya7J88NDM5r90+t6cf+7ZM5s/TEOQAQDo3PLB\nQ7l+/83zLgO2lWtkAACA7ggyAABAdwQZAACgO4IMAADQHUEGAADojiADAAB0R5ABAAC6I8gAAADd\nEWQAAIDuCDIAAEB3BBkAAKA7ggwAANAdQQYAAOiOIAMAAHRHkAEAALojyAAAAN0RZAAAgO4IMgAA\nQHcEGQAAoDuCDAAA0B1BBgAA6I4gAwAAdEeQAQAAuiPIAAAA3RFkAACA7ggyAABAdwQZAACgO4IM\nAADQHUEGAADojiADAAB0R5ABAAC6I8gAAADdEWQAAIDuCDIAAEB39mzmSVV1/yQvba09vKq+Ksml\nSQ4nuTbJM1prq1X11CTnJbk1yYWttbfNqGYAAGCHO+4Rmar6ySSvSXLKeNLLk1zQWntIkl1JHl9V\nd03yrCQPSPKoJC+pqpNnUzIAALDTbebUsuuSfEdGoSVJ7tNau3L8+O1JHpHkvkmuaq3d0lr7zPg1\n99rqYgEAAJJNBJnW2pszOl1sza51j29IclqSOyf59AbTAQAAttymrpE5wuF1j++c5GCSzyTZt276\nviQHjjWTM844NXv27J5i8ZNZWtp3/CcNWO/1J8YwFMawdVZWVrZlOWeeeafs3j377eQkhtKDZHv6\nMOserK6u5vDhw8d/4jorKys588xTN/38k046Kbt27Tr+E7fZkP6WpjWUMdgmDcMibJOmNa8+TBNk\n/ryqHtpae0+SRyd5V5Krk7yoqk5JsjfJPTO6EcBRHThw0xSLnszS0r4sL98w8+XMSu/1J8YwFMaw\n1Va3ZSn799+Y2x4En69h9SDZjj7Mvgerueiya7J88NBM5r50+t6cf+7ZGdLfUTLEv6XJDWsMtknD\nsAjbpMnNug/HCkmTBJm17vx4kteML+b/SJIrxnctuzjJezM6Xe2C1trnp6wXAHaM5YOHcv3+m+dd\nBkB3NhVkWmt/k9EdydJa+1iSh23wnEuSXLKFtQEAAGzIF2ICAADdEWQAAIDuCDIAAEB3BBkAAKA7\nggwAANAdQQYAAOiOIAMAAHRHkAEAALojyAAAAN0RZAAAgO4IMgAAQHcEGQAAoDuCDAAA0B1BBgAA\n6I4gAwAAdGfPvAsAAACGYHXiV6ysrEzxul0TL2cjggwAAJAkueiya7J88NBM5r10+t6cf+7ZWzY/\nQQYAAEiSLB88lOv33zzvMjbFNTIAAEB3BBkAAKA7ggwAANAdQQYAAOiOIAMAAHRHkAEAALojyAAA\nAN0RZAAAgO4IMgAAQHcEGQAAoDuCDAAA0B1BBgAA6I4gAwAAdEeQAQAAuiPIAAAA3RFkAACA7ggy\nAABAdwQZAACgO4IMAADQHUEGAADojiADAAB0Z8+8C4DttTrxK1ZWVqZ43a6JlwMAwOYJMuw4F112\nTZYPHprJvJdO35vzzz17JvMGAOCLBBl2nOWDh3L9/pvnXQYAACfANTIAAEB3BBkAAKA7ggwAANAd\nQQYAAOiOIAMAAHRHkAEAALoz9e2Xq+rPknx6/OPHk7wkyaVJDie5NskzWmuTf/sgAADAcUwVZKpq\nb5K01h6+btpvJ7mgtXZlVf2PJI9P8ltbUiUAAMA60x6R+fokp1bV743n8bwk92mtXTn+/duTPDKC\nDAAAMAPTXiNzY5JfaK09KsnTkvzaEb//bJLTTqQwAACAo5n2iMxHk1yXJK21j1XVvyS597rf70ty\n8FgzOOOMU7Nnz+4pF795S0v7Zr6MWeq9/mRYY1hZWZn5Ms48807ZvXv2f9uTGlIfpjWUMWzH31Ey\nzL+lofQgWYz1eRHGMK0h/S1NayhjsE0ahkVYn3sbw7RB5ilJ7pXkGVV1t4yCyzur6qGttfckeXSS\ndx1rBgcO3DTlojdvaWlflpdvmPlyZqX3+pMhjmH295/Yv//GJLtmvpxJDK8PkxvWGLbnPiZD+1sa\nVg+SxVifF2EMkxve39LkhjUG26RhWIT1eXhjOFZYnTbIvDbJr1bV2jUxT0nyL0leU1UnJ/lIkium\nnDcAAMAxTRVkWmu3JnnSBr962AlVAwAAsAm+EBMAAOiOIAMAAHRn2mtkAABgi0x+kfnoDlvTXJw+\nnBsWcGIEGQAA5u6iy67J8sFDM5v/0ul7c/65Z89s/mw/QQYAgLlbPngo1++/ed5l0BHXyAAAAN0R\nZAAAgO4IMgAAQHcEGQAAoDuCDAAA0B1BBgAA6I4gAwAAdEeQAQAAuiPIAAAA3RFkAACA7ggyAABA\ndwQZAACgO4IMAADQHUEGAADojiADAAB0Z8+8CwB2otWJX7GysjLF63ZNvBwAoA+CDDAXF112TZYP\nHprJvJdO35vzzz17JvMGAIZBkAHmYvngoVy//+Z5lwEAdMo1MgAAQHcEGQAAoDuCDAAA0B1BBgAA\n6I4gAwAAdEeQAQAAuuP2ywDACfAFt8B8CDIAwAnxBbfAPAgyAMAJ6fsLbic/opQ4qgRDIMgAADva\nLI8oJY4qwawIMgDAjtb3ESXYuQSZbbNdF0MmDl0DALDoBJlt5NA1AABsDUFmG/V/6NotNgEAGAZB\nhom4xSYAAEMgyDCR/o8qASOu2wOgb4IMwA7luj0AeibIAOxQjrAC0LOT5l0AAADApAQZAACgO4IM\nAADQHdfIQHd8nw8AQCdBZprbfdp5Y3H5Ph8AYKfrJMi4TSis525TAMBO102QseMGAACs2dIgU1Un\nJXl1knsl+VySH2qt/fVWLgMAAGCr71r27UlObq09IMlPJ3nZFs8fAABgy4PMA5O8I0laax9I8g1b\nPH8AAIAtv0bmzkk+s+7nlao6qbV2+MgnnnPO1244gz/902s3nH75y78/K4dvfwey737u6zd8/pt+\n8fs2nH6051/+8u/P7/3yyZuuZ9L6zznn6/KZGz9/uzFsVf1v+sXvy+6Tdt1uDFtX/+j5R46ht/qT\n245hK+tPcrsxzKL+5Itj2Or6k9uOYVb1J6Mx/Kcf+5/HrWe9zY53bQy91p+MxvDi89qm60km79cT\nf+oNm64nmXx9edaFl01Uz+T9+nCWTt+76XqGV//o+UeOYavq/+7nvn7D/z9bXX9y2zFsZf1Hznuz\n9Uz6/FnWf+T8Z1H/+mXMov71859V/WvLmFX9a/M/55yv23Q9ybD2V7ej/lnvr06zv/3JT/7thr9L\nkl2rq9Pd2ngjVfWyJO9vrV0+/vnvWmt337IFAAAAZOtPLbsqyWOSpKq+MclfbPH8AQAAtvzUsrck\n+Zaqumr881O2eP4AAABbe2oZAADAdtjqU8sAAABmTpABAAC6I8gAAADdEWQAAIDubPVdy9jhqmp3\na22lqu6fZG+Sw0ne11pzVwnojPUZgCFbmCBTVY9N8l1J/lWSv0/yxtbaH863qs2rqle11p4xfnyf\n1tqfzbumSVXVg5JcnOQ+SV6b5MNJ/n2SVyW5ZI6lTaWqzkpyVpJ/aK19tKr+bZLPtdb+ca6FHUNV\nvXDdj6tJbk5ydWvtPXMqaSrW5/mzPg9D7+t0VZ2Z5AVJfjzJPZNcmuRQkh9srbU5ljaRqnpmbr9N\net18q9q8qvrN1toTxo8f01r73/OuaRpVdXpr7WBVPSHJl2T04cpvtNY+P+fSJlJVJyV5SMbbpCR/\nmNH6cWNr7eNzLO2YqupX1/34he1Rkje01g7Po6aFCDJV9Ywkj07yiiT/lOTLk1xQVV/VWvuVuRa3\neV+z7vHLkjx8XoWcgBckecL48XJr7Xuq6q4Zfb9QNzs+VfUlSd6Y0RvW3yS5R1V9NskNSZ41x9I2\n4x8z2ris2ZfkZ6rqga21F8+ppolYnwfD+jwMva/Tv5zkfRmN4ZXjfx/OKCQ/ao51bVpV/Zckd0ny\nA/niNum5VXW31tqFcy1u885c9/gnknQXZKrq25P85yTnjP/79iT3TnLXJL8wx9ImUlV3SfK7Sa7L\naJv0uIzWh39I8j3zq2xTfiO33x59a0Z9eM48ClqIIJPkiUke3Fq7dfzzh6rq95L8fpJednwWwZ7W\n2ifGj1uStNaur6pb5ljTNH4+yeWttdevTaiqi5N85brxDVJr7ZeOnFZVFyX54yQ97PQk1uehsD4P\nwAKs03dtrb2iqu6c5OuSvL61tlpVd5p3YRN4VGvtG9f9/BdV9eQk707SS5BZBD+aL4bfA621n6mq\n05L8QToKMklenuSnW2vvWptQVb+Q5GuHfoS4tfaODSZfUVUf2PZixhblYv/PrdvpSZK01j6XpLc3\n3N6duvagtfa0eRZygr5+/U7P2Fdk9Ilcd1prK+lrXbA+D4P1eaA6W6dvGv/3IUneu+76qp6CzM1H\nThifRtNLD5LkpKo6uapOWff45Ko6ed6FTWBXa+2fx4/fkySttU8nuXF+JU3l7utDzNhSbnvUrDdz\nOa0sWZwjMke78LSnoPbAqvrU+PGZ6x6vttbuNq+iJvRXVfX41tpb1yZU1eOS/NUca5rGrg2mfVuS\nK7e7kK1QVfdMX+uC9XkYrM8D1dk6/amqekmSRya5sKr2ZXQKyofmW9aW6KUHyeh0uLVrknate7ya\nUbDvwR3XHrTWfnbd9N3bX8oJucORE1prT66qP5pHMSeqqh6e0XVvc7EoQeacqvrjDaZ/zQbTBqm1\ntuGnIlV1x42mD9RPJHlrVf1ARud+/ruMzl193FyrmtxyVd23tfYn66Z9Q5LleRW0WRusB6dkdA7r\nk7e/mqkt7PrcGevzACzAOv30JE9J8qLW2luq6psyul7pmfMtayLrP5hYr5tP0FtrZ827hi3w/qp6\nZmvtv69NqKqnZ3SaZU+uq6rHttZ+d23C+EOi6+ZY06ZU1fW57YeNh5N8IslT51NRsmt1tf+7aI7v\nRnOk1SSnttb+cpvL2RJV9RVJnpHkia21bk6BqKpdSe6X0U7PJzPawJzSWptbWp9UVX1lkt/O6Lzb\nT2T0adUjkjx2yHcTSTZcF25urf1jVe3tpQfjMazmi5+krz1eba397bzqmkRVPSpHObLUWnvnNpcz\ntXXr81lJ/i7JH/d26+We1+fkduvDakbr9D/NtSi6NL6N+tMzOjrzt0l+qbX2/vlWtXnj66pel+Qe\nST6e0XbpE0m+r7V20zFeOihV9aVJfiejG3l8PF881fWxrbUuPmAZkoUIMkcav3H9SPoMAY/JKMA8\nKMlLk1zaWvuHuRY2pXV9eFJr7UvnXc8kxhvMb8tog//JJG9NstJLGFjT47qwLgSs7bglXwwyXYSA\nqro0Rw8yT9neaqZzRB/WrCZ9hbFk4/W5tdbFefVHWR+S9NGHqtrohgprf0e9nNK0CCHgW5P87Pjf\nJ5J8dUZ3Jvy59UcGejC+69dZSf6+tfb/5lzOVMb7e/fPaByfbK11c1rZ0NaFRTm17MgQ8MCM7lRz\n77kWNYGqem5Gpwp8KKM7WpzUya01b+Moffj6uRY1hfFOzhuTL4SBn0vypCSDD2RHCcTdrAsZ3X7y\naJ+wDH7HLRmd75wkR57SVFUPm1dNU+i+D8ltgsD+8b9kdKpQF0Eg/ffhnuser2a0bbooo/e5Lhwl\nBFxcVT2FgJ9K8pjW2to60Krq6iRXZHQr4C4cuRNdVV0FynXul+Rp+eI4DvcwjiGuCwsRZBYkBDw3\nox3n17XWPlxVPz7vgia1IH1IctRAdvZci9qERejBIoSAqnpwRtf0nF9VaztsuzO6LuA/zK2wCSxC\nH8a6DgK992HtKPb4qNjLk9wrySNbax+da2GTWYgQsK7+tZ//afxe14Uh7kRPo/NxDG5dWIggkwUI\nARkdXnxCkleMN/h3qvE32M63rIksQh96DwPd92ARQkCSA0m+LMne8X+T0UWRPzG3iia0IH3oPggs\nQh+q6iEZfQfU65M8fV7fAH4ieg8BSfZU1cmttc+vTRjfevl2d9AasMHtRE+p63EMbV3o6daBx3JW\nkg9mFAI+kOTuVXX6fEuaTGvtUGvt11pr35zRFwK+Lck1VXXFnEubxFnpvA9jz83oyxdf2lr7gxz9\n09whOiv99+DIEPBlGd3lqJsQ0Fq7dnx70Ack+W9JfivJRa21nr5Nu/s+JKMgUFU/nOQNVXXe+N/T\nk7xq3rVtUtd9GIevN2V0PcZvJvmqqrpHVd1jvpVNZE8d8X0rHYaANyZ5bVWdmSRVdUaS146nd2Oj\nnehsfIv1Qet4HINbFxbuYv+q+uokP5Tku5J8sLX2nXMuaSJV9Q2ttQ+OH98hyeNaa2+ec1kT67kP\nVbU3o6NjP5jRl7bdKcmDOjs61nUPkqSq7pbRF52dleSvW2ufnW9Fk6uq70zyvIyOfl+e5HBrratv\nAu+9D1X1tUm+M6OjrL86nnw4yZ/2FCx77UNVvXv88HY7G621h29vNdOpqh9J8k1Jnt1a2z8OARdn\ntF19xXyr27xxoH9mRkH4M0leuf5WxkNXVe9N8h83OKp0ZWvtG+dX2WR6HscQ14WFCjI9h4B1pw/8\nWJKXZZTMT0ryzNZaF6cPrOm5D0fqNQwsQg8WJAT8UZJvTvL2JN+S5OrW2n3mW9VkFqEPSb9BYE3v\nfdjo1L7W2rvnWNJEeg8Bazbow0Nba++ZZ02bNcSd6Gn0Po6hrQsLcY3M+hBQVbcJAUl62XlbO33g\nlPR/Tn3PffiCdWHgp6rq+engiwAXrAc/ltHG/u1JXpzk6iTd7LiNrbTWDo3vkHVrVXW18zy2CH1I\nRqf5fSEIjO8S1NM4uuzDIlzjkySttV+uqj/rOAR034fW2quraiXJe6pqEDvR0+h9HENbFxYiyGQB\nQkBr7dok11bVr6TfTw2770PSfRhYiB6MLUIIeF9VvTHJv66qX0ryJ8d7wQAtQh+SToPAOr32wY0v\nhqH7PiS5fQNQAAADJklEQVTD24meVq/jGOK6sBBBZkFCwJpuPzVcoD50GwYWqAfJYoSAVyX59iQf\nSfKUjK696s0i9CHpNwis6bIPC7JN6j4ELEIfhrgTPY3OxzG4dWEhgsw63YaAdXr/1DDpvA+LsMFP\n5z0YW4QQ8OtJXpjRG9TzMrqddxcXOK+zCH1IOg0C6/Teh263SQvynrCm2z5kgDvRU+p2HENcFxbl\n9str1kLAP2cUAr5jvuVMZaWNv0CstXZrkh43lIvQh2S0wX93kv+V0Wlmz59vORNZhB78epL/m9EX\nka6FgN4cTvLeJKe11t44/rk3i9CHZBQErkryhiSPzGi97knvfViEbVLP7wlruu3DgtzWflHGMZh1\nYdGCzCKEgN4/NUwWow9Jxxv8LEYPFiEE3CHJzye5sqoenuTk4zx/iBahD0n/QaD3PizCNqnn94Q1\ni9CHwexEn6CexzGYdWHRgswihIDePzVMFqMPSd8b/EXowSKEgKck+euMxrGU5PvnW85UFqEPSf9B\noPc+LMI2qef3hDWL0IfB7ESfoJ7HMZh1YdGCzCKEgN4/NUwWow9J3xv8RehB9yGgtfbR1tqrW2uf\na61d1lr7+LxrmkL3fRjrPQj03odF2Cb1/J6wZhH6MJid6BPU8zgGsy4s2sX+i3Bh7dqnhs9rrb2x\nqn5o3gVNYRH6kPR9cW33PWitfTTJR8c/XjbPWnayBerDU5I8Islrkzw+nQWBBehD99uk9P2esGYR\n+jCYnegT1PM4BrMuLNoRmd5PHUj6/9QwWYw+JH0fHVuUHsCWWJCjYz1bhG1Sz+8JaxahD4twVCnp\nexyDWRcWLcgsQgjo/fSBZDH6kPS9wV+UHgCLYRG2ST2/J6xZhD4MZif6BPU8jsGsC4sWZLoPAQvy\nqWH3fRjreYO/KD0AFsMibJN6fk9Yswh9GMxO9AnqeRyDWRd2ra6uzmvZMGhVdY/c9pz6D3YaLAE4\nQd4ThqGqrkry/iSfzigI/Fxr7cHzrWpyPY9jSOuCIAMAQBeGtBN9IhZlHPMmyAAAAN1ZtGtkAACA\nHUCQAQAAuiPIAAAA3RFkAACA7vx/VTU0sDiviOIAAAAASUVORK5CYII=\n",
       "text": [
        "<matplotlib.figure.Figure at 0x123dddf90>"
       ]
      }
     ],
     "prompt_number": 222
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = small_doc_experiment(texts, labels_settings)\n",
      "score_df['settings']['small_doc'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.47 (+/- 0.04)\n"
       ]
      }
     ],
     "prompt_number": 165
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = big_doc_experiment(texts, labels_settings)\n",
      "score_df['settings']['big_doc'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.46 (+/- 0.03)\n"
       ]
      }
     ],
     "prompt_number": 166
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = ml_experiment(texts, labels_settings)\n",
      "score_df['settings']['ml'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.57 (+/- 0.03)\n"
       ]
      }
     ],
     "prompt_number": 167
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = np.zeros(11)\n",
      "for i, k in enumerate(range(1, 102, 10)):\n",
      "    scores[i] = ml_experiment(texts, labels_settings, num_neighbors=k).mean()\n",
      "sb.plt.plot(range(1, 102, 10), scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 202,
       "text": [
        "[<matplotlib.lines.Line2D at 0x116ecd290>]"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAECCAYAAAD5OrxGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl0XGed5vFvlVTaN8uSN0neZPu1HUfeSbBJ7CwOCSSQ\nQHJIhq0zBBiggaSHoUPowzn0gW7OMGlIgNB0oEnSwyS0ISEJIcYJ2Ryns9uS19eWbMeyLMvWvi9V\ndeePKsWKsFW2VHZV3ft8ztGx6t4q6fe6pOdevfe97+tzHAcREfEOf6ILEBGR80vBLyLiMQp+ERGP\nUfCLiHiMgl9ExGMU/CIiHpM+1k5jjB+4D6gCBoDbrLV1I/bfAXwOOBHd9AWgDvglsAAIA5+31tr4\nly4iIuMR64z/eiDDWrsGuBO4e9T+FcCnrbWXRT/2A1cBudbaDwD/CHw/3kWLiMj4xQr+tcAmAGvt\na8CqUftXAncZY7YYY+6MbusDCo0xPqAQGIxjvSIiMkFjdvUABUDniMchY4zfWhuOPn4Y+BnQBTxm\njPkwkQNFFrAXmAxcF9+SRURkImKd8XcC+SOfPyL0Ae6x1rZaa4eAp4DlwN8DW621BlgGPGiMyYhn\n0SIiMn6xzvi3Ejlj32iMuRioGd5hjCkEaowxi4Fe4HLgV8AlnPwroQ0IAGljfRPHcRyfzzeuBoiI\neNi4gtM31iRt0X764VE9ALcS6dfPs9beb4y5BbiDyIifZ6213zXGFAG/BkqIhP6PrbWPxKjDOXGi\nazz1p4TS0nzUvtTk5raB2pfqSkvz4x/855GCP4W5uX1ubhuofaluvMGvG7hERDxGwS8i4jEKfhER\nj1Hwi4h4jIJfRMRjFPwiIh6j4BcR8RgFv4iIxyj4RUQ8RsEvIuIxCn4REY9R8IuIeIyCX0TEYxT8\nIiIeo+AXEfEYBb+IiMco+EXE9RpbehgKhhJdRtJQ8IuIa7V19fOLJ3bx7ftf48lXDiW6nKQRa7F1\nEZGU4zgOW2oa+d0LdXT3DTF3RgGXVM1IdFlJQ8EvIq7S2NLDQ5sstr6d7Mx0PrlhAZctL8PvH9fy\ntK6k4BcRVwiGwvzp1Xf44yuHCIYcls8v4Ws3r8AZCia6tKSj4BeRlLf/SDsPbrIcbe6hKC+DT24w\nrDSllBRlc+JEV6LLSzoKfhFJWb39Q/zuhTpe2H4UH3DZijI+fmklOVmKtrHof0dEUo7jOLxlT/Cb\nZ/bR0TNIWWkun716IfPKChNdWkpQ8ItISmnp6Of/brZU17WQnubnY5fO5eqLZpKeptHpZ0rBLyIp\nIRx2+MtbR3j0pQMMDIVYNGsSn/mgYWpxTqJLSzkKfhFJeoebunjg6b0cOtZFblY6n7pqEWuWTMPn\n0xDN8VDwi0jSGhgK8fjLB9n8ej1hx+H9F0zlE1fMpyAnI9GlpTQFv4gkpZ0HWnjoz5bmjn5Ki7L4\nzAcXcsGc4kSX5QoKfhFJKp09gzzy3H5e3dWE3+fjmotn8pG1c8gMpCW6NNdQ8IskmaFgiJeqG2lq\n62VSfiaT8jMpzs9iUn4mRXmZBNLdOXrFcRxe3tHIfz5XS09/kDnT8/ns1QuZOTU/0aW5joJfJEkE\nQ2Fe3tHIk1sP0dY1cNrn5ecEIgeEvEwmFWRFDwyZFEX/nZSfSVZGav1qH2vt5aFNe9l7uJ3MjDRu\nuXI+V6wo1/w650hq/XSIuFA47PDa7iYef/kgx9v7yEj3c/VFM1lpSunsGaSta4C2rgFaOwdo7x6g\ntWuAY629HG7qPu3XzM5Mf/evhb8+MEQOFrlJcHdrMBTm6Vff4clX3iEYCrNsXgmfumoBxQVZiS7N\n1RL/zot4lOM4vL2vmT9sOUBDcw9pfh+Xryjj2jWzKcrLjPna3oHguweFkx/9tEY/b+8a4Ghzz2m/\nRiDdT0lRNoU5AYryM8nNCpCR7icQ/chITyMQ8BNI85MRSBuxPbIv/d3Ph1+TRnqa74yHWNYe6eCB\nTXs52txDYV4Gn7xyAStNqYZongcKfpHzzHEcdh1s5dGXDnDoWBc+H6y9cBofXTuHkqLsM/oaPp+P\n3KwAuVkBykvzTvu8gcEQbd0nDwptXZG/GNqj/3b0DNI4xsHhbPng5AEikEYgzU8gED04pPkJBNLI\nSPcTCjvU1LUAsH55GTeum0tOViBudcjYFPwi59G++nYefekA++rbAVi9cArXXzKH6ZNzz8n3y8xI\nY1pxDtNOc3draWk+jcc6aO8aoHcgyFAwzGAwzFAwzFAwNOLzMIPB0MnPh8IMhcIMDYXGfP5gMERP\n/9C720eaUZLLZ682zC8vOidtl9NT8IucB4eOdfLoSwfYeaAVgKWVk7nh0rlJMWIlPc1/xn9pTITj\nOARDkQNLMBgmPzcDv7p1EmLM4DfG+IH7gCpgALjNWls3Yv8dwOeAE9FNX7DW7jfGfAu4DggAP7XW\nPnguihdJdg3NPfxhywHespFfkYUzi/jYukpPziLp8/kIpKcRSNd4/ESLdcZ/PZBhrV1jjLkIuDu6\nbdgK4NPW2m3DG4wx64H3R1+TC3wzzjWLJL3j7X08vuUgr+4+huPA3BkFfOzSuSyerTtPJfFiBf9a\nYBOAtfY1Y8yqUftXAncZY6YBT1lrfwB8ENhhjPkDUAD8rzjXLJK02roGePKVQ2ypPkoo7FBemssN\nl85l2bwSjVaRpBEr+AuAzhGPQ8YYv7V2+CrNw8DPgC7gMWPMh4HJwEzgWmAu8ASwMK5ViySZrt5B\n/vTqOzz3dgNDwTBTJ2Vz/SVzWb1oivqxJenECv5OYOTVp5GhD3CPtbYTwBjzFLAcaAH2WmuDwD5j\nTL8xpsRa2zzWNyotTfxFrnNJ7UtdY7Wtp2+Ix16s5YmX6ugbCFFSlM0tVxmuWFVBWoosDOLm9w7c\n377xiBX8W4lcpN1ojLkYqBneYYwpBGqMMYuBXuBy4FdAGvB14F+MMTOAXCIHgzG5eUHk0tJ8tS9F\nna5tA4Mhnn2rnk2vHaanP0hBbgY3XDKXdcvKCKT7aW2N39j4c8nN7x14o33jESv4HwM2GGO2Rh/f\naoy5Bciz1t5vjLkTeJ7IiJ9nrbWbAIwxlxpjXgf8wJettc64qhNJMkPBMC9ub+CP//UOnT2D5Gal\n8/F1c7lyZQWZGRqtIqnB5zhJkcmO24/Kal9qGm5bKBzmlR3HeGLrQVo6B8jMSGPDqgqufl9FSt9x\n6ub3DjzRvnFdQNINXCIxdPYM8pNHa6hr6CQ9zc9Vqyv40PtnaRUoSVkKfpEx1Dd18b2H3qS5o5/V\nC6dw8xXzmZQ/9gRqIslOwS9yGnvfaeNnf9hJT98QH/3AHD6ydrbG4osrKPhFTmHrjkYeeHovPh/c\ndu0i1iyZnuiSROJGwS8yguM4PP7yQZ7YeoiczHT+4b9fxLRCde2Iuyj4RaKGgmF+/fQeXt3VRGlR\nFrfftJQL55W4elSIeJOCXwTo7hvip4/uYF99O5UzCvjqjVUatSOupeAXz2tq6+XHG2toau1l1cIp\n3PbhRWQEdDOWuJeCXzyt9kgH9/6+hu6+Ia65eCYfX1epSdXE9RT84lmv72nil3/cQzjs8NmrDeuW\nlSW6JJHzQsEvnuM4Dn969R1+/+IBsjLS+PKNF7JkzuRElyVy3ij4xVOCoTD/8WfLlppGigsyuf3G\npZRPyUt0WSLnlYJfPKO3P8h9f9jB7kNtzJqaz9durNL0C+JJCn7xhOaOPu7ZWENDcw/L5pXwxY9c\noGmUxbMU/OJ6Bxs7ufd3NXT0DHLlqnJuvnw+fr9G7oh3KfjF1d7ed4J/e2IXQ6Ew/+3K+Vy5qiLR\nJYkknIJfXMlxHJ55o57fPldLIODnqx+rYtn8kkSXJZIUFPwyYUPBMK/vaeLA0U5mTc1nfkUhkwuy\nEjaFcSgc5uFn9/Pc2w0U5mVw+41LmTVNC26LDFPwy7gdb+vlxe1HeWXXMTq6B9+zb1J+JvPLC5lf\nXsSCiiLKSnLPS796/2CQf318FzV1LZSX5nL7TUspLsg6599XJJUo+OWsBENhtu9v5oXtDew+1AZA\nfk6Aq1ZXUFU5mSPHu9l3pIP9R9p5fc9xXt9zHIDszHTmlRUyv7yQBRVFzJmeTyA9vqNq2roGuGdj\nNYePd7NkTjFfun4J2Zn6ERcZTb8Vckaa2/t4sfooL9c00tETObufX17I+mVlXP2BuXS09wKweHYx\nV70v0sfe1NbH/vp29h/pYN+RdnYcaGHHgRYA0tN8zJ5WEPmroKKIeWWF5GWPf9Hyw01d3PO7Gtq6\nBli3bAaf3LCA9DT/xBsu4kIKfjmtUDhMdW0LL2xvYNeBVhwgJzOdK1eWs27ZDMpKI3e8nmomS5/P\nx7TiHKYV53DJ0hkAdHQPsP9Ix7sHgrqjHdQ2dPD0a4cBKCvJffdAML/8zK8T1NS18PPHdzIwGOKm\nyyq5+n0ztUSiyBgU/PJXWjr6ean6KFtqjtIe7buvLCtg/bIyVi2cQuY4pywuzMtk1cIprFo4BYC+\ngSAHGjvf/aug7mgHDc09vLD9KHDyOsGCiiLml5/6OsHz2xr4zeZ9pKX5+PL1S9792iJyegp+ASAc\ndqg50MKL2xqoOdCC40B2ZhqXrShj/bIyKs7BfDbZmelcMLuYC2YXA5HrB/XHu9lf3z7mdYIFFZGL\nxtv2n+DPr9eTnxPgax+vorKsMO41iriRgt/j2roG2FJ9lJdqjtLaOQDAnOn5rFtWxkWLpp7XaQ3S\n0/zMmV7AnOkFZ3SdAGD65By+ftNSphRln7c6RVKdgt+Dwo7DroOtvLCtgeraFsKOQ2ZGGuuXzWDd\nsrKkGfMe6zqB4zh89JI55GaN/6KwiBcp+D2ko3uALTWNvFR9lOaOfgBmTs1j/bIyLlo8NSWGPo6+\nTiAiZy/5f9NlQsKOw55DbbywvYHt+5sJhR0yAn4uqZrO+uVlzJ6WrxEwIh6j4Hep7r4htlQf5cXt\nRzne3gdAeWke65fP4OLF08jJ0lsv4lX67XeZo809PPNmPa/sPMZQMEwg3c/aJdNYv7yMuTMKdHYv\nIgp+N3CiF2s3v1HPzoOtAJQUZnHlynLWVk3XxU8ReQ8FfwobGArxXzuP8cyb9TS2RKZMWFBRxIZV\nFSyfX6LFRkTklBT8Kaita4Dn3j7CC9sa6OkPkub38f4LprFhdTmzpxUkujwRSXIK/hRysLGTZ96o\n5429xwmFHfKyA1y7ZjaXLS/TouEicsYU/EkuFA6zbV8zm9+sp/ZIBxCZzGzD6gouXjz1lBOkiYiM\nRcGfpHr7g2ypOcqzbx6hpTNys1VV5WQ2rKpg8exJGp0jIuM2ZvAbY/zAfUAVMADcZq2tG7H/DuBz\nwInopi9aa/dF900B3gKuGN4msTW19fLsm0d4eUcjA4MhMtL9XLa8jCtXlTN9cm6iyxMRF4h1xn89\nkGGtXWOMuQi4O7pt2Arg09babSNfZIwJAL8AeuJZrFs5joM93M7mN+qprm3GITIl8XVrZnPp0hkT\nWqBERGS0WMG/FtgEYK19zRizatT+lcBdxphpwFPW2h9Et/8Q+DnwrXgW6zbDi5Q/80Y9h493A0Rm\nplxdwUpTqhWkROSciBX8BUDniMchY4zfWhuOPn4Y+BnQBTxmjPkwUAqcsNZuNsZ8C1Bn9CidPYO8\nsK2B57Y10NkziN/nY/XCKWxYXcE8zSkvIudYrODvBEbO0Tsy9AHusdZ2AhhjngKWAxsAxxhzJbAM\neNAY81FrbVMc605JR453s/nNel7d1UQwFCY7M52rL5rJFSvKmVyYlejyRMQjYgX/VuA6YKMx5mKg\nZniHMaYQqDHGLAZ6gcuBX1lrvzfiOc8TueAbM/RLS5NjDvhzZcvOY/z6j7sBmFGSy0cumcvlq2em\nxFTIZ8LN75+b2wZqnxfFSp3HgA3GmK3Rx7caY24B8qy19xtj7gSeJzLi51lr7abxFnLiRNd4X5rU\nHMdh89sN/PaZfUzKz+TTHzRUVU7G7/PR3dlHd6ILjIPS0nzXvn9ubhuofaluvAc1n+M4cS5lXBw3\nvjmO4/Db52rZ/EY9U4qy+cbNyyhx4RKBbv7lcnPbQO1LdaWl+eO6huqOfoYkFA47/Mdmy4vbj1Ix\nNZ/bb6zStAoikhQU/OdAKBzmV0/t4dVdTcycmsc/fXktg32DiS5LRAQADRSPs6FgmPse28mru5qo\nLCvgm7cspzBPZ/oikjx0xh9HA0MhfvroDnYdbGXRrEl89eMXkpWh/2IRSS5KpTjpGwhyz8Zq9h3p\noKpyMl+5YQmBdM2cKSLJR8EfB919Q/zoP7dzsLGLVQun8IXrFmu6BRFJWgr+CeroGeTuR7Zx5EQP\nay+cxq3XLNKShyKS1BT8E9Da2c8PH9lOU2svV6wo55YN8/FrnnwRSXIK/nE63tbLDx/eTktnP9dc\nPJMb11VqcRQRSQkK/nFoaO7h/zyyjY7uQW64dC7Xvn+WQl9EUoaC/yy9c6yLu3+7ne6+IW6+Yj5X\nra5IdEkiImdFwX8Wao908KON1fQPBPmbaxZy6dIZiS5JROSsKfjP0J5Drdz7+x0MBcN8/iOLuXjx\ntESXJCIyLgr+M7C9tpn7HtsJOHzlhiUsX1Ca6JJERMZNwR/D63uauP/J3aT5fXz140u5YE5xoksS\nEZkQBf8YXq5p5NdP7yEzkMbtNy1lQUVRoksSEZkwBf9p/OWtI/zmmX3kZqXzd59YxpzpBYkuSUQk\nLhT8p/CnV9/hdy/UUZCbwTc+sYzyKXmJLklEJG4U/CM4jsNjWw7yx1cOUVyQyTduXs604pxElyUi\nElcK/ijHcXjkL7U882Z0fdxbllFS6L71cUVEFPxE1sd96M97eam6kRkluXzj5mUUadUsEXEpzwd/\nMBRZH/e13U3MmprP331iKfk5GYkuS0TknPF08A8FQ/zr47vYtr+ZeWWF3H7TUnKyPP1fIiIe4NmU\nC4bC3Pt7rY8rIt7j2aTbvr+ZXQdbWTK3mK9+7EKtjysinuHZhWGr65oBuOGSuQp9EfEUTwZ/2HHY\nUddCQW4Gs6blJ7ocEZHzypPBf6ixi87eIaoqJ2uNXBHxHE8Gf3VtpJtnaWVJgisRETn/vBn8dc2k\n+X0snj0p0aWIiJx3ngv+tq4BDjd1s3BmEdmZnh3UJCIe5rngr4mO5qmap24eEfEmzwV/dW0LAEsr\nJye4EhGRxPBU8A8Ohdj9TivTJ+cwZZKmWxYRb/JU8O893M7gUFijeUTE0zwV/MP9+0vnqZtHRLzL\nM8HvOA7VtS1kZ6ZTWVaY6HJERBJmzPGMxhg/cB9QBQwAt1lr60bsvwP4HHAiuukLwCHg34FZQCbw\nPWvtk3Gv/Cw1NPfQ0tnP+xZNIT3NM8c7EZG/EisBrwcyrLVrgDuBu0ftXwF82lp7WfRjP/BJ4IS1\n9lLgauCn8S56PHS3rohIRKzgXwtsArDWvgasGrV/JXCXMWaLMebO6LaNwHdGfP1gnGqdkJq6Fnw+\nWDK3ONGliIgkVKzgLwA6RzwORbt/hj0MfBG4HPiAMebD1toea223MSafyEHg23GteBy6+4aobeig\nckahllUUEc+LFfydwMh5i/3W2vCIx/dYa1uttUPAU8ByAGNMBfAc8JC19pF4FjweOw604DgazSMi\nArFX4NoKXAdsNMZcDNQM7zDGFAI1xpjFQC+Rs/5fGWOmApuBL1trnz/TQkpLz928+PaIBWD96lnn\n9PuMJVHf93xxc/vc3DZQ+7zI5zjOaXcaY3ycHNUDcCuRfv08a+39xphbgDuIjPh51lr7XWPMPcBN\ngB3xpa6x1vaPUYdz4kTXBJpxeqFwmK/f8zLZmWn87y+twZeA+fdLS/M5V+1LBm5un5vbBmpfqist\nzR9XoI15xm+tdYAvjdq8b8T+h4n08498zdeBr4+nmHOh9kgHvQNBLlo8NSGhLyKSbFw/oL26Ljop\nm/r3RUQADwR/TV0LGel+Fs7UoisiIuDy4D/e3sfR5h4Wzy4mI5CW6HJERJKCq4O/Jnq3bpXm3hcR\neZerg3+4f1/BLyJykmuDv38wiD3cxswpeRQXZCW6HBGRpOHa4N99qI1gyNHauiIio7g2+E/Oxqlu\nHhGRkVwZ/GHHoaauhfycAHOmFyS6HBGRpOLK4D/c1EVHzyBVcyfj9+tuXRGRkVwZ/NW1w3frqn9f\nRGQ0lwZ/M2l+H4tna9EVEZHRXBf8Hd0DHDrWxYKKInKyYs06LSLiPa4L/prhSdk0mkdE5JRcF/wn\nZ+NU/76IyKm4KviHgmF2HWxl6qRsphbnJLocEZGk5Krg31ffzsBQSGf7IiJjcFXw625dEZHYXBP8\njuOwvbaZ7Mw05lcUJbocEZGk5Zrgb2zppbmjnwtmF5Oe5ppmiYjEnWsSskajeUREzohrgr+6thkf\ncOFc9e+LiIzFFcHf0z/E/iMdzJ1RQEFuRqLLERFJaq4I/p0HWgk7jpZYFBE5A64I/pq66DBO9e+L\niMSU8sEfDkcWXZmUn0nFlLxElyMikvRSPvjrjnbQ0x+kqnIyPp8WXRERiSXlg//kbJzq5hERORMp\nH/zVtc0E0v0smj0p0aWIiKSElA7+5o4+jpzoYdGsSWQG0hJdjohISkjp4B/u5tEwThGRM6fgFxHx\nmJQN/oHBELsPtVFemktJYXaiyxERSRkpG/x73mkjGArrpi0RkbOUssFfHb1bV908IiJnJyWD33Ei\nd+vmZqVTOaMw0eWIiKSUlAz++uPdtHUNUFU5Gb9fd+uKiJyN9LF2GmP8wH1AFTAA3GatrRux/w7g\nc8CJ6KYvALXAz0/3mnh4d21d9e+LiJy1WGf81wMZ1to1wJ3A3aP2rwA+ba29LPqxH7ghxmsmrLqu\nBb/Px5I5xfH+0iIirhcr+NcCmwCsta8Bq0btXwncZYzZYoy58wxfMyGdPYMcPNrJ/PJCcrIC8fzS\nIiKeECv4C4DOEY9D0e6fYQ8DXwQuBz5gjPnwGbxmQnYcaMFB3TwiIuM1Zh8/kQDPH/HYb60Nj3h8\nj7W2E8AY8xSw/Axec0qlpfmxngLA3vq9AKxfPfOMX5MMUqnW8XBz+9zcNlD7vChW8G8FrgM2GmMu\nBmqGdxhjCoEaY8xioJfIWf+vgJzTvWYsJ050xXxOMBTmbdtEaVEWmT7njF6TDEpL81Om1vFwc/vc\n3DZQ+1LdeA9qsYL/MWCDMWZr9PGtxphbgDxr7f3Rfv3niYzeedZau8kY4xv9mnFVdgr769vpGwix\ndsl0LboiIjJOYwa/tdYBvjRq874R+x8m0s8f6zVxUT286Ir690VExi2lbuCqrm0mMyONBRVFiS5F\nRCRlpUzwH2vtpamtjwtmFxNIT5myRUSSTsokaM3w3bqalE1EZEJSJvirteiKiEhcpETw9/YH2Vff\nzpzp+RTmZSa6HBGRlJYSwb/7UCuhsENVpUbziIhMVEoE/8nZONXNIyIyUUkf/OGwQ82BFgrzMpg5\nVbdei4hMVNIH/8HGTrp6h1haORm/7tYVEZmwpA/+k6N51L8vIhIPSR/8NbXNpKf5WDx7UqJLERFx\nhaQO/tbOfg4f72bhzElkZcSaT05ERM5EUgd/zQHdtCUiEm/JHfy10eDXbJwiInGTtME/OBRi96FW\nZpTkMqUoO9HliIi4RtIG/97DbQwGw5qUTUQkzpI2+DUpm4jIuZGUwe84DjW1zeRkpjOvvDDR5YiI\nuEpSBn/DiR5aOge4sHIyaf6kLFFEJGUlZapW12nRFRGRcyVJg78Fnw+WzFXwi4jEW9IFf1fvIHUN\nHcwrKyQvO5DockREXCfpgn/ngVYcB5bqpi0RkXMi6YJf/fsiIudWUgV/MBRm54FWJhdkMaMkN9Hl\niIi4UlIFf11DB70DQZbOm4xPi66IiJwTSRX81dFJ2dS/LyJy7iRX8Nc1kxHws3BmUaJLERFxraQJ\n/t7+II0tvVwwu5hAelqiyxERca2kWdYqOzON265dxLxyne2LiJxLSRP8Pp+PNUumJ7oMERHXS5qu\nHhEROT8U/CIiHqPgFxHxGAW/iIjHKPhFRDxmzFE9xhg/cB9QBQwAt1lr607xvH8DWqy134q+5pfA\nAiAMfN5aa+NeuYiIjEusM/7rgQxr7RrgTuDu0U8wxnwRWAI40U1XAbnW2g8A/wh8P37liojIRMUK\n/rXAJgBr7WvAqpE7jTFrgPcBvwCGZ1XrAwqNMT6gEBiMZ8EiIjIxsYK/AOgc8TgU7crBGDMd+A7w\nt5wMfYCtQBawl8gB4Sdxq1ZERCYs1p27nUD+iMd+a204+vmNQAnwJ2AakGOM2QuUAVuttd82xpQD\nzxljllhrdeYvIpIEYgX/VuA6YKMx5mKgZniHtfYnRM/mjTGfBYy19kFjzPc5+VdCGxAAYs265ist\nzY/xlNSm9qUuN7cN1D4vihX8jwEbjDFbo49vNcbcAuRZa+8/zWt+CPzaGLOFSOh/y1rbF59yRURk\nonyO48R+loiIuIZu4BIR8RgFv4iIxyj4RUQ8RsEvIuIxCV2B60znAkolxpgA8O/ALCAT+B6wB3iA\nyNxFO4GvWGtT9qq6MWYK8BZwBZE2PYB72vYtIkOYA8BPiQxpfgAXtO9U82gBIVK8fcaYi4AfWGsv\nM8bM4xTtMcZ8HvgCEAS+Z619KmEFn6VR7VsG3EvkfRsAPmOtPX627Uv0GX/MuYBS0CeBE9baS4Gr\ngZ8Raddd0W0+4KMJrG9Coge2XwA9RNryL7inbeuB90d/HtcDc3HRe8dfz6P1T6R4+4wx3wTuJ3KS\nBaf4eTTGTAO+CqwBPgj8szEmIxH1nq1TtO/HwN9aay8DHgX+3hgzlbNsX6KDf8y5gFLURiJTWUDk\n/3cIWGGtfSm67WngykQUFic/BH4ONEYfu6ltVwE7jDF/AJ4EngBWuqh9p5pHK9XbVwt8jJPTxpzq\n53E1kdkEhqy1ndHXVJ33SsdndPtuttYO30gbIPKevo+zbF+ig/+0cwGlKmttj7W22xiTT+Qg8A+8\n9/+5m8gvXcoxxvwNkb9mNkc3+XjvPE0p27aoUmAlkelI/gfw/3BX+0bPo3UvKd4+a+2jRLo3ho1s\nTxeR9hSIIvL0AAABn0lEQVQAHafYnvRGt89aewzenSDzK8CPGEf7Eh2yY80FlLKMMRXAc8BD1tqH\nifQ3DssH2hNS2MTdSuRO7ueBZcCDRMJyWCq3DaAZ2GytDVpr9wH9vPcXKNXb900iZ4aGyPv3EJGz\nxmGp3j547+9aAZH2jM6ZfCLTyaQkY8wniPzV/SFrbQvjaF+ig38r8CGA0XMBpapof9tm4JvW2gei\nm7cZY9ZFP78GeOlUr0121tp11tr10f7F7cBngE1uaFvUy0Suy2CMmQHkAH9xUftyee88Wum45Gdz\nhFO153XgEmNMpjGmEFhE5MJvyjHGfIrImf56a+2h6Oazbl9CR/VwirmAEllMnNxF5CzxO8aY4b7+\nrwP3Ri+47AZ+l6ji4swB/idwvxvaZq19yhhzqTHmdSInRV8GDuGS9nGKebSIjM5yQ/uGRyL91c9j\ndFTPvcAWIu/rXSk4W7AT7Qa/B3gHeNQYA/CCtfa7Z9s+zdUjIuIxie7qERGR80zBLyLiMQp+ERGP\nUfCLiHiMgl9ExGMU/CIiHqPgFxHxGAW/iIjH/H80ZMs4fhUcQwAAAABJRU5ErkJggg==\n",
       "text": [
        "<matplotlib.figure.Figure at 0x117b64d50>"
       ]
      }
     ],
     "prompt_number": 202
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Objects"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now, let's focus on the actions annotations:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "labels_objects = list(read_labels(\"data/labels_obj.txt\"))\n",
      "labels_objects, texts = zip(*match_labels_documents(documents, labels_objects))\n",
      "labels_objects, texts = np.array(labels_objects), np.array(texts)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 203
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The labels are distributed as follows:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "label_counts = pd.Series(Counter(flatten_labels(list(labels_objects))))\n",
      "label_counts.plot(kind='bar', figsize=(14, 6))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 223,
       "text": [
        "<matplotlib.axes._subplots.AxesSubplot at 0x1247bff50>"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAzIAAAGFCAYAAAAmZcbJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuYJXV95/H3MAOMYHOZ2JE1Gi8b/YYYcQQvERUwYtQk\nj2SNGY13N4IXRCSy0eBtE0FcFVC8B9Tx7gKiiRK8RgHJGpSIkVW/ivdViZ30DIwwI0NP7x9VLYem\nb9N9qur8ut6v55lnTp9zur6/6qpTpz71q1/VmunpaSRJkiSpJHt03QBJkiRJ2l0GGUmSJEnFMchI\nkiRJKo5BRpIkSVJxDDKSJEmSimOQkSRJklScdUt5U0T8OnAl8AhgF7C5/v9q4PjMnI6IY4HjgJuB\nUzPzokZaLEmSJKn3Fu2RiYg9gXcANwBrgDOBUzLziPrnYyLiIOAE4HDgUcDpEbFXY62WJEmS1GtL\nObXsdcDbgJ/VPx+amZfWjy8GjgYeAFyemTsz83rgGuCQYTdWkiRJkmCRIBMRzwAmMvPT9VNr6n8z\ntgH7A/sB183xvCRJkiQN3WJjZJ4JTEfE0cBG4D3A+MDr+wFbgeuBsYHnx4AtC0345punptetW7vb\nDZYkSZLUK2vmfHJ6enpJvx0RnweeQ3Wq2RmZeUlEvB34HHAp8BmqU8zWA18C7puZN803vYmJbUsr\nPI/x8TEmJratZBLWLqy+tbvR13nva+2u61u7G32d977W7rq+tbtR8ryPj4/NGWSWdNWyAdPAi4Bz\n6sH83wAuqK9adjZwGdXpaqcsFGIkSZIkaSWWHGQy8+EDPx41x+vnAucOoU1SyxbvHJyamlrC++Y8\nWCBJkqQG7G6PjLQqnXXeVUxs3bGs3x0/YD0nbdo45BZJkiRpIQYZCZjYuoNrJ7d33QxJkiQt0VLu\nIyNJkiRJI8UgI0mSJKk4BhlJkiRJxTHISJIkSSqOQUaSJElScQwykiRJkopjkJEkSZJUHIOMJEmS\npOIYZCRJkiQVxyAjSZIkqTgGGUmSJEnFMchIkiRJKo5BRpIkSVJxDDKSJEmSimOQkSRJklQcg4wk\nSZKk4hhkJEmSJBXHICNJkiSpOAYZSZIkScUxyEiSJEkqjkFGkiRJUnEMMpIkSZKKY5CRJEmSVByD\njCRJkqTiGGQkSZIkFccgI0mSJKk4BhlJkiRJxTHISJIkSSqOQUaSJElScQwykiRJkopjkJEkSZJU\nHIOMJEmSpOKsW+wNEbEWOAe4FzANPAfYC/gE8O36bW/NzPMj4ljgOOBm4NTMvKiRVkuSJEnqtUWD\nDPDHwK7MfGhEHAmcBnwcOCMzz5x5U0QcBJwAHAbcDvhiRHwmM29qoN2SJEmSemzRIJOZfx8Rn6h/\nvBuwlSqsREQcA3wHeCHwQODyzNwJ7IyIa4BDgK800XBJkiRJ/bWkMTKZORURm4E3Ah8ArgBOzswj\nge8BrwTGgOsGfm0bsP9QWytJkiRJwJrp6eklvzki7gj8C3B4Zv60fu5g4E1UIefRmXl8/fyFVONk\n/nWuad1889T0unVrV9h8aeWmpqZ4zms+x7WT25f1+wdtuB1vf8kjWLvW9VmSJKkBa+Z6cimD/Z8K\n3DkzTwe2A7uACyPihMz8MnA01eljVwCnRcTewHrgYODq+aa7ZcuNuz0Hg8bHx5iY2LaiaVi7rPrN\n1V56mJ/P5OQNzPMZWzGXubX7VN/a3ejrvPe1dtf1rd2Nkud9fHxszueXMtj/AmBzRFwC7AmcCPwI\neEtE7AR+BhyXmb+IiLOBy6hOWTvFgf6SJEmSmrCUwf7bgSfM8dJD53jvucC5Q2iXJEmSJM3LG2JK\nkiRJKo5BRpIkSVJxDDKSJEmSimOQkSRJklQcg4wkSZKk4hhkJEmSJBXHICNJkiSpOAYZSZIkScUx\nyEiSJEkqjkFGkiRJUnEMMpIkSZKKY5CRJEmSVByDjCRJkqTiGGQkSZIkFccgI0mSJKk4BhlJkiRJ\nxTHISJIkSSqOQUaSJElScQwykiRJkopjkJEkSZJUHIOMJEmSpOIYZCRJkiQVxyAjSZIkqTgGGUmS\nJEnFMchIkiRJKo5BRpIkSVJxDDKSJEmSimOQkSRJklQcg4wkSZKk4hhkJEmSJBXHICNJkiSpOAYZ\nSZIkScUxyEiSJEkqjkFGkiRJUnHWLfaGiFgLnAPcC5gGngP8EtgM7AKuBo7PzOmIOBY4DrgZODUz\nL2qo3ZIkSZJ6bCk9Mn8M7MrMhwIvA14NnAGckplHAGuAYyLiIOAE4HDgUcDpEbFXM82WJEmS1GeL\nBpnM/Hvg2fWPdwO2AIdl5qX1cxcDRwMPAC7PzJ2ZeT1wDXDI0FssSZIkqfeWNEYmM6ciYjPwRuAD\nVL0wM7YB+wP7AdfN8bwkSZIkDdWiY2RmZOYzIuKOwBXA+oGX9gO2AtcDYwPPj1H13szpwAP3Yd26\ntbvX2lnGx8cWf1ND+lq76/pN1J6amlrxNDZs2Je1a1e2Pi/EZW7tPtW3dv/qW7t/9a3dv/pN1F7K\nYP+nAnfOzNOB7cAU8JWIODIzLwEeA3yOKuCcFhF7UwWdg6kuBDCnLVtuXFHDx8fHmJjYtqJpWLus\n+s3Vnl7xFCYnb+DWHZXD4zK3dp/qW7sbfZ33vtbuur61u1HyvM8XgpbSI3MBsDkiLgH2BE4EvgWc\nUw/m/wZwQX3VsrOBy6hOWTslM29adoslSZIkaR6LBpnM3A48YY6XjprjvecC5668WZIkSZI0P2+I\nKUmSJKk4BhlJkiRJxTHISJIkSSqOQUaSJElScQwykiRJkopjkJEkSZJUHIOMJEmSpOIs5YaYkiSp\nWNOLvmNqamoJ71szlNZI0rAYZCRJWuXOOu8qJrbuWNbvjh+wnpM2bRxyiyRp5QwykiStchNbd3Dt\n5PaumyFJQ+UYGUmSJEnFMchIkiRJKo5BRpIkSVJxDDKSJEmSimOQkSRJklQcg4wkSZKk4hhkJEmS\nJBXH+8hIkiRJWqbpRd8xNTW1hPet2e3KBhlJkiRJy3bWeVcxsXXHsn53/ID1nLRp47J+1yAjSZIk\nadkmtu7g2sntrdd1jIwkSZKk4hhkJEmSJBXHICNJkiSpOAYZSZIkScUxyEiSJEkqjkFGkiRJUnEM\nMpIkSZKKY5CRJEmSVByDjCRJkqTiGGQkSZIkFccgI0mSJKk4BhlJkiRJxTHISJIkSSqOQUaSJElS\ncdYt9GJE7Am8C7grsDdwKvD/gE8A367f9tbMPD8ijgWOA24GTs3MixprtSRJkqReWzDIAE8GJjLz\nqRFxIPA14G+AMzLzzJk3RcRBwAnAYcDtgC9GxGcy86aG2i1JkiSpxxYLMucDF9SP9wB2UoWViIhj\ngO8ALwQeCFyemTuBnRFxDXAI8JVGWi1JkiSp1xYcI5OZN2TmLyJijCrUvBS4Ajg5M48Evge8EhgD\nrhv41W3A/s00WZJUpulF/01NTS3hfZIkLd4jQ0TcBbgQeEtmfjgi9s/MmdDyUeBNwKVUYWbGGLBl\noekeeOA+rFu3dnmtro2Pjy3+pob0tXbX9ZuoXe04rcyGDfuydu3K1ueFuMytvRrqT01N8fK3X87E\n1h3L+v3xA9bzquc8ZNV+1pqq7TbO2qNY39qrp36X25jFBvvfEfg08LzM/Hz99Ccj4gWZ+WXgaKrT\nx64ATouIvYH1wMHA1QtNe8uWG3e7sYPGx8eYmNi2omlYu6z6zdVe+RHeyckbgDUrb8ocXObWXj31\np5nYuoNrJ7cvewqr9bPWbG23cdYerfrW7kbJ+1HzBbDFemROoTpF7BUR8Yr6uRcCZ0XETuBnwHH1\n6WdnA5dRna52igP9JUmSJDVlwSCTmScCJ87x0kPneO+5wLlDapckSZIkzcsbYkqSJEkqjkFGkiRJ\nUnEMMpIkSZKKY5CRJEmSVJxF7yPTP4tfQu6WG7YtpJnLVEqSJEkyyMzprPOuWtEN207atHHILZIk\nSZI0yCAzh5XesE2SJElSsxwjI0mSJKk4BhlJkiRJxTHISJIkSSqOQUaSJElScQwykiRJkopjkJEk\nSZJUHIOMJEmSpOJ4HxlJkiStAtOLvmNqamoJ71szlNaoeQYZSZIkrQpnnXcVE1t3LOt3xw9Yz0mb\nNg65RWqSQUaSJEmrwsTWHVw7ub3rZqgljpGRJEmSVByDjCRJkqTiGGQkSZIkFccxMpIkSdKKLXw1\nNK+YNnwGGUmSJGkIlnvVNK+YtjwGGUmSJGkIvGpauwwykiRJUtGGcTPQ8k5rM8hIkiRJhevjaW0G\nGUmSJKlwfTytzcsvS5IkSSqOQUaSJElScQwykiRJkopjkJEkSZJUHIOMJEmSpOIYZCRJkiQVxyAj\nSZIkqTgGGUmSJEnFWfCGmBGxJ/Au4K7A3sCpwDeBzcAu4Grg+MycjohjgeOAm4FTM/OiBtstSZIk\nqccW65F5MjCRmUcAjwbeApwBnFI/twY4JiIOAk4ADgceBZweEXs112xJkiRJfbZgjwxwPnBB/XgP\nYCdwaGZeWj93MfAHwBRweWbuBHZGxDXAIcBXht9kSZIkSX23YJDJzBsAImKMKtS8DHj9wFu2AfsD\n+wHXzfG8JEmSJA3dYj0yRMRdgAuBt2TmhyLitQMv7wdsBa4HxgaeHwO2LDTdAw/ch3Xr1u5+iweM\nj48t/qbdNDU1teJpbNiwL2vXrmzeFtLEfJdS32XejdW2zK3dTX0/a93U9u9u7VGsvxq3MSut39fa\nK6m/2GD/OwKfBp6XmZ+vn/5qRByZmZcAjwE+B1wBnBYRewPrgYOpLgQwry1bbtztxg4aHx9jYmLb\niqYxt+kVT2Fy8gaq4UPD19x8j359l3k3Vucyt3Y39f2sdVPbv7u1R6v+6t3GrKx+X2svpf58wXex\nHplTqE4Re0VEvKJ+7kTg7How/zeAC+qrlp0NXEY1luaUzLxpt+ZAkiRJkpZosTEyJ1IFl9mOmuO9\n5wLnDqdZkiRJkjQ/b4gpSZIkqTgGGUmSJEnFMchIkiRJKo5BRpIkSVJxDDKSJEmSimOQkSRJklQc\ng4wkSZKk4ix2Q0xJjVv4jrhTU1OLvqepO25LkiSNKoOMNALOOu8qJrbu2O3fGz9gPSdt2thAiyRJ\nkkabQUYaARNbd3Dt5PaumyFJklQMx8hIkiRJKo5BRpIkSVJxDDKSJEmSimOQkSRJklQcg4wkSZKk\n4hhkJEmSJBXHICNJkiSpOAYZSZIkScUxyEiSJEkqjkFGkiRJUnEMMpIkSZKKY5CRJEmSVByDjCRJ\nkqTiGGQkSZIkFccgI0mSJKk4BhlJkiRJxVnXdQMkqX3Ti75jampqCe9bM5TWSJKk3WeQkdRLZ513\nFRNbdyzrd8cPWM9JmzYOuUWSJGl3GGQk9dLE1h1cO7m962ZIkqRlcoyMJEmSpOIYZCRJkiQVxyAj\nSZIkqTiOkZEk9YBXqpOk1WZJQSYiHgS8JjMfHhH3Az4OfKd++a2ZeX5EHAscB9wMnJqZFzXSYkmS\nlsEr1UnS6rJokImIvwKeAvyifuow4MzMPHPgPQcBJ9Sv3Q74YkR8JjNvGn6TJUnafV6pTpJWl6X0\nyFwDPA54X/3zYcC9IuIYql6ZFwIPBC7PzJ3Azoi4BjgE+MrwmyxJkiSp7xYd7J+ZF1KdLjbjX4CT\nM/NI4HvAK4Ex4LqB92wD9h9iOyVJkiTpV5Zz1bKPZuZXZx4D9wOupwozM8aALStsmyRJkiTNaTlX\nLftkRLwgM78MHE11+tgVwGkRsTewHjgYuHqhiRx44D6sW7d2GeVvMT4+tvibdlN11ZqV2bBhX9au\nXdm8LaSJ+S6l/mpc5iut3/T6Bi7zufg5331d/t37vMz7PO/WHt36q20bM4z6fa29kvq7E2Rmrkn5\nHOAtEbET+BlwXGb+IiLOBi6j6uU5ZbGB/lu23LjbjR00Pj7GxMS2FU1jbotfonMxk5M30NQlOpub\n79Gvv3qX+crqN7m+gct8Pn7Ol6PLv3ufl3mf593ao1h/dW5jVl6/r7WXUn++4LukIJOZPwAOrx9/\nDXjoHO85Fzh3KdOTJEmSpJVYzhgZSZIkSeqUQUaSJElScQwykiRJkoqznKuWSdIQLD44sLoSymLv\na+5CB5IkaXQZZCR15qzzrmJi645l/e74Aes5adPGIbdIkiSVwiAjqTMTW3dw7eT2rpshSZIK5BgZ\nSZIkScUxyEiSJEkqjkFGkiRJUnEMMpIkSZKKY5CRJEmSVByDjCRJkqTiGGQkSZIkFccgI0mSJKk4\nBhlJkiRJxTHISJIkSSqOQUaSJElScQwykiRJkopjkJEkSZJUHIOMJEmSpOIYZCRJkiQVxyAjSZIk\nqTgGGUmSJEnFMchIkiRJKo5BRpIkSVJxDDKSJEmSimOQkSRJklQcg4wkSZKk4qzrugGabXrBV6em\nphZ9D6wZWmskSZKkUWSQGUFnnXcVE1t37PbvjR+wnpM2bWygRZIkSdJoMciMoImtO7h2cnvXzZAk\nSZJGlmNkJEmSJBXHICNJkiSpOAYZSZIkScVZ0hiZiHgQ8JrMfHhE/BawGdgFXA0cn5nTEXEscBxw\nM3BqZl7UUJslSZIk9dyiPTIR8VfAOcDe9VNnAqdk5hFU1/k9JiIOAk4ADgceBZweEXs102RJkiRJ\nfbeUU8uuAR7HLTcnOTQzL60fXwwcDTwAuDwzd2bm9fXvHDLsxkqSJEkSLCHIZOaFVKeLzRi82+I2\nYH9gP+C6OZ6XJEmSpKFbzn1kdg083g/YClwPjA08PwZsWWgiBx64D+vWrV1G+VuMj48t/qbdNDU1\nteJpbNiwL2vXLm/eVlp/JbWXqom/e5e1XeaLW21/966X+VJ0+Tlrqr7LfGFNLfM+z7u1R7f+atvG\nDKN+X2uvpP5ygsxXI+LIzLwEeAzwOeAK4LSI2BtYDxxMdSGAeW3ZcuMySt9ifHyMiYltK5rG3KZX\nPIXJyRu4dcdVe/VXVntxzf3du6ztMl/I6vy7d73MF9bl56zZ+i7z+TS7zPs879Yexfqrcxuz8vp9\nrb2U+vMF390JMjOtfBFwTj2Y/xvABfVVy84GLqM6Xe2UzLxpN6YtSZKk4i2+U1sdwV/sfc0doNPq\nsaQgk5k/oLoiGZn5HeCoOd5zLnDuENsmSZKkwpx13lVMbN2xrN8dP2A9J23aOOQWabVazqllkiRJ\n0pwmtu7g2sntXTdDPbCUyy9LkiRJ0kgxyEiSJEkqjkFGkiRJUnEMMpIkSZKKY5CRJEmSVByDjCRJ\nkqTiGGQkSZIkFccgI0mSJKk4BhlJkiRJxTHISJIkSSqOQUaSJElScQwykiRJkopjkJEkSZJUHIOM\nJEmSpOIYZCRJkiQVxyAjSZIkqTgGGUmSJEnFMchIkiRJKs66rhsgSdLqN73gq1NTU4u+B9YMrTWS\ntBoYZCRJasFZ513FxNYdu/174wes56RNGxtokSSVzSAjSVILJrbu4NrJ7V03Q5JWDcfISJIkSSqO\nQUaSJElScTy1TJIkaei8wIPUNIOMJElSA7zAg9Qsg4wkSVIDvMCD1CzHyEiSJEkqjj0yktQri52T\n77n7kqQyGGQkqWeWe94+eO6+JGl0GGQkqWc8b1+StBo4RkaSJElScQwykiRJkopjkJEkSZJUnGWP\nkYmIfwWuq3/8HnA6sBnYBVwNHJ+Zi18eRyPEqxlJkiSpDMsKMhGxHiAzHz7w3D8Ap2TmpRHxNuAY\n4GNDaaVa49WMJEmSVILl9sjcF9gnIj5VT+OlwKGZeWn9+sXAH2CQKY5XM5IkDY89/ZKas9wgcwPw\nusx8Z0TcE/jkrNd/Aey/opZJkqTi2dMvqSnLDTLfBq4ByMzvRMR/AvcbeH0M2LrQBA48cB/WrVu7\nzPKV8fGxFf3+XKojQyuzYcO+rF27vHlbaf0ua6+0/lK4zIdbe6lW29+962W+FE38zaG/f/eul3lf\nt+1TU1Mr7unv62dtJfM9PT3Nrl27FnzP1NQUGzbsM+/re+yxB2vW7H5PWNfrW1e1h1G/r7VXUn+5\nQeaZwCHA8RFxJ6rg8umIODIzLwEeA3xuoQls2XLjMktXxsfHmJjYtqJpzG3l1yeYnLyB5XeDr6x+\nl7VXXn9hLvMmai9udf7du17mC2vubw79/bt3vcz7um3v+u++sFH+rK10mQ+nJ6y0Zd71+lbu53zU\ntzHzHXBYbpB5J/DuiJgZE/NM4D+BcyJiL+AbwAXLnLYkSZJWwDGv6oNlBZnMvBl46hwvHbWi1kiS\nJEnSEnhDTEmSJEnFMchIkiRJKo5BRpIkSVJxDDKSJEmSimOQkSRJklSc5V5+WdKqsPi136sbXS32\nvubuYyNJkjQXg4zUc8O5aZokSVK7DDJSz3nTNEmSVCLHyEiSJEkqjkFGkiRJUnEMMpIkSZKK4xgZ\nSZK0CnlVRmm1M8hIkqRVyasySqubQUaSJK1KXpVRWt0cIyNJkiSpOAYZSZIkScUxyEiSJEkqjkFG\nkiRJUnEMMpIkSZKKY5CRJEmSVByDjCRJkqTieB8ZjQjvwCxJkqSlM8hoZHgHZkmSJC2VQUYjwzsw\nS5IkaakMMpLUuoVPkfQ0SkmSFmeQkaQOLPdUSk+jlCSpYpCRpA54KqUkSSvj5ZclSZIkFccgI0mS\nJKk4BhlJkiRJxTHISJIkSSrOiA729y7vkiRJkuY3okHGu7xLkiRJmt/IBhkvTSpJkiRpPkMNMhGx\nB/BW4BDgl8CzMvO7w6whSZIkScMe7P8nwF6ZeTjwEuCMIU9fkiRJkoYeZB4CfBIgM/8FuP+Qpy9J\nkiRJQx8jsx9w/cDPUxGxR2bumv3Gww773TkncOWVVwPVgP1BH3790+Z8/xNPfu9tnhs/YD2HHXaf\nBae/UHuuv+EmpnZNzzv9hdpzwqnnLTr9xdozfsD63Zrfmfas3WMNn3rHXotOf6H2POXF75t3+ou1\nZ3CZ7c78Vu+/z63+7nNNf6H2zMz77s7vKKxvcMs6V9L6BtxmnXN9W3p7Bttfyvr2xJPfe5vPylzT\nX6g9/VzfqvfPXudKWN8Azj/z6bf5bpk9/YXa09f17corvz5n+0d9fYNq3l99XC44/YXa09f1Dar2\nz/5umZn+Qu1xfVt4ffvRj3445+trpqcXv9TxUkXEGcCXMvP8+ucfZ+ZdhlZAkiRJkhj+qWWXA38I\nEBG/B/zbkKcvSZIkSUM/teyjwCMj4vL652cOefqSJEmSNNxTyyRJkiSpDcM+tUySJEmSGmeQkSRJ\nklQcg4wkSZKk4hhkJEmSJBVn2Fcta1xE7A2Qmb/sui1diYj1mbmj63ZITWtrXY+Ipw/8OA2smXmc\nmXPf1au5thwATGXmtjbr9k1E/Gb9cA3VMv+VzPxR+y3SahYRh2Smt6QYERHxG5n5k67boZUb+SAT\nERuBVwH/Dvxv4MP18ye1tYMREfcFHg/cAfgxcH5mfqeFuudl5qb68cmZ+fr6pYuBhzdc+//M89J0\nZh7ecO13c+udycHa/73J2nX9PYHHApOZ+fn6uYOAs2eWR4O13we8IzO/2GSdpYqIOwDPyszXNFyn\ns3W9djC3rHN/DnywhZoARMShwLuABwJ/DLwd2Fr/Hf6h4drnNb1O766IeG1m/lULpX5Q//v3OV57\ncNPFI+JBwHOBuwI/BN6emV9qoe7353lpOjPv0XDtzratda0u1/ez6/B8CdV27dOZubWt4hFxMvBs\nYB/gJuCtwOszs9FL10bEBuAVwIuotrObgR3AX2Tm3Ldyb7Y9vw8cDzwUuGML9Y7OzM9GxOuo9iF3\nAX+dmT9vuO4obtvvk5lfH/Z0Rz7IAG+j+hBsAD4G3A/4OfApoPEgExF/BrwYeAfwZeBuwEci4hWZ\n+bGGy48PPP4j4PXzvbEBfz7Hc3OFiyYcRrWx/QDwz/Vztzlq2qAPADuB/xIR96ba2TkXOLuF2hcC\nL46ItwLvBN7T5pfdjIh4IPB84A+Aj7RQsst1ncx8yczjiHhQZv51i+VfDzw9M2+KiNOAxwDfAT4J\nNBpkuPXffVS0EVyhOjj1RGBv4ALgwsy8oY3CEfFo4H/W/74P3JNqR/dvM/MTDZc/eODxp6g+421s\n16HbbSt0uL5n5lERsZ4qJB8JHBcRa4BLMvNvm6wdEScB9wIOzcxtEbEf8AbgZOB1Tdam2nf6ItX3\n95vqf1+nWuaParg2ABFxe+DpVAcODgJeADy5hbovA+4DfBZ4GPBK4Ajg5cAJDZfvbF2vt2+vAyap\n7if5Y+A0qv3Kuwy7XglB5peZ+RmAiDgxM79dP27rtIsXAkcOfsFFxGaqHYymg0xnMvMHgz/XG76Z\nDcHvNFz7kIi4D/AUqhB5GfC+zLymyboD7pGZ94+IvYArqY5ePTwzv9l04cz8KPDR+ijl04B/ioj/\nC/xdZl7WZO36tM0nUh2t+iWwP3D3zNzeZF2xR2Z+LSJ+A9gnM68EiIhdLdS+R0S8mrl7P09poX5n\nMvNC4ML6VL7HAx+OiEngQ5n5yYbLvxj4w8ycnGlORFxBFagaDTKDp2pGxK6WT9PubNs6U7/L9T0z\nd0TElcCBwH7AoVQHZ5v2eOCIzJyq23F9RDyb6ru16SBzUGa+sd6HuA/w3sycjoh9G64LQES8Gfh9\nqhu2/zeq3r+2etwfCTyifrw9Mz8VEZ8Frmihdpfr+muBP6U68P8a4NeBHwGHNFGshCAzeBR+cIO7\ntqX6O2cfpas3Aje3VL9T9VGz44FNVL0FT1/4N4aj7n58cd2GI4DXRMSdM/P3Wih/fd2GmyJiD+CR\nAzscrcjMa4HXRsQbqXokP0t15LhJ3wc+BDwlM78dERcbYlqxs/7/UVTLeeYUnNu3UPtGoPXTOwAi\nIrhtL+saYH2b7ah7PM+tDxi8iOrUl4NaqDs56+ef10foV7Out61dru8nA38IHED1Of848OLM3Lng\nLw7HTTMhZkZm7oyINmrfWP9/BHDZwKlsrQQZqlPIvgJ8CfheSzV/JTNn9hXfWP88FRHXtVC6s3Ud\n2FJ3Onw7It4JvCoz/66pYiUEmXtHxAepvuB+JyI+VD/faK/AgPlOZ2ojSHU27xHxeKoAsxfwbiAy\n87im687EFgqXAAAMnElEQVRqw37A46h6CfYF3t9S6cGdiZ+3HWIAIuJhwFOpuqM/Bty7hbJvoOoF\nu1u98Wnzqoadfs4H6jGr/nRmPqnh8p+LiMuB3wQeGxH3AN4CnNdwXYBrM/M9LdSZyzuYe/v6H201\noB7/+OdUO5hfBc6h2t40bV1E7JWZNw20ZS9gz6YLDwTINcA+EXGv+vH0zBkPDep629rl+v5yqtNF\nT6c6neymRd4/TLsi4o6Z+avxYBFxR2Bqgd8Zlp9FxOlUpzCeGhFjVGe6fK2F2mTmxoh4CPAs4Exg\nj4g4uKVewD1nPuczQxHqz3kb+49druuDZxP8qMkQA2UEmU3cstF9R/3c9MDjpt171k7OjDZ2sLqc\n9/dSHUE4IzP/IyL+tIWaAETEE6h2Jn6TanzGczNzvgGqTZhvp7rxndqI+BvgSVRjJM4Bjm/piB2Z\n+VqqXqCjqDb694+I/0V1Wt/VDZcfHJQ4uIPb1uf8HbMetzUei8x8TUT8A3BdZv4kIv4r1amEH22h\n/JUt1JhTZh41+HNErBs4etm4iPgG1XL+ENVpnDO9j/cAmt6h/xDwzvp06cmIOJBqzMBc3zXDNrh+\nb+fW637T45M627bWOlvfqcYsPIxqDNxpEXEt8I/AP7ZwlbxTgX+sx+B9F7g78DKgjdNHn0s1TuK0\nzPxoRDyYatD781uoDUBmXg5cXh8cfTLw/oiYzsz7N1z6g8C7IuIFA5/zN9DOxWS6XNcHD5Csa/pg\nyZrp6da+r4diYKzG8zLz4MXeP4R6R3HbQe7TwJrM/ELT9We1pbV5j4g7UW18nkI1MO/uwAObvsJJ\nXXsX8C1ue8SmlS+7Lpd5HWTeOfPFVh+9egZVmGurF3KmLQdSLf+/yMyNLdS7A3BjZt4YEU+mOjr9\ngbaCXJf1+1g7Iu5M1ev0R5m5JSKeRDUI90+zhcuiRsQX6oe32aZlZuMXHKjHKDyfaqfueuBNmfnm\nput2aY5t68zj6cy8pIX6c53OCEALvVGz2/Jo4KXA4ZnZ+BH6iPhd4DlU3+U/Ad6WmV9tuu6oioj7\ntTH/EfFc4HlUF6zaBrwlM9/UdN269n3r8Zd7AcdRXS3uXZnZ6PjLets63+ds6NvWYoLMHGM1zsnM\nL3fQjlaDVF2zs3mvz9l+BHAs8CDggsw8ueGaR9UPb/OF13Z4rNvTm2U+z07tB5s+DSIi/gfVpUF3\nUl2p7jepL4ubmU9psnbX9Xtc+yKq9fpjA8/9GfDUzHxsk7XnaU9rPUJt9z7Nqv35gR9n3zPp9xuu\nPXha2+zajQeJtnewZtV+AFWPzMOA36Y6UPcZ4LOZ+cMma3epPjA5yS1jAWdMZ+adWqj/+TmengZo\nen2fT0QcVI+DbbLGX1Kd2fIQ4AyqbfsPATLzxIZrH5qZ/9pkjUEjf2rZKIzVqNsxe8fyaS3U7Hze\n6x6YzwKfrXdyn9pCzS8M/jwYJLj1pUMb1bdlPrhTGxGDO7VHU/XMNGkT1bK9PfBN4C71YNRLG647\nCvVnau9L1RPZl9q3z1mXsM/M8yOi0S/ZGbN7hIBNEdFWj9BnqE/jiohnZua7G6436LncEiLeQ7Vd\na+siAwudttl4L9js0xlbdjrVcn8VcFXTR8UHDYSJm5h1pkELYWLmIgfXUPX0Nnr1zTk8d+DxNLCR\n6rT5xk/viojDqC6xPgmcnJkTEfE8qtP6mv67bwIOp5rnJwH3rHu+57tH4DCdQXuX0R/9IEOHYzWg\n8zDR5TiVl2fmq+YaHxQR7wE+npkXNNyG1oNEXbeXy5xud2pvqE9l2hIR3xo4ramto9Zd1p+pvbVn\ntefbeW5zp/q1dYghMz8Y1VWc3kZ108a2PI1qO9OKzPzWzOOIuDGzvZsSdhwkFvxeo9rJb/J77fj6\n/2ngt6rOqUoLvVGdhYnMPBM4MyIOBp5Unz79z8D7B9fFBut/CyCqq+S9mOqg6BPaOJWRapzrS6gu\nQ/zqqC45fSeqK7g1bVtm3hzVDZe/O7Odo73ta2tKCDK/RTVW47KI+Dpwh4hY08ZYjVqXO5ZdzvvM\njfjmOoK2F9V1whvZ4I9AT1Rfl3mnO7X1ebxrZj1u6zLrXdbva+0r6kGwv7oZYkScAPxbC7Wh4x6h\nPuo4SECH32vA2xd4rdGj112HiboN3wReXveEnkl1al3TtxQAIKrB5puBq4H7Z+Yv2qgL/CIzP123\n4eVU+xZPbun7fFc938+kutQ3EXFPbnuKXxMeEhE/m+P5RnoARz7IZOZPqa7w8WpuGavx/YhofKxG\nrbMdyy7nPTNnBtpfDfw1EFQbntdkdWfgZzVYvtNeOHq6zOl2p/au3HLN+zUDj9vSZf2+1n4Z8IaI\n+ClwLdX9NT4F/GVL9bvsERq8qk+rl0CuP9fU9fYY+Jmmx8LRbZDo9HutjQtILKENnYSJiPg14M+o\nev3XAB/m1qd8NVn7BVSXe/5L4GJgemadb2F9H7y89U8z86UN1xv0cuB9VKeHnxIRR9Y/P6GF2pe3\nub4XM9h/0MxYjcw8q8WarQ96n6cdrc57RHwCuIjqZlJHAEdm5uMartnZFdNmtaNXyzwifsAtOxdr\nBh6TmXdvsrb6K6qbf/4a8B9tDoCPiNdT3eNgdo/Q72RmoztZswadz/6sNT3o/AcL1G7lc15v0+YK\nEodlZuOXje3oe63T3qh5wsQFmfmfTdWs615MdTrV+XXNH9cvTbcQJGbWd5j76oSNru8RcQnwSKq/\n96frxzO127yPEBGxN7Ar27kS5ucNMgO6/vDP06a2diw7n/eI+KfBK3u0uYKOSpCo29KbZd6Frue7\ny/rW7myZr6e6p8NjmdUjlJnbF/pdrUwXQWJW/da/1+KWS+EexTy9UZl5v4ZqdxYmugwSXevyoEHH\n2/ajM/OzTUx7LiN/ahkdd0UvtDJE84PeuxynsoHqg/eTiHgk8AXgwTR/o7hfyQ6umAa9XuZd7lh2\n+jnvuL61uznNaAfwnLoXptUeoT4HyNo+mfm2+vFXI+JPGq4HdPu91vHp2jNX+/yL+t+gRneoM/Nu\nTU5/MV2u7x3Pe5fb1wdT7be18jcf+SDT8YcfOlwZOp73jww8PpPqbtc/AH7eYE2g8yAB/V3mfZ3v\nrs+dt3YHy3xmO0M1Jm/muZmHqzm49/kAWWffawM2U/VGvZ+qN+o9wOOaPKWuyx3qEQjOvTw42Kd9\niZEPMgM20/KHHzpfGWZspv0N38w9Dp4AnErVM/K7wLuaqjmg6yO1fV3mvZzvEapv7XZr9zK49/UA\nGXT+vTaj9d6oPve292mHfh6bWeX7EiUFmU66ogdsprsv+i7n/STgfpn5i4gYAz5PdeWLxozIDjX0\nd5lvpp/z3XV9a7dYe0S2M5vpUYAckSABHXyvddwb1cvQPstmVvkO/TxW/b7EyAeZEeiKntHFUZRR\nmPeprK+5Xn/w2hwEu5luj873dZn3br67rG/tzrftm+lncO/VAbJZuvhe66w3akR2qDfTs+/zAZtp\ned5HZPvayt985IMMHXdFd7wyjML5vN+PiDOAy4CHAd9tsXbvBoTS7dikXs73CNS3dnfbN+hZcB+R\nHZwuD5BBB99rI9IbtZmehfYRWd+7mPfe7EuMfJAZgQ9/l0dRup53qO7n8mzgaOCbwEuaLjgCG56+\nLvO+znen9a3dzTLvcXAfhQDZ5QEy6OB7bUCXvVG9Cu213uzQD+rTvsTIB5kBnXz4u/6yrXW24cvq\n5klvbqPWAAeEdjM2qZfzPUL1rd1u7V4G9xH5nHcZJLr6XpvRem9Uj0N71+v7KBw0WPX7Ens0MdGG\n3OrDD7TdFT2zMhwDbARObLF21/Peqsx8eP1BeDuwnupDsJ7uBoT2bZn3db67rm/tFmuPyHamy89a\nZ7Uzc2dmvjkzn5+Zb8kW7jY+Qr4fEWdExJ/UvVJt9EZ9hGow/52pdqi3Am8D7tV04RH5nEEH6/uI\nzPuq35coqUem667oLs/p7Xreu9L10fm+LvO+znfX9a3dzTLvcjvT5Wet63EqfdV6b9SI9ML1+fu8\ny3lf9fsSJQWZTrui6XZl6Hreu9L1F21fl3lf57vr+tbuZpn3Nbh3HSB7qePT2voa2qEHO/TzWPX7\nEmump2dfUlxziYg9qVaG36ZaGf6uZ93hrYuI9wP/zi0fgl/LzGe0WL+Xy7yv861+6nI70+Vnzc95\n/0TE5Zn5kIGfL8vMh7VUu7ff513Pe1fa+psbZDSy/KKV1DS3M+qLvob2rvV53ttgkJEkSVrl3KHW\namSQkSRJklScki6/LEmSJEmAQUaSJElSgQwykiRJkopjkJEkSZJUHIOMJEmSpOL8f5XUUvSKFy+o\nAAAAAElFTkSuQmCC\n",
       "text": [
        "<matplotlib.figure.Figure at 0x124726610>"
       ]
      }
     ],
     "prompt_number": 223
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Let's see how well the different models perform."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = small_doc_experiment(texts, labels_objects)\n",
      "score_df['objects']['small_doc'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.48 (+/- 0.02)\n"
       ]
      }
     ],
     "prompt_number": 169
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = big_doc_experiment(texts, labels_objects)\n",
      "score_df['objects']['big_doc'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.44 (+/- 0.01)\n"
       ]
      }
     ],
     "prompt_number": 170
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = ml_experiment(texts, labels_objects)\n",
      "score_df['objects']['ml'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.66 (+/- 0.02)\n"
       ]
      }
     ],
     "prompt_number": 171
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = np.zeros(11)\n",
      "for i, k in enumerate(range(1, 102, 10)):\n",
      "    scores[i] = ml_experiment(texts, labels_objects, num_neighbors=k).mean()\n",
      "sb.plt.plot(range(1, 102, 10), scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 204,
       "text": [
        "[<matplotlib.lines.Line2D at 0x11df9ee50>]"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAECCAYAAAD5OrxGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XuQXOV55/Fv99zvkmZ6pBHIunB5MLZlQGLBAjnIgJPY\nIcZZZxPKawoV2MT2JhjvroNJylWbdW7rIjGOjR0LX8hWTCq4jGObWEsgcgVPdnHAMQIDj9BlBEYz\n0szo0nO/9dk/zplRa5Cm597dp3+fqqnpc86cmfeRZn7n7fdc3kQQBIiISOlI5rsBIiKyvBT8IiIl\nRsEvIlJiFPwiIiVGwS8iUmIU/CIiJaZ8po1mlgQeADYDI8Ad7n4ga/uVwH1AAngduBUYBx4ELgYy\nwIfd3Zek9SIiMme5evw3A5Xuvg24hzDkATCzBPBV4DZ33w48CWwE3g3Uufu1wB8Bf7wUDRcRkfnJ\nFfzXALsB3P1pYGvWtouBXuCTZvYjYEXUsx8CmqIDQxMwutiNFhGR+ZtxqAdoBNJZyxNmlnT3DNAC\nbAM+DhwAfmBmzwBPAdXAy0AzcNOit1pEROYtV48/DTRkf30U+hD29vd7aJzwncFW4FNAu7sbcBnw\nkJlVLnK7RURknnIFfzvwHgAzuxrYm7XtIFBvZhdEy9uBnwN1nH6XcAKoAMpm+iFB+MAgfehDH/rQ\nx9w+5iUx00PaonH6yat6AHYCW4B6d99lZjuAPyO8qqfd3e82sxXANwiHgiqAz7v73+VoR9Dd3Tff\nGgpeKtWA6itOca4NVF+xS6UaEvPZb8bgX0YK/iIW5/riXBuovmI33+DXDVwiIiVGwS8iUmIU/CIi\nJUbBLyJSYhT8IiIlRsEvIlJiFPwiIiVGwS8iUmIU/CIiJUbBLyJSYhT8IiIlRsEvIlJiFPwiIiVG\nwS8iUmIU/CIiJUbBLyJSYhT8IiIlpnymjWaW5PTUiyPAHe5+IGv7lcB9hFMvvg7c6u6jZvZp4CbC\nqRe/6O4PLVH7RURkjnL1+G8GKt19G3APYcgDU/PxfhW4zd23A08CG83sOuAd0T7XAZuWoN0iIjJP\nM/b4gWuA3QDu/rSZbc3adjHQC3zSzN4KPObubma3Ac+b2XeBRuC/L36zReJnfCLD8fQw3SeH6T41\nBAGkVtSQWllDc2MVZUmNzMriyBX8jUA6a3nCzJLungFagG3Ax4EDwA/M7BmgGXgT8GuEvf3vAZcs\ndsNFik0QBPQNjtF9cojuU0NhwJ8coudk+Pp43zBBcPZ9k4kEzU1VtK6oIbWyltSK6vB19FFTletP\nWeS0XL8taaAha3ky9CHs7e93dwcws93A1mj9y+4+Duwzs2Eza3H3npl+UCrVMNPmoqf6itdcahse\nHefo8UGO9g7SdXyAo72DHD0+SFfvAEePDzI8OnHW/VY1VnPJ+lWsaa5lTXMdq1fVkkgk6OodiD4G\n6ewd4OcdJ6DjxBv2b6qvZE1zHW3NdaxurqWtuS5cbqljZUMViURiUeorRnGvbz5yBX874UnaR8zs\namBv1raDQL2ZXRCd8N0OPEh43uAu4C/MbC1QR3gwmFF3d988ml8cUqkG1VekpteWyQSc6Bs5o9fe\nk/U6PTB61u9TXVlGakUNLU3VU7301IrwdXNjNZUVZWfd723rV5yxPDw6Ts/JYY6dHOLYiaGwHSeH\nOHZyiP2vncQPv/GgUFmenPqZrStrznh9yQUpTp4YWMC/UGGL8+8mzP+glgjO9d6SqRO4k1f1AOwE\ntgD17r7LzHYAf0Z4VU+7u98d7ffnwA7Cg8Cn3f2fcrQjiPt/juorPsOj45wcmuDZFzs5eCTN6z0D\n9J4aZiLzxr+ZsmSCVY1VUbifDvXJj7rq8hl73YthIpPhRHokPCicHKL7xOmDQvfJIYZG3vhuI5GA\nVQ1hu2urKwiCYGq4KRO9yEyuCwImS5/8uiAIyABEr4NoWyZ447roW0ztW5ZMUF6WpLw8QUVZkvLy\nZPi5LElFefR58nV5Ymo5++vCfcuomNyetV95eZLVrQ2kTw1RWZ6M5XBYKtUwr1+qGYN/GSn4i1gc\n6stkAo70DHCwM83BI6emgj77z6O+pmKqxzzVc48+ryzwk69BENA/NEb3yWGOnRyMDgrDnBgY5Uh3\nPyf6Rub9vRMJSJAIPycgkUiQIPqcmL49MbVPJhMwNpFhfDyYOsgspVtuuIgbt65b8p+znOYb/PE7\nBIrMwsn+EQ4eSUcfpzjU1cdI1vh7ZXmSi85r4i0XpmhbUc2mtY2saqzOY4sXJpFI0FBbSUNtJZvW\nNk6tnzxoj45NMDqeiUL6dEAnoxXJBBCF9+S67K9bqIlMeAAYm8gwPpFhfDzD2ESGsfEM4xMB49Hr\nsWjb+NS2DGNZ26d/Lqsoo39glCATsHFNY+6GlAgFvyzY4PAYhzrTnOofpb6mgsb6SppqK6mqPPu4\n9XIbGZvgcFffVMgf7ExzPH1mD7etuZZNaxu5YG0Tm9Y2cl6qjrJkMhbvZmajsqLsnOcZlkNZMklZ\nJVSxuG0olf+/uVLwy6wEQUB6cIzOngE6ewfojK4yOdI7eM5hgqqKMprqKmmMPrJfN9ZW0lQfrV/E\ng0QmCDh6fJADr6enhm1+cWzgjKGEhtoKLruwhY1rG9m0tpGNaxqprdafgpQO/bbLGTKZgJ5TQ1Gw\nD3KkNwz6rt5BBobH3/D1KxuquOyiFM2NVaxsqGJgaIz0wCinBkbDz4OjHDySzjmGW1VZRlPtWQ4Q\n05anHyTS0fc/eCTNoSOnONjZx9DI6XaWlyXZuLaBTW1hT37T2kZamqqX/ESrSCFT8JeosfEJuo4P\nhb32ngG6jg9ypGeQoycGGRvPnPG1yUSC1pU1XLxuBW3NdbQ117K2pY41q2qpqSrP+XY6E51YTE8e\nDKLPb3g9x4NEJgjoOTV8xrbVK2u47MJmNkVDNuta6ykvK9yTriL5oOCPuYHhMTp7BqeGZyZ78D0n\nh5ker5UVSdY219HWEt4A1LaqlraWOlavrFlQeCYTCRprw+EdUjN/bfZBIvugcLaDRCYT8NZNq6bG\n5Te2NVJfUzHvdoqUCgV/zIyNZ3ix4zjP+DFeOHicU2e5oaihtoKL1q2gLbrDc230eWVjVXjFRh5l\nHyTOz3GQEJH5UfDHwNj4BC8cDMP+Z/t7pm7Uaaqr5G2bmqOAj0K+pU69YpESp+AvUiNjE7xwsJdn\nvJuf7e+Zuga9ubGK7ZvXsvWSVjatbcx7D15ECo+Cv4iMjE6w92Avz7x8jL0HehkZC8O+pamaHZef\nx1ZrZWNbg65YEZEZKfgL3NDIOM8d6OHZl7t5/mAvo9EVN60ra7jykla2WitvWl2vsBeRWVPwF6DB\n4XGe29/DM36M5w8eZ3wiDPs1q2rZekkrWy3FulaFvYjMj4K/QAwMj/GzV3r4t5eP8WLHccYnwost\nz2upmwr7tS11CnsRWTAFfx71D43x033dPOPHeKnjxNTjfte11rPVUmyxVta21OW5lSISNwr+ZZYe\nGOWnr3Tz7MvHeOnwyam7VNevbmDrJSm2WiurV9XmuZUiEmcK/mVypGeAbz2xj5cOn5h6xvvGtka2\nXhL27FtX1OS3gSJSMhT8y2Dv/m7+5H8/y+DIOBec18hWa2WLpWhpUtiLyPKbMfjNLMnpqRdHgDui\n+XUnt18J3Ec4J8PrwK3uPhptawWeBa53931L0/zC968vdPLNH75MEMAdv/Zmtr21Ld9NEpESl+vJ\nWzcDle6+DbiHMOSBqfl4vwrc5u7bgSeBjdG2CuCvgfjO4pxDEAR878eHePAHL1FVUcYnf+syhb6I\nFIRcwX8NsBvA3Z8GtmZtuxjoBT5pZj8CVri7R9s+B3wZ6FzU1haJ8YkMX//Hl/jujw/R3FjN//rd\n7bx5/cp8N0tEBMgd/I1AOmt5Ihr+AWgBtgF/BdwAXG9mO8zsNqDb3R+Pvq6kLjwfHB7jL//+Odqf\n72LDmgb+8NYtvElzfYpIAcl1cjcNNGQtJ919cpaOXmD/ZC/fzHYTviP4NSAwsxuAy4CHzOx97n50\nph+USjXMtLkoHDs+yJ8//G+8drSPq96yhv/2wS1UV4X/xHGobyZxri/OtYHqK0W5gr8duAl4xMyu\nBvZmbTsI1JvZBdEJ3+3Ag+7+uckvMLM9wJ25Qh8o+gmRD3Wm+cK393JqYJQbtp7Pb7/rIvrSQ/QR\n/wmf41xfnGsD1Vfs5ntQyxX8jwI3mll7tLzTzG4B6t19l5ndDnwrOtHb7u4/nFcritzPXunhK997\ngbGxDLdcfxE3Xrku300SETmnRJBjftNlEhTrUfnJZ3/Bt57YR0VZkjt//S1cfvEbp40qhV5HXOuL\nc22g+opdKtUwr3OouoFrnjKZgL/fs5/H/+01GmsruOs3387GNp3EFZHCp+Cfh5GxCXZ9/0V+uq+b\ntuZaPvGbbyelRy6ISJFQ8M9RemCU+7+9l0OdaS550wo+/htvo65ac9iKSPFQ8M9BZ+8Af/n3z9Fz\naph3vGUNO99zCeVluW6FEBEpLAr+WfJXT/DF7zzPwPA4v37NBt537UZNiiIiRUnBPwv/9+ddfOMf\nXyII4Pb3vplr3qZn7ohI8VLwzyAIAn7wrx08+tQhaqrK+fj738qlG1blu1kiIgui4D+H8YkMf/N/\nnB/v7aS5sYpP/ObbOS9Vn+9miYgsmIL/LAaHx3ngu8/zYscJ1q9p4BMf2ExTfVW+myUisigU/NP0\nnhrm899+jte7B7jswhbu/PW3UFVZlu9miYgsGgV/lsNdfXz+289xqn+U6684n1tuuIhkUlfuiEi8\nKPgjz+3v4Sv/8HNGxyb47esv4sat5+tyTRGJJQU/8M8//QV/+0/hg9Y+9v63scXe+KA1EZG4KOng\nzwQB395zgN0/eZWG2gp+7wObuWBtU76bJSKypEo2+DOZgK/8wws8492sWVXLJ/7T22nVg9ZEpASU\nbPC/2HGcZ7ybC89v4vf+42bqa/SgNREpDSX7hLGDR8I55N9z1XqFvoiUlBl7/GaWBB4ANgMjwB3R\n/LqT268E7gMSwOvArUAAfB1YD1QBn3X37y9J6xfgUGcY/BvbNBGziJSWXD3+m4FKd98G3EMY8gBE\n8+x+FbjN3bcDTwIbgQ8C3e7+TuBXgC8uRcMXIggCDnX1sbKhSnfkikjJyRX81wC7Adz9aWBr1raL\ngV7gk2b2I2CFuzvwCPCZrO8/vpgNXgwn+kZID4xqqkQRKUm5gr8RSGctT0TDPwAtwDbgr4AbgOvN\nbIe7D7h7v5k1EB4E/mCxG71QGuYRkVKW66qeNJCdjkl3z0Sve4H9US8fM9tN+I5gj5mtA74DfMnd\n/242DUmlli+Ej/7kNQAuu2T1sv3c5awvH+JcX5xrA9VXinIFfztwE/CImV0N7M3adhCoN7MLohO+\n24EHzWw18DjwMXffM9uGdHf3za3lC/DzAz0ArKwpX5afm0o1LGt9yy3O9cW5NlB9xW6+B7Vcwf8o\ncKOZtUfLO83sFqDe3XeZ2e3At6ITve3u/kMzux9oAj5jZpNj/b/q7sPzauEiywQBHV19rF5ZQ60m\nSReREjRj8Lt7AHx02up9Wdv3AFdN2+cu4K7FauBi6z4xxNDIOJsvaM53U0RE8qLkbuCaOrG7RuN+\nIlKaSjD4w/G+DbqUU0RKVOkFf1eaRALWr1aPX0RKU0kF/0Qmw6tdfZzXUqfpFEWkZJVU8B/pGWR0\nPKNhHhEpaSUV/Kfv2FXwi0jpKqng79CjGkRESiv4D3X1UV6W4PxUfb6bIiKSNyUT/GPjGX5xrJ91\nrfWUl5VM2SIib1AyCfjasX4mMoFO7IpIySuZ4D99x66CX0RKW8kEv07sioiESib4D3X1UVVRRltz\nXb6bIiKSVyUR/EMj43T2DLB+TQPJZCLfzRERyauSCP5Xj/YRoGEeEREokeCffCKn7tgVEckxEUs0\nsfoDwGZgBLgjmmZxcvuVwH1AAngduBUYn2mffOjoCk/s6lJOEZHcPf6bgUp33wbcQxjyAETTLX4V\nuM3dtwNPAhujfarOtk++HOpMU1ddTqqpOt9NERHJu1zBfw2wG8Ddnwa2Zm27GOgFPmlmPwJWuLtH\n+/zwHPssu/6hMbpPDrOhrZFEQid2RURyBX8jkM5anoiGfwBagG3AXwE3ANeb2Y4c+yw7Xb8vInKm\nGcf4CQM8OzGT7p6JXvcC+6NePma2m7B3P9M+y0537IqInClX8LcDNwGPmNnVwN6sbQeBejO7IDp5\nux14EDgwwz7nlEotTY/8yPEhALa8tY3mppol+RmzsVT1FYo41xfn2kD1laJcwf8ocKOZtUfLO83s\nFqDe3XeZ2e3At6ITve3u/sPo9Rn7zKYh3d1982l/Ti8fPs6K+koyo+NL9jNySaUa8vazl0Oc64tz\nbaD6it18D2ozBr+7B8BHp63el7V9D3DVLPbJixN9I5zqH+Xyi1ry3RQRkYIR6xu4Jk/s6vp9EZHT\nYh38h7p0RY+IyHTxDv7oUQ0bdEWPiMiU2AZ/EAR0dKZJraimvqYi380RESkYsQ3+7pNDDAyP68Fs\nIiLTxDb4NcwjInJ2MQ5+ndgVETmb2AZ/R2eaRALWr1Hwi4hki2XwZzIBh4/2s7a5jurKXDcni4iU\nllgGf2fvACNjE2zQMI+IyBvEMvh1YldE5NziGfxTd+wq+EVEpotl8Hd0pilLJljXWp/vpoiIFJzY\nBf/4RIbXjvVzfms9FeWxK09EZMFil4yvHetnfCLQMI+IyDnELvin5tjV9fsiImcVu+CfvKJHPX4R\nkbOb8e4mM0sCDwCbgRHgjmh+3cntdwO3A93Rqo8Qzrn7IHAxkAE+PDkh+3Lo6EpTWZGkraV2uX6k\niEhRydXjvxmodPdtwD3AfdO2XwF8yN13RB+vAO8G6tz9WuCPgD9e7Eafy8joBK/3DLB+dQNlydi9\nmRERWRS50vEaYDeAuz8NbJ22fQtwr5k9ZWb3ROuGgKZo0vUmYHQR2zujw0f7CALduCUiMpNcD7Jp\nBNJZyxNmlnT3TLT8MPAloA941MzeS3igqAZeBpqBmxa3yefWoSdyiojklKvHnwayUzQ79AHud/fj\n7j4GPAZcDvw+0O7uBlwGPGRmlYvZ6HM51KUTuyIiueTq8bcT9tgfMbOrgb2TG8ysCdhrZpcCg8C7\ngK8B2zn9LuEEUAGU5WpIKrXwXvqrx/qpq6ngLRe3kkgkFvz9FtNi1FfI4lxfnGsD1VeKcgX/o8CN\nZtYeLe80s1uAenffFY3r7yG84ucJd99tZv8P+IaZPUUY+p9296FcDenu7pt/FcDA8BidPQNcumEl\nPT39C/peiy2ValhwfYUszvXFuTZQfcVuvge1GYPf3QPgo9NW78va/jDhOH/2PieB98+rNQvQoev3\nRURmJTbXPE5OtagrekREZhab4O+YOrGr8TwRkZnEJvgPdaZpqqtkZUNVvpsiIlLQYhH8p/pHONE3\nwoY1DQV3NY+ISKGJRfDrwWwiIrMXk+CPTuwq+EVEcopH8HfpUQ0iIrNV9MEfBAEdnX20NFXTULss\nT4YQESlqRR/8PaeG6R8a0zCPiMgsFX3w6/p9EZG5KfrgPzQ1x656/CIis1H0wd/RmSYBrNfk6iIi\ns1LUwZ8JAjq6+ljTXEtNVa4HjYqICBR58Hf1DjI8OqEHs4mIzEFRB/8hTbUoIjJnRR38ega/iMjc\nFXXwH+pKU5ZM8KbV9fluiohI0ZjxjKiZJYEHgM2E0yve4e4HsrbfDdwOdEerPuLur5jZpwnn6q0A\nvujuDy12w8cnMrx6tJ/zUnVUlOec0ldERCK5LoW5Gah0921mdhVwX7Ru0hXAh9z93ydXmNl1wDui\nfeqATy1ymwF4vXuA8YmMhnlEROYo11DPNcBuAHd/Gtg6bfsW4F4zeyqaeB3gl4Hnzey7wPeB7y1i\ne6ecfjCbgl9EZC5yBX8jkM5anoiGfyY9DNwJvAu41szeCzQTHhA+APwO8LeL19zTOqbm2NUVPSIi\nc5FrqCcNZCdr0t0zWcv3u3sawMweAy4HeoGX3X0c2Gdmw2bW4u49M/2gVGpuAf5a9wCV5Une/uY1\nlJcV/jnqudZXbOJcX5xrA9VXinIFfzvhSdpHzOxqYO/kBjNrAvaa2aXAIGGv/2tAGXAX8Bdmthao\nIzwYzKi7u2/WjR4Zm+BwZx8b1zZw4vjArPfLl1SqYU71FZs41xfn2kD1Fbv5HtRyBf+jwI1m1h4t\n7zSzW4B6d98VjevvIbzi5wl33w1gZu80s58QDiV9zN2DebXuHF472k8mCPRgNhGReZgx+KPA/ui0\n1fuytj9MOM4/fb/fX5TWncPpO3YV/CIic1X4g+NnMXlFzwY9qkFEZM6KMvg7OvuoqSpj9arafDdF\nRKToFF3wDw6P03V8kA1rGkkmEvlujohI0Sm64D+sYR4RkQUpuuA/NDnHrq7oERGZl+IL/k71+EVE\nFqLogr+jM01DbQXNjdX5boqISFEqquBPD4zSmx5hY1sjCZ3YFRGZl6IK/kN6MJuIyIIVZfDrjl0R\nkfkrquDv6NIcuyIiC1U0wR8EAR2daZobq2isq8x3c0REilbRBP/x9AjpwTE2qLcvIrIgRRP8OrEr\nIrI4iif4NceuiMiiKJrg7+gMT+yqxy8isjBFEfyZIKCjK83qVbXUVlfkuzkiIkVtxhm4zCwJPABs\nJpxe8Q53P5C1/W7gdqA7WnWnu++LtrUCzwLXT66br6PHBxkameDtF6q3LyKyULnm3L0ZqHT3bWZ2\nFXBftG7SFcCH3P3fs3cyswrgr4FFmQm9Q0/kFBFZNLmGeq4BdgO4+9PA1mnbtwD3mtlT0cTrkz4H\nfBnoXIxG6o5dEZHFkyv4G4F01vJENPwz6WHgTuBdwLVm9l4zuw3odvfHo69Z8NPUOjr7SCYSrFtd\nv9BvJSJS8nIN9aSB7IH1pLtnspbvd/c0gJk9BlwO3AgEZnYDcBnwkJm9z92PzvSDUqmzj99PTGR4\n9Vg/69saOH/tihzNLVznqi8u4lxfnGsD1VeKcgV/O3AT8IiZXQ3sndxgZk3AXjO7FBgk7PV/zd0/\nm/U1ewhP+M4Y+gDd3X1nXf/q0T5GxyY4v6XunF9T6FKphqJt+2zEub441waqr9jN96CWK/gfBW40\ns/ZoeaeZ3QLUu/uuaFx/D+EVP0+4++55tWIGejCbiMjimjH43T0APjpt9b6s7Q8TjvOfa/8dC2od\nOrErIrLYCv4GrkOdacrLkpyXqst3U0REYqGgg39sfILXuwd40+p6yssKuqkiIkWjoNP01WP9TGQC\n3bglIrKICjr4px7M1qbLsUREFktBB79O7IqILL6CD/6qyjLWrKrNd1NERGKjYIN/aGScrt5BNqxu\nIJlc8FMfREQkUrDBf7irjwAN84iILLaCDf7JqRZ1YldEZHEVbvB36lENIiJLoWCDv6MzTX1NBS1N\n1fluiohIrBRk8PcNjtJzapgNbQ0kEjqxKyKymAoy+DXVoojI0inI4NeNWyIiS6cgg1+PahARWToF\nF/xBEHCoM83KhipW1FfluzkiIrFTcMF/om+EUwOjbFij3r6IyFKYcQYuM0sCDwCbCadXvMPdD2Rt\nvxu4HeiOVn0E6AC+DqwHqoDPuvv3Z9sgXb8vIrK0cvX4bwYq3X0bcA9w37TtVwAfcvcd0ccrwAeB\nbnd/J/ArwBfn0qCOLp3YFRFZSrmC/xpgN4C7Pw1snbZ9C3CvmT0VTbwO8AjwmazvPz6XBnV06lEN\nIiJLKVfwNwLprOWJaPhn0sPAncC7gGvN7L3uPuDu/WbWQHgQ+IPZNiYIAjq6+mhdWUNddcVsdxMR\nkTmYcYyfMPSzu95Jd89kLd/v7mkAM3sMuBx4zMzWAd8BvuTufzebhqRSDRzp6WdgeJwtb15NKhWv\nHn/c6pkuzvXFuTZQfaUoV/C3AzcBj5jZ1cDeyQ1m1gTsNbNLgUHCXv/XzGw18DjwMXffM9uGdHf3\n8eyLXQCsXVlDd3ffnAopZKlUQ6zqmS7O9cW5NlB9xW6+B7Vcwf8ocKOZtUfLO83sFqDe3XdF4/p7\nCK/4ecLdd5vZ/UAT8Bkzmxzr/1V3H87VmNM3bunErojIUpkx+N09AD46bfW+rO0PE47zZ+9zF3DX\nfBpzqDNNIgHrV+utmYjIUimYG7gmMhkOH+1jbUsdVZVl+W6OiEhsFUzwj4xOMDaWwdatyHdTRERi\nLdcY/7Kpra7gf95xFasa9XweEZGlVDDBD7C2pS7fTRARib2CGeoREZHloeAXESkxCn4RkRKj4BcR\nKTEKfhGREqPgFxEpMQp+EZESo+AXESkxCn4RkRKj4BcRKTEKfhGREqPgFxEpMTM+pC2aWP0BYDPh\nLFt3uPuBrO13A7cD3dGqjwD7gS+fax8REcmvXD3+m4FKd98G3APcN237FcCH3H1H9PEK8P4c+4iI\nSB7lCv5rgN0A7v40sHXa9i3AvWb2VDT/7mz2ERGRPMoV/I1AOmt5Ihr+mfQwcCfwLuBaM3vvLPYR\nEZE8yjURSxrInvk86e6ZrOX73T0NYGaPAZfPYh8REcmjXMHfDtwEPGJmVwN7JzeYWROw18wuBQYJ\ne/1fA2rPtc8MEqlUQ+6vKmKqr3jFuTZQfaUoEQTBOTeaWYLTV/UA7CQc1693911mdgtwN+HVO0+4\n+/842z7uvm+pChARkbmZMfhFRCR+dNJVRKTEKPhFREqMgl9EpMQo+EVESkyuyzmXVK5nARUjM6sA\nvg6sB6qAzwIvAd8EMsALwMfdvWjPqptZK/AscD1hTd8kPrV9mvBy5Argi4SXNH+TGNQX/b09CFxM\nWM+HgQmKvD4zuwr4M3ffYWYXcpZ6zOzDhM8SGwc+6+6P5a3BczStvsuALxD+v40At7r7sbnWl+8e\nf65nARWjDwLd7v5O4FeALxHWdW+0LgG8L4/tW5DowPbXwABhLX9BfGq7DnhH9Pt4HbCJGP3fAe8G\n6tz9WuCPgD+hyOszs08Buwg7WXCW30czWwP8LrAN+GXgT82sMh/tnauz1Pd54L+4+w7gO8Dvm9lq\n5lhfvoNi+1woAAACaElEQVQ/js/1eQT4TPQ6CYwBV7j7v0TrfgjckI+GLZLPET59tTNajlNt7wae\nN7PvAt8HvgdsiVF9Q0BTdK9NEzBK8de3H/gNwpCHs/8+Xgm0u/tY9KSB/Zy+z6jQTa/vt9198qbY\nCsL/0//AHOvLd/DH7rk+7j7g7v1m1kB4EPhDzvx37if8oys6ZnYb4buZx6NVCU7/QkIR1xZJEd6g\n+AHgd4BvEa/62oFq4GXCd21foMjrc/fvEA5vTMqup4+wnkbg1FnWF7zp9bl7F4CZbQM+Dvwl86gv\n3yEby+f6mNk64J+Bv3H3hwnHGyc1ACfz0rCF2wncaGZ7gMuAhwjDclIx1wbQAzzu7uPR3ebDnPkH\nVOz1fYqwZ2iE/39/Q9hrnFTs9cGZf2uNhPVMz5kG4MRyNmoxmdlvEb7rfo+79zKP+vId/O3AewDm\n8FyfghaNtz0OfMrdvxmt/ncz+6Xo9a8C/3K2fQudu/+Su18XjS/+DLgV2B2H2iI/Jjwvg5mtJXzu\n1JMxqq+O0++wTxBe3BGL380sZ6vnJ8B2M6uKnjH2ZsITv0XHzP4zYU//OnfviFbPub68XtUDPErY\ng2yPlnfmszGL5F7CXuJnzGxyrP8u4AvRCZcXgW/nq3GLLAD+K7ArDrW5+2Nm9k4z+wlhp+hjQAcx\nqY/w/Mw3zOwpwp7+pwmvzopDfZNXIr3h9zG6qucLwFOE/6/3uvtonto5X0E0DH4/cBj4jpkB/Ch6\nRtqc6tOzekRESky+h3pERGSZKfhFREqMgl9EpMQo+EVESoyCX0SkxCj4RURKjIJfRKTEKPhFRErM\n/wfwmTSyvvtAUgAAAABJRU5ErkJggg==\n",
       "text": [
        "<matplotlib.figure.Figure at 0x1178167d0>"
       ]
      }
     ],
     "prompt_number": 204
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Emotions"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We start with reading into memory the emotion labels:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "labels_emotions = list(read_labels(\"data/labels_emo.txt\"))\n",
      "labels_emotions, texts = zip(*match_labels_documents(documents, labels_emotions))\n",
      "labels_emotions, texts = np.array(labels_emotions), np.array(texts)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 205
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The dreambank distinguishes 5 different labels which are distributed as follows:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "label_counts = pd.Series(Counter(flatten_labels(list(labels_emotions))))\n",
      "label_counts.plot(kind='bar', figsize=(14, 6))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 225,
       "text": [
        "<matplotlib.axes._subplots.AxesSubplot at 0x124a6c5d0>"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAzIAAAGKCAYAAADXM3QcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGRhJREFUeJzt3X+w5Xdd3/HXZi/Jkrj5ZW9h6jBYCrxNq2kgiCX8iHH4\nIdKWSm3oKFbpkAwlDULNYLtYGG0ClF8pkUAtEWkpYglDaisCUWoJRCRECJIBPhAjoELsHXY32fxY\nsrm5/eOcZS6Z3bu7J+ec7/2c+3jMZHLv93z33PfMfuZ+v8893+8529bW1gIAANCT44YeAAAA4FgJ\nGQAAoDtCBgAA6I6QAQAAuiNkAACA7ggZAACgO0sbPVhVD0nyziSPTHJCkkuT/GWS303y5fFub2ut\nXV1VFyS5MMl9SS5trX1wZlMDAABb2raNPkemqn4+yZmttX9TVacl+VySX0lySmvtzev2e3iSa5Oc\nneShST6R5AmttXtnODsAALBFbfiKTJKrk7x//PVxSQ5kFCtVVc9N8pUkL0vyxCTXt9YOJDlQVbck\nOTPJjTOZGgAA2NI2vEemtXZXa+3OqtqZUdS8MskNSS5prZ2b5NYkr06yM8nt6/7oviSnzGZkAABg\nqzvSKzKpqkck+UCSK1trv11Vp7TWDkbLNUl+Lcl1GcXMQTuT7Nnoee+7b3VtaWn7ZFMDAABbwbbD\nPXCkm/0fltG9Ly9prf3hePOHq+qlrbVPJ3l6RpeP3ZDksqo6IcmOJGckuXmj596z5+6jH5/vWF7e\nmZWVfUOPwRZizTFP1hvzZL0xT9bbZJaXdx72sSO9IrMro0vEXlVVrxpve1mSy6vqQJJvJrlwfPnZ\nFUk+ntHlarvc6A8AAMzKhu9aNksrK/uG+cGdU/PMmzXHPFlvzJP1xjxZb5NZXt552EvLfCAmAADQ\nHSEDAAB0R8gAAADdETIAAEB3hAwAANAdIQMAAHRHyAAAAN0RMgAAQHeEDAAA0B0hAwAAdEfIAAAA\n3REyAABAd4QMAADQHSEDAAB0R8gAAADdETIAAEB3hAwAANAdIQMAAHRHyAAAAN0RMgAAQHeEDAAA\n0B0hAwAAdEfIAAAA3REyAABAd4QMAADQHSEDAAB0R8gAAADdETIAAEB3hAwAANAdIQMAAHRHyAAA\nAN0RMgAAQHeEDAAA0B0hAwAAdEfIAAAA3REyAABAd4QMAADQHSEDAAB0R8gAAADdETIAAEB3hAwA\nANCdpaEHgK1tbegBjmh1dTWbe85tQw8AAAxAyMDALn/fTVnZu3/oMbqzfOqOvPz8s4YeAwAYiJCB\nga3s3Z/bdt8z9BgAAF1xjwwAANAdIQMAAHRHyAAAAN0RMgAAQHeEDAAA0B0hAwAAdEfIAAAA3REy\nAABAd4QMAADQHSEDAAB0R8gAAADdETIAAEB3hAwAANAdIQMAAHRHyAAAAN1Z2ujBqnpIkncmeWSS\nE5JcmuSLSd6V5P4kNye5qLW2VlUXJLkwyX1JLm2tfXCGcwMAAFvYkV6R+ZkkK621pyX58SRXJnlT\nkl3jbduSPLeqHp7k4iTnJHlWktdW1fGzGxsAANjKNnxFJsnVSd4//vq4JAeSPL61dt1424eSPDPJ\napLrW2sHkhyoqluSnJnkxumPDAAAbHUbhkxr7a4kqaqdGUXNLyd547pd9iU5JcnJSW4/xHYAAICp\nO9IrMqmqRyT5QJIrW2vvrarXr3v45CR7k9yRZOe67TuT7NnoeU877cQsLW0/9onJ8vLOI+9EF1ZX\nV4ceoWunn35Stm/3e2TR+B3HPFlvzJP1Nl1Hutn/YUmuTfKS1tofjjd/tqrOba19LMmzk3w0yQ1J\nLquqE5LsSHJGRm8EcFh79tz9YGffkpaXd2ZlZd/QYzA1a0MP0LXdu+/K6FY9FoXfccyT9cY8WW+T\n2Sj+jvSKzK6MLhF7VVW9arztF5JcMb6Z/wtJ3j9+17Irknw8o3tpdrXW7n3QkwMAABzCke6R+YWM\nwuWBfvQQ+16V5KrpjAUAAHB4PhATAADojpABAAC6I2QAAIDuCBkAAKA7QgYAAOiOkAEAALojZAAA\ngO4IGQAAoDtCBgAA6I6QAQAAuiNkAACA7ggZAACgO0IGAADojpABAAC6I2QAAIDuCBkAAKA7QgYA\nAOiOkAEAALojZAAAgO4IGQAAoDtCBgAA6I6QAQAAuiNkAACA7ggZAACgO0IGAADojpABAAC6I2QA\nAIDuCBkAAKA7QgYAAOiOkAEAALojZAAAgO4IGQAAoDtCBgAA6I6QAQAAuiNkAACA7ggZAACgO0IG\nAADojpABAAC6I2QAAIDuCBkAAKA7QgYAAOiOkAEAALojZAAAgO4IGQAAoDtCBgAA6I6QAQAAuiNk\nAACA7ggZAACgO0IGAADojpABAAC6I2QAAIDuCBkAAKA7QgYAAOiOkAEAALojZAAAgO4IGQAAoDtC\nBgAA6I6QAQAAuiNkAACA7iwdzU5V9SNJXtdaO6+qHpfkfyf5yvjht7XWrq6qC5JcmOS+JJe21j44\nk4kBAIAt74ghU1WvSPKCJHeON52d5M2ttTev2+fhSS4eP/bQJJ+oqt9vrd07/ZEBAICt7mhekbkl\nyfOSvHv8/dlJHltVz83oVZmXJXlikutbaweSHKiqW5KcmeTG6Y8MAABsdUe8R6a19oGMLhc76FNJ\nLmmtnZvk1iSvTrIzye3r9tmX5JQpzgkAAPAdR3WPzANc01o7GC3XJPm1JNdlFDMH7UyyZ6MnOe20\nE7O0tH2CH8/y8s4j70QXVldXhx6ha6efflK2b/d7ZNH4Hcc8WW/Mk/U2XZOEzIer6qWttU8neXpG\nl4/dkOSyqjohyY4kZyS5eaMn2bPn7gl+NMvLO7Oysm/oMZiataEH6Nru3Xcl2Tb0GEyR33HMk/XG\nPFlvk9ko/o4lZA6ecb04yZVVdSDJN5Nc2Fq7s6quSPLxjC5X2+VGfwAAYFaOKmRaa19Ncs74688l\necoh9rkqyVXTHA4AAOBQfCAmAADQHSEDAAB0R8gAAADdETIAAEB3hAwAANAdIQMAAHRHyAAAAN0R\nMgAAQHeEDAAA0B0hAwAAdEfIAAAA3REyAABAd4QMAADQHSEDAAB0R8gAAADdETIAAEB3hAwAANAd\nIQMAAHRHyAAAAN0RMgAAQHeEDAAA0B0hAwAAdEfIAAAA3REyAABAd4QMAADQHSEDAAB0R8gAAADd\nETIAAEB3hAwAANAdIQMAAHRnaegBAJintaEH2NDq6mo294zbhh4AgDEhA7DFXP6+m7Kyd//QY3Rl\n+dQdefn5Zw09BgDrCBmALWZl7/7ctvueoccAgAfFPTIAAEB3hAwAANAdIQMAAHRHyAAAAN0RMgAA\nQHeEDAAA0B0hAwAAdEfIAAAA3REyAABAd4QMAADQHSEDAAB0Z2noAQCARbU29AAbWl1dzeaecdvQ\nA8CmJmQAgJm5/H03ZWXv/qHH6MryqTvy8vPPGnoM2PSEDAAwMyt79+e23fcMPQawgNwjAwAAdEfI\nAAAA3REyAABAd4QMAADQHSEDAAB0R8gAAADdETIAAEB3hAwAANAdIQMAAHRHyAAAAN0RMgAAQHeE\nDAAA0J2lo9mpqn4kyetaa+dV1aOTvCvJ/UluTnJRa22tqi5IcmGS+5Jc2lr74IxmBgAAtrgjviJT\nVa9I8o4kJ4w3vTnJrtba05JsS/Lcqnp4kouTnJPkWUleW1XHz2ZkAABgqzuaS8tuSfK8jKIlSR7f\nWrtu/PWHkjw9yQ8nub61dqC1dsf4z5w57WEBAACSowiZ1toHMrpc7KBt677el+SUJCcnuf0Q2wEA\nAKbuqO6ReYD71319cpK9Se5IsnPd9p1J9mz0JKeddmKWlrZP8ONZXt555J3owurq6tAjdO3000/K\n9u1+jxwLa25y1tuxs94mZ70tJudw0zVJyHy2qs5trX0sybOTfDTJDUkuq6oTkuxIckZGbwRwWHv2\n3D3Bj2Z5eWdWVvYNPQZTszb0AF3bvfuufPeLxByZNTcp620S1tukrLfF4xxuMhvF37GEzMHfRr+Y\n5B3jm/m/kOT943ctuyLJxzO6XG1Xa+3eCecFAADY0FGFTGvtqxm9I1laa19J8qOH2OeqJFdNcTYA\nAIBD8oGYAABAd4QMAADQHSEDAAB0R8gAAADdETIAAEB3hAwAANAdIQMAAHRHyAAAAN0RMgAAQHeE\nDAAA0B0hAwAAdEfIAAAA3REyAABAd4QMAADQHSEDAAB0Z2noATaftaEH2NDq6mo294zbhh4AAIAt\nQMgcwuXvuykre/cPPUZXlk/dkZeff9bQYwAAsEUImUNY2bs/t+2+Z+gxAACAw3CPDAAA0B0hAwAA\ndEfIAAAA3REyAABAd4QMAADQHSEDAAB0R8gAAADdETIAAEB3hAwAANAdIQMAAHRHyAAAAN0RMgAA\nQHeWhh4AAAAevLWhB9jQ6upqNveM24Ye4JgJGQAAFsLl77spK3v3Dz1GV5ZP3ZGXn3/W0GNMRMgA\nALAQVvbuz2277xl6DObEPTIAAEB3hAwAANAdIQMAAHRHyAAAAN0RMgAAQHeEDAAA0B0hAwAAdEfI\nAAAA3REyAABAd4QMAADQHSEDAAB0R8gAAADdETIAAEB3hAwAANAdIQMAAHRHyAAAAN0RMgAAQHeE\nDAAA0B0hAwAAdEfIAAAA3REyAABAd4QMAADQHSEDAAB0R8gAAADdETIAAEB3hAwAANCdpUn/YFV9\nJsnt429vTfLaJO9Kcn+Sm5Nc1Fpbe7ADAgAAPNBEIVNVO5KktXbeum3/K8mu1tp1VfX2JM9N8j+n\nMiUAAMA6k74i8/eTnFhVHxk/xyuTPL61dt348Q8leWaEDAAAMAOT3iNzV5I3tNaeleTFSd7zgMfv\nTHLKgxkMAADgcCZ9RebLSW5JktbaV6rqW0ket+7xnUn2bvQEp512YpaWtk/442dndXV16BG6dfrp\nJ2X79s33d7qZWW8PjjV37Ky5yVlvx856m5z1duyst8n1ut4mDZkXJjkzyUVV9bcyCpdrq+rc1trH\nkjw7yUc3eoI9e+6e8EfPmvcnmNTu3Xcl2Tb0GJ2x3h4Ma24S1tykrLdJWG+Tst4mYb1NajOvt+Xl\nnYd9bNKQ+Y0kv1lVB++JeWGSbyV5R1Udn+QLSd4/4XMDAABsaKKQaa3dl+RnD/HQjz6oaQAAAI6C\nD8QEAAC6I2QAAIDuCBkAAKA7QgYAAOiOkAEAALojZAAAgO4IGQAAoDtCBgAA6I6QAQAAuiNkAACA\n7ggZAACgO0IGAADojpABAAC6I2QAAIDuCBkAAKA7QgYAAOiOkAEAALojZAAAgO4IGQAAoDtCBgAA\n6I6QAQAAuiNkAACA7ggZAACgO0IGAADojpABAAC6I2QAAIDuCBkAAKA7QgYAAOiOkAEAALojZAAA\ngO4IGQAAoDtCBgAA6I6QAQAAuiNkAACA7ggZAACgO0IGAADojpABAAC6I2QAAIDuCBkAAKA7QgYA\nAOiOkAEAALojZAAAgO4IGQAAoDtCBgAA6I6QAQAAuiNkAACA7ggZAACgO0IGAADojpABAAC6I2QA\nAIDuCBkAAKA7QgYAAOiOkAEAALojZAAAgO4IGQAAoDtCBgAA6I6QAQAAuiNkAACA7ggZAACgO0vT\nfLKqOi7J25KcmeTbSV7UWvuzaf4MAACAab8i80+SHN9aOyfJv03ypik/PwAAwNRD5slJPpwkrbVP\nJXnClJ8fAABgupeWJTk5yR3rvl+tquNaa/c/cMezz/7BQz7Bn/zJzYfcPs/977jr3qzev5Yk+eeX\n/LdD7v/bb/wXh9y+VfdfPnXHd23fzH+/m21/622y/devuc3897sZ9z+45jbz3+9m2996m3x/6+3Y\n97feJt/fejv2/Tf7evv61792yMeSZNva2tphHzxWVfWmJH/cWrt6/P1ftNYeMbUfAAAAkOlfWnZ9\nkp9Ikqr6B0n+dMrPDwAAMPVLy65J8oyqun78/Qun/PwAAADTvbQMAABgHnwgJgAA0B0hAwAAdEfI\nAAAA3REyAABAd4QM8B1V9UNV9aih5wCYl6r6vqFnACYz7bdfZoqq6tWH2LyWJK21X53zOCy4qros\nyXlJTqiq/9Rae/fQM7HYquqnkvzrJI9M8vUkbz34gcowa1X1Y0kuSvKUJA8beBwWUFX9wyTPT/I3\nkvxlkve21v7PsFMtFiGzuf11xuEydlKSX0rytSRChmk7L8mTkzw0ye8kETLMTFX9bEYH+Bcn+fMk\nj0ny+qr6ntbabw46HAurqr4nyc8l+VdJHp7kpUl+ZtChWEhVdVGSZyd5S5L/l9E/2Oyqqke31v7L\noMMtECGzibXW/vPBr6vqyUmuSvLWJK8ZbCgW2f7W2lqSu6tq+9DDsPAuTPKM1tr+8fc3V9X5Sa5N\nImSYuqp6a5Ify+jDu38yyRWttd8adioW2AuSPLW1dt/4+89V1UeS/H4SITMlQmaTq6rjk1yW5BlJ\nfrq19tmBR2JxbTvM1zAL962LmCRJa+3OqlodaiAW3lOS3Jjkj5PcOvAsLL5vr4uYJElr7dtVdWCo\ngRaRm/03sap6fJJPJ1lN8kQRw4ydXVWfrKpPJnn8wa+r6o+GHoyFdFxV7Vy/Yfy94xIz0Vo7K8mv\nJ3leki8leWxVnTHsVCywtcNs9ztuirwis7l9KsntSc5N8rGqOrh9rbV2zmBTsajOPMS2tSQnznsQ\ntoS3Jrmmql6R5M+S/O0kb0hy5aBTsdBaa9cnub6qTs7o3pj/XlVrrbUnDDwai+fs8T8MPtDfnfsk\nC0zIbG6POcQ2J5bMRGvtq+u/r6q/k+QlGV3n6x19mKrW2tVVdUeSX0nyqCR/leQtrbXfHXYytoLW\n2h1J3p7k7VX1uKHnYSGdmdE528FLtQ++QuPS7SkSMpuYE0vmraq2JfmJjN6S9MlJ/mOSswYdikX2\nB621jyTfuazsnoHnYYFV1Z8f5qG1jGIapulrSf5xa+13quqUJP8+yf4krx12rMXiOr1Nrqq2VdVz\nqur3knwmybfixJIZqKpLknw+yU8neXOST7bWXtNa++awk7GIquoHk3y5qk4bb3r6+Pu/N+BYLLYz\n1v339SQ/MP7apT7MwmuT/FxVLWV0Ke1JSXZn9EogU+IVmU1sfGL580k+l9GJ5XGtNW+9zKxckuS9\nSd7ZWvt8Vf3i0AOx0K5I8vzW2p4kaa1dU1V/ndFnLjx90MlYSOvfJa+q7m+tfXvIeVh457bWnlRV\nD0nynCSPaK3d5Q10pssrMpvbJRm93/jrWmt/kMO/AwZMw/dn9Nakb6mqTyV5RFWdOuxILLBtrbUb\n129orf1RkuMHmgdgmu4Y//+Hk3y+tXbX+PuHDDTPQvKKzOb2/Un+aUYnliclOamqTm2t7R12LBbR\n+F8r35PkPVX1mCQvSnJTVd3YWvupYadjAR3uQ1cd5JmJGr3158Gbr0+sqseOv15rrX150OFYRAeq\n6plJXpjkA0lSVU9NsmfQqRbMtrU1/8jfg3Unls9P4sSSmauqv5nR9bz/qLV2zdDzsFiqaleS05P8\nh9ba7eOb/V+d0YfIvXLY6VhEVfV/893vHPWdE6DW2nlDzMTiqqpHJ3lNktsyusLmvCSvz+iS2i8N\nOdsiETIdcWLJPFTVeUl+I6OXxU9NcmFr7dphp2LRjN8h75eSXJDRW8rvSfJfk7yhtXb/kLOxNYyP\nqd9qra0OPQuLz3qbDSHTASeWzFNVXZ/kn7XWvlFV35fkmtbaE4eei8XmIM+8OKYyT9bbbLnZvw+X\nJnlKa+2sjD7b49KB52Gx3dda+0aStNb+Kj7bgxmqqvOq6tYk1ya5dXxNOcySYyrzZL3NkJDpgxNL\n5mlfVV1cVWdV1cUZXc4Is7L+IH9OHOSZPcdU5sl6myHvWtaHfeMTyo8neWqcWDJbL0jyy0meleSL\nSf7lsOOw4L7rIF9VDvLMmmMq82S9zZCQ6YMTS+bpziTvTrJj/P1jktww3DgsOAd55s0xlXmy3mZI\nyPTBiSXz9HsZfSjh+ve6/8mBZmHxOcgzb46pzJP1NkNCpg9OLJmnE1pr5w49BFuGgzzz5pjKPFlv\nMyRk+uDEknm6rqp+PMkXDm5orX19wHlYbA7yzJtjKvNkvc2QkOmDE0vm6WFJLk+yd922Jw00C4vP\nQZ55c0xlnqy3GRIyfXBiyTz9QGvtjKGHYMtwkGfeHFOZJ+tthoRMH5xYMk9/WlVPSvKZJGtJ0lq7\nd9iRWGAO8sybYyrzZL3NkJDpgxNL5uncJM9J8r0Z3bewmuRRg07EInOQZ94cU5kn622GhEwfnFgy\nTxcnuTLJt5P8jyR/Mew4LDgHeebNMZV5st5m6LihB+CoXJzk7iTfSPKuJK8ZdBoW3a8meVqSbyZ5\nY5KXDDsOC+7cJL+VZCXJV5J8adhx2AIcU5kn622GhEwfnFgyT2uttW8lSWttX5I7Bp6HxeYgz7w5\npjJP1tsMCZk+OLFknm6pqtcl+d6q+ndJvjb0QCw0B3nmzTGVebLeZkjI9MGJJfP04ozW2Ccy+tT1\nC4YdhwXnIM+8OaYyT9bbDLnZvw8vTvKiOLFkDlprB5K8feg52DIc5Jk3x1TmyXqboW1ra2tDzwDA\nFlVVD8noIP9DSb6Y5Ne9axkAR0PIAAAA3XGPDAAA0B0hAwAAdEfIAAAA3REyAABAd4QMAADQnf8P\nX6at5ag12NgAAAAASUVORK5CYII=\n",
       "text": [
        "<matplotlib.figure.Figure at 0x1247c3b10>"
       ]
      }
     ],
     "prompt_number": 225
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = small_doc_experiment(texts, labels_emotions)\n",
      "score_df['emotions']['small_doc'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.58 (+/- 0.03)\n"
       ]
      }
     ],
     "prompt_number": 173
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = big_doc_experiment(texts, labels_emotions)\n",
      "score_df['emotions']['big_doc'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.58 (+/- 0.02)\n"
       ]
      }
     ],
     "prompt_number": 174
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = ml_experiment(texts, labels_emotions)\n",
      "score_df['emotions']['ml'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.67 (+/- 0.04)\n"
       ]
      }
     ],
     "prompt_number": 175
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = np.zeros(11)\n",
      "for i, k in enumerate(range(1, 102, 10)):\n",
      "    scores[i] = ml_experiment(texts, labels_emotions, num_neighbors=k).mean()\n",
      "sb.plt.plot(range(1, 102, 10), scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 206,
       "text": [
        "[<matplotlib.lines.Line2D at 0x117c35b10>]"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAECCAYAAAD5OrxGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VfWd//HXvdn3hKwkJCEE8mUNWwLIpshORajV2tZq\nYaqtttPNmWmd+c30MZ3pzNhaO9Z2bKt21E61VlrFpUJZBFlUdggE8wUSyEYSkpB9v/ee3x/3UiNC\nAuHenLt8no9HHnLvucvnY27e93u277EYhoEQQojAYTW7ACGEEMNLgl8IIQKMBL8QQgQYCX4hhAgw\nEvxCCBFgJPiFECLABA+0UCllBZ4C8oEe4H6tdWm/5YXA44AFqAbuA2zAs0Ae4AAe0Fprj1QvhBDi\nug024l8LhGqt5wKP4Ax5AJRSFuBpYJ3WegGwHcgBlgFRWuv5wL8B/+GJwoUQQgzNYME/D9gMoLXe\nBxT0W5YHNAIPK6V2AvGukX0XEOf6YogDet1dtBBCiKEbcFMPEAu09rttV0pZtdYOIAmYC3wdKAXe\nUkodBHYD4UAJkAisdnvVQgghhmywEX8rENP/8a7QB+do/4x2suFcMygAvgvs1VorYBrwglIq1M11\nCyGEGKLBgn8vsApAKTUHKOq3rAyIVkrlum4vAIqBKD5aS2gCQoCggd7EcE4YJD/yIz/yIz/X9zMk\nloEmaXNtp790VA/AemAmEK21fkYptQh4FOdRPXu11t9RSsUDz+HcFBQCPKG1fnmQOoz6+rah9uD1\nkpNjkP58kz/3BtKfr0tOjrEM5XkDBv8wkuD3Yf7cnz/3BtKfrxtq8MsJXEIIEWAGO6pHCOHnDMPg\nF68ex+4w+Ortk4gIk1jwdzLiFyLAlVQ0c+R0A0Wljfzk5aN0dveZXZLwMAl+IQLc1gOVAIzPiuds\nTSuPvXyU9i4Jf38mwS9EADvf0M6xMw3kpsfy95+bzvwpIymvbeMnvz8i4e/HJPiFCGBv7i7DAJYW\nZmK1Wli3ajw3T0un4kI7P37pCK2dMuOKP5LgFyJAdXb3sW1/BSNiw5ipkgGwWizcu1yxaEYGVfXt\nPPbSEVo6JPz9jQS/EAFq17EaunvtLJ4xiiDrR1FgtVj44tI8lswcRXVDBz9+6TDN7T0mVircTYJf\niABkdzjYfqiSsNAgFk5L/8Ryi8XC55eMY1lhJjWNnfzopSM0tUn4+wsJfiEC0JFTDTS29rC4IJOo\n8JArPsZisXD3rWNZOSeLuoud/OjFw1xs7R7mSoUnSPALEYC2uA7hvH1h7oCPs1gs3HlzLrfNHc2F\n5i4effEwDc1dw1Gi8CAJfiECTNn5Vs5Ut5Cfm0hGcvSgj7dYLNyxcAxr5ufQ0NLNj146zAUJf58m\nwS9EgNl60DnaX1qYeV3PWzM/h08vHENjaw8/evEwdU2dnihPDAMJfiECyMXWbg6WXCAjOYqJ2QnX\n/fzVc0dz1y25NLU5w7+mscMDVQpPk+AXIoC8c7gau8NgaUEmFsuQZvRl5Zxs7r51LM3tvfz4pSOc\nb5Dw9zUS/EIEiJ4+O+8erSY6IoSbJqXe0Gstn5XF55eMo6Wjlx+/dJiq+nY3VSmGgwS/EAHi/RO1\ndHTbWDQ9g5DgAa+Gek2WFmRy77I8Wjv7+PFLR6io898LnvgbCX4hAoDDMNh6sJIgq4VbZ2S47XUX\nzRjFl1Yo2rv6eOz3RyivlfD3BRL8QgSA4rMXqWnsZPbEVOKiw9z62jdPy2D9qvF0dtt47PdHOFvT\n6tbXF+4nwS9EALh0wtbSgus7hPNaLchP58u3TaCr18ZPXj5K6fkWj7yPcA8JfiH8XHV9O8VnL6Iy\n48lOi/HY+8ydPJIHbptId6+Nx18+ypkqCX9vJcEvhJ/berAKgGXXecLWUMyZlMaDaybT2+fg8VeO\ncqqy2ePvKa6fBL8Qfqyts5f3i2tJjg9n6tikYXnPwvEpPLR2Ejabg5++cpSS8qZheV9x7ST4hfBj\nO4+ep8/mYEmB8wpbw2WmSuFraydjtxs8seEYxecuDtt7i8FJ8Avhp2x2B+8criIiLIj5U0YO+/tP\nz0vmb++YgsMwePKPRZwoaxz2GsSVSfAL4acOfHiBlvZeFuSnExEWbEoNU8cm8c3P5GMY8OSfiigq\nbTClDvFxEvxC+CHDMNhyoBKLBRbPHGVqLZPHJPKtu/KxWiz8/E/HOXK63tR6hAS/EH7pdFUL5XVt\nzBiXTHJ8hNnlMGn0CL5911SCgiw89doJDukLZpcU0CT4hfBDWw8Mbc59TxqfncDDn51GcJCVX24s\nZk9RDYZhmF1WQBpww59Sygo8BeQDPcD9WuvSfssLgccBC1AN3Ad8AfiS6yERwFQgVWst53ELMQzq\nm7s4fLqe7LQYxo2KM7ucj8nLjOfv7p7GT185yv++/SGHT9XzxWV5jIgNN7u0gDLYiH8tEKq1ngs8\ngjPkAVBKWYCngXVa6wXAdiBHa/281nqR1noRcBD4hoS+EMNn+6EqDMN5wtZQ59z3pLGj4vjX9YWM\nz4rn6JkG/uU3+9h5tBqHjP6HzWDBPw/YDKC13gcU9FuWBzQCDyuldgLxWmt9aaFSqgCYpLV+1q0V\nCyGuqqvHxq5j54mLDqVwfIrZ5VxVSkIk//D56axbOR6A327W/OT3R+RyjsNksOCPBfqP1u2uzT8A\nScBc4OfAEmCxUmpRv8f+E/CvbqpTCHEN9hTV0N1rZ/GMUQQHefcuPIvFwsKp6fzw/jlMG5tESUUz\n3//NfjbtK8fucJhdnl8b7JPRCvSf1cmqtb70G2kEzmgnG841gwIApVQ8kKe1ftfdBQshrszhcM65\nHxJs5eZp6WaXc80SYsL4xmem8OCaSYSHBrFhRyk//O0hubCLBw12VsdeYDWwQSk1Byjqt6wMiFZK\n5bp2+C4ALm3WWYhzm/81S0723KyB3kD6812+0tv7x2toaOlm+ZxsxmQnXvPzvKW/T6XEsmBmFs++\nfpwdh6r49xcOcuet47h7ad4NXTHMW/rzJpaBDqdy7cC9dFQPwHpgJhCttX7GtWnnUZxH9ezVWn/H\n9by/B3q11k9eYx1Gfb3/frsnJ8cg/fkmX+rt0RcPc6qymX+/fzYZSVHX9Bxv7e94WSO/3VxCY2sP\nIxMjWb9yAmOHcISSt/bnLsnJMUPaez9g8A8jCX4f5s/9+Upv5bVt/OD5A0zOGcHDd0+75ud5c39d\nPTZefbeM7YersOA8A/mOm8cQHnrt0094c3/uMNTg9+69P0KIa7L1oPedsHWjIsKCuWdZHo/cM4PU\nEZFsO1TFvzy7nxNnZbK3GyXBL4SPa27vYd/JOkYmRjI5Z4TZ5bhdXmY8P/ibQj51UzZNbT389A/H\n+M2fT9Le1Wd2aT5Lgl8IH7fjcDV2h8HSAu88YcsdQoKD+MzNuXx/XQFZqdHsPV7LPz+7j4MlMufP\nUEjwC+HDevvs7DhSTVR4MDdNTjO7HI/LSo3hX75UwJ235NLZbeOpjSf4xavHaW7vMbs0n2LOJN1C\nCLf44GQd7V19fOqmbMJChn7Ioy8JslpZNSebGXnJPO+a76ekvIm7bx3L/PyRfrvW404y4hfCRxmG\nwdYDlQRZLdw6w9w5982QNiKS794zg3uX5eEwDJ7bVMLjfzjKheYus0vzehL8Qviok+VNVDd0UDg+\nhYSYMLPLMYXVYmHRjFH88P7Z5OcmcvJcE9//zT62HKjE4fCKQ9W9kgS/ED7KG+fcN8uI2HC+dWc+\nX1k9kdDgIF7efpr//N0hymtlYuArkeAXwgfVNHZQVNrI2FFx5IyMNbscr2CxWJgzKY0fPjCb2RNT\nKTvfyrd/upM39pzFZpdJ3/qT4BfCB207WAXAsgIZ7V8uNjKUr94+iW9+Jp+46DA27jnLD54/QE1j\nh9mleQ0JfiF8THtXH3tP1JAYG870vCSzy/Fa08Yl8T//cCu3TEunur6D3UU1ZpfkNeRwTiF8zK5j\n5+ntc7B4/iiCrDJ2G0hURAj3rRjPijnZJESHml2O15DgF8KH2OwOth+qIiw0iIVTfWfOfbOlxEeY\nXYJXkeGCED7kkK6nqa2H+VNGEhku4zYxNBL8QviQrQcrsQBLCgLvhC3hPhL8QviIM9UtlJ1vZerY\nJFITIs0uR/gwCX4hfMSlE7aWyQlb4gZJ8AvhAxpbujmk68lMiUZlxZtdjvBxEvxC+IDth6twGP49\n574YPhL8Qni57l4bu46eJzYyhNkTU80uR/gBCX4hvNze47V09thYNGMUIcHyJytunHyKhPBiDsNg\n28FKgoMsLJqeYXY5wk9I8AvhxYpKG6lr6mLOxDRio2TKAeEeEvxCeDGZc194ggS/EF6q8kI7H5Y3\nMSE7gcyUaLPLEX5Egl8IL7X1oIz2hWdI8AvhhVo7evmguI7UhAjycxPNLkf4GQl+IbzQjiPV2OwO\nlhRkYpUTtoSbSfAL4WVKq1t4+4NyosKDmTclzexyhB+S4BfCi1xo7uLJPxVhszt4YPVEwkNlzn3h\nfgN+qpRSVuApIB/oAe7XWpf2W14IPA5YgGrgPq11r1LqH4HVQAjwC631Cx6qXwi/0d7VxxOvHKOt\ns497lyvyc+V6usIzBhvxrwVCtdZzgUdwhjwASikL8DSwTmu9ANgO5CilbgFucj3nFmCMB+oWwq/0\n2Rz84tXj1F7sZMWsLDlLV3jUYME/D9gMoLXeBxT0W5YHNAIPK6V2AvFaaw0sB44rpTYCbwJvuLto\nIfyJYRg8t+lDTlU2M1Mlc+eiXLNLEn5usOCPBVr73ba7Nv8AJAFzgZ8DS4DFSqlFQCIwE7gTeBB4\n0a0VC+FnNu4+ywfFdeSmx/LAbRPlKB7hcYPtOWoFYvrdtmqtHa5/NwJnXKN8lFKbca4RNAIlWmsb\ncEop1a2UStJaNwz0RsnJMQMt9nnSn+/yZG/b9lfw5nvnSEuM5AdfnUtcdJjH3utq/Pl3B/7f31AM\nFvx7ce6k3aCUmgMU9VtWBkQrpXJdO3wXAM/iXIv4FvBTpVQ6EIXzy2BA9fVtQyjfNyQnx0h/PsqT\nvZ08d5FfbDhGVHgw37hjCr1dvdR39Xrkva7Gn393EBj9DcVgwf8asFQptdd1e71S6vNAtNb6GaXU\nl4GXXDt692qtNwEopRYqpfbj/BL4mtbaGFJ1Qvip6vp2/ue141gs8Ld3TGFkYpTZJYkAMmDwuwL7\nocvuPtVv+Q5g9hWe9z23VCeGrL2rj5LyJpLiw8lIiiIkOMjskoRLS3sPT2w4RlePna+snojKSjC7\nJBFg5OwQP1PT2MHWg1W8d7yGXptzd0yQ1cLIxEgyU2LISo0mKzWGzJRooiNCTK428PT02nnij0U0\ntvbw6YVjmDNJzswVw0+C3w8YhkFJeRN/OVBJUalzd0pibDjzpqTR1tVHZV07lRfaqarv4P3ij56X\nGBv2sS+DrJRoEuPC5WLeHuJwGPz6jWLKa9uYnz+S227KNrskEaAk+H1Yn83BvpN1bDlQSVV9OwC5\nGbEsL8xiel4SQdaPjtZ1OAwuNHdRUddG5YV2yuvaqKxr5+iZBo6e+eiAq4iwYLJSXF8EqdFkpkST\nnhRFcJDM7nGjXt5+mqNnGpg4OoH7liv5ghWmkeD3Qa2dvew8Us07h6tp7ejFarFQOD6FZYWZ5GbE\nXfE5VquFtBGRpI2IZNaE1L/e39Le89EXwYV2KuraOVXZjK5s/utjgoMspCdFkZUSQ2ZqNFkp0WSm\nxBAZLh+fa7X1QCXbDlWRkRTF19ZOkS9SYSr5y/Uh1Q0dbD1QyfvFtfTZHESEBbFiVhaLZ44iMS58\nSK8ZFx1GXHQYk8d8NOd7T6+dyvp2KuvaqLjQTkVdG1X1HVTUtcPxj56bHB9OVkoM48ckkhgVSk56\nLHFyXdhPOHKqnpe3nyYuKpRv3zVVvjCF6eQT6OUMw+DkuSb+cqCCE2UXAWfgLinIZP6UkUSEuf9X\nGBYaxNiMOMb2W3uwOxzUXuxyfhnUtVNxwfnfQ6fqOXSqHoDQYCvf/cIMxqTHur0mX3W2ppVfv1FM\nSIiVb92VP+QvaCHcSYLfS/XZ7LxfXMfWg5VU13cAkDcqjqWFWUwfl4TVOrzbh4OsVjKSoshIimLO\nJOd9hmHQ3N5LS7eNQydr+fP75by6q5S//9z0Ya3NWzU0d/GzPxbRZ3fwjTvyGZ0mX4jCO0jwe5mW\njl52HK5ix5Fq2jr7CLJamDMxlaWFmeSM9K7gsFgsJMSEkTcmidHJUZyrbaP47EV0RVPAH5ve2d3H\nf284RmtHL/cszWPaOJliWXgPCX4vUVXfzpYDlXxQXIvNbhAZFszKOVksnjGKEbG+sXng0wvGUHz2\nIq/tPsv3vhAfsEet2OzOKZZrGjtZVpjJ4pmjzC5JiI+R4DeRwzA4UXaRrQcqKD7XBEBKQgRLCzKZ\nNyXN566+NCY9lqm5iRwrbeRkeROTRo8wu6RhZxgGL2wqoaSimRl5yXx20VizSxLiE3wrWfxEb5+d\n94pr2XqgkprGTgDGZ8WztDCTqWOTfHpa3rULxnCstJGNu8qYmJ0QcKP+N/eeY++JWnJGxvDA6onD\nvi9GiGshwT+Mmtt7eOdwNTuPVNPe5dx+f9OkNJYVZpKd5h9Tx2anxTAzL5lDp+o5XtYYUJcPfO9E\nDRv3nCUpLpxv3jmVsBCZH0l4Jwn+YXKutpVHf3eYXpuDqPBgPnVTNrfOGEVCzPDPv+5paxbkcPhU\nPa/tPsuUMYkBMeovKW/iubdLiAwL5tt3TZXzGYRXk+AfJm/uPUevzcFnF41l0YwMvx4NjkqOpnBC\nCvs/vMCR0w3MyEs2uySPOt/QwS9edZ7Z9vU7ppCeJFMsC+8m540Pg8q6No6ebiA3PZblszL9OvQv\nWTM/B4sFNu4uw2H47+UYWjp6eWLDMTp7bKxbOZ4J2YF9GKvwDRL8w2Dju6UYwIrZWQGx2QNgZGIU\ncyamUVXfwcGSC2aX4xE9fXae/GMRDS3drJmfw7wpI80uSYhrIsHvYc3tPbxzsJLUhAimj/PvTR6X\nu33+aKwWC6/vOYvD4V+jfofD4Jk3T3K2ppW5k9O4fd5os0sS4ppJ8HvY9kNV2OwOls/OCrhD+1IT\nIpk3JY2axk72nawzuxy3emXHGQ6fqmd8VjzrVo4PmDU54R8k+D2oq8fGO4eriY8OY97kwLzS0up5\nowmyWnh971nsDofZ5bjF9kNVbDlQycjESL5+h0yxLHyPfGI9aNex83T12LhtQU7AXvM2KS6ChVPT\nudDUxXvHa80u54YdPdPAS9tOERsZwnfumkpUuFy+UvgeCX4PsdkdbDlQSVhIEKvm5phdjqlumzua\n4CArb+w9h83uu6P+c7Wt/Or1E4QEWfnWXVNJio8wuyQhhkSC30P2f1hHU1sPC6aOJCYysE/mSYgJ\n45bp6TS2drO7qMbscoaksaWbn20ooq/PwVdun+R1M6UKcT0k+D3AMAw276vAarGwrDDT7HK8wqfm\nZBMabOWt987RZ7ObXc516bM5Z9ts6ejl7sXj/P6ENOH/JPg94MTZi1TVdzBrQgpJcbI5AJyXeLx1\n5iia2nrYefS82eVclw07zlBe18b8/JEsLZAploXvk+D3gE0flAPOE7bER1bOziIsNIg/v19OT59v\njPrfP17DtkNVpCdFcc/SPDlsU/gFCX43O1vTSklFM5NyRpCV6h8zbrpLTGQoSwtG0drRy47D1WaX\nM6iGli5+9ocjhAZbeWjNpICYakMEBgl+N9u8rwKQ0f7VLJ+VRURYMG9/UE53r83scq7KZnfw6zeK\n6ejq4wtL88hIjja7JCHcRoLfjS40d3FQXyArNZqJMlnXFUWFh7C8MJP2rj62H6oyu5yrem13GaXV\nrSycnsGCfJmDR/gXCX432rK/AsOAlbOzZVvwAJYWZhIVHszmfRV0dnvfqP9EWSObPqggJT6Cr985\nVX6Xwu8MOB+/UsoKPAXkAz3A/Vrr0n7LC4HHAQtQDdynte5VSh0GWlwPK9Naf9kTxXuT1s5e9hTV\nkBQXTsF4OdxvIBFhwayYncWf3i1j68FK1sz3nhPcmtp6eOatkwQHWXho7WQiw0PoaOs2uywh3Gqw\nEf9aIFRrPRd4BGfIA6CUsgBPA+u01guA7UCOUiocQGu9yPXj96EP8M6hKnptDpYVZhJklRWpwSye\nOYqYyBC2HKigvavP7HKASzNuFtPW2cdnF431m8thCnG5wRJqHrAZQGu9DyjotywPaAQeVkrtBOK1\n1hqYCkQqpf6ilNqulJrt/rK9S0+fnXcOVxMVHsyC/HSzy/EJ4aHBrJqTTVePnb/srzC7HADeeu8c\nJRXNTB+XxOKZcry+8F+DBX8s0Nrvtt21+QcgCZgL/BxYAixWSi0COoDHtNbLgQeBF/s9xy/tKaqh\nvauPW2eMIixUDvm7VrdMzyAuKpRtB6to7ew1tRZd0cTre8+SGBvG+lUTZLu+8GuDXXO3Fei/vmvV\nWl+aZasROOMa5aOU2oxzjeBnwBkArfVppVQjMBLnPoCrSk72zdVqu93BtkNVhAZb+eyy8cRf5eLp\nvtrftRpqf3cvVTy98TjvFtXyN6snubmqa9PS3sMzb32IxWLhe/fNIidrxMeWy+/Ot/l7f0MxWPDv\nBVYDG5RSc4CifsvKgGilVK5rh+8C4FlgPc6dwV9XSqXjXGsYdGau+vq2IZRvvv0f1lF3sZNF0zPo\n6+6lvvuTI9fk5Bif7e9a3Eh/M8eOICEmjD/vKWPh5FTioq/8xekpDsPgZxuKuNjazZ235JIUHfKx\nXuR359sCob+hGGwTzGtAt1JqL84du99RSn1eKfWA1roX+DLwklJqP1Chtd4E/AaIVUrtAl4G1vdb\nS/ArhmGwaV8FFmDZLJmMbShCgoNYPXc0vTYHf36/fNjff8v+So6XNTI5Z4ScdCcCxoAjfq21ATx0\n2d2n+i3fAcy+7Dk24F53FejNSiqaKa9to0Alk5oQaXY5Pmt+/kje/qCcnUerWTE7ixGx4cPyvqXV\nLfzp3VLiokO5/7aJWGW7vggQfr3T1dM27bs0GVu2yZX4tuAgK6vnjcZmN3hrmEb9Hd19/Or1YhwO\ng6+snkRsVGBfM0EEFgn+Iaq80M6JsouozHjGpMtFOW7U3MlppCZEsPvYeRqauzz6XoZh8PzbJTS2\ndrN63mgmyPQaIsBI8A/RpcnYVs6R7cLuEGS1cvv8HOwOgzfeO+fR93rncDWHTtWjMuO5fZ73nDUs\nxHCR4B+CxpZu9n9YR0ZSFFPGJJpdjt+YPSGV9KQo3jteS11Tp0feo6KujT+8c5roiBC+cvskrFbZ\nri8CjwT/EGw9WIndYbBidpac6ONGVquFNfNzcBgGb+w56/bX7+qx8cuNJ7DZDe6/bSIJVznnQgh/\nJ8F/nTq6+3j32HkSYsKYPTHV7HL8zkyVzKjkaD4oruN8Q4fbXtcwDP5vi6auqYsVs7PIz5U1NRG4\nJPiv084j1fT02llakElwkPzvczerxcKnF+RgAK+7cdS/53gNHxTXMSY9ljsWjnHb6wrhiyS5rkOf\nzc7Wg1VEhAVx8zSZjM1Tpo1LYnRaDAdKLlB5of2GX6+6oYMXt5wiIiyYB2+fJF/YIuDJX8B1eL+4\njtaOXm6ZnkFE2GCzXYihslgsrF3gHJVv3F12Q6/V02fnV6+foNfm4G9WjScpPsIdJQrh0yT4r5HD\nNT1DkNXCkpkyPYOnTRkzgtyMWI6cbuBcbevgT7iK3287TXV9B7fOyGCmSnFjhUL4Lgn+a3T0dAN1\nFzu5aXKaHA0yDCwWC5/+66h/aNv6952sY9ex82SlRHP3rWPdWZ4QPk2C/xpdOmFrxSw5YWu4TMhO\nQGXGU1TaSGl1y+BP6KeuqZMXNpcQFhLEg2snExIs10kQ4hIJ/mtwuqqZM9UtTBubRHpSlNnlBAzn\ntn7nmbXXs62/z+bgVxuL6e61c98KRdoImUBPiP4k+K/BX0f7Mm3vsFNZCUwcnUDxuSZOVTZf03M2\n7DxDeV0b8/NHctOkNA9XKITvkeAfRE1jB0dON5CbHsu4UXFmlxOQLm3rf21XGYZhDPjYw6fq2Xaw\nipGJkdyzJG84yhPC50jwD+Kj0X62TM9gktyMOPJzE9GVzXxY3nTVxzW2dPPc2x8SEmzlobWT5frH\nQlyFBP8Amtt7eL+4ltQRkUwfl2R2OQHt0rb+13ZfedRvszv49RvFdHTb+MKScYxKjh7uEoXwGRL8\nA9h2sAqb3WD5rEyZxdFko9NimT4uidLqVo6XXfzE8o27z3KmuoVZE1JYOFXOqhZiIBL8V9HVY2PH\nkWpiI0OYN1l2EHqD/mfz9h/1nyhr5O0PykmJj+BLK8bLJjkhBiHBfxW7jp2nq8fGkoJMOQbcS2Sm\nRFM4PoVztW0cPd0AODfHPfPWSYKsFh5cO0mm0hDiGkjwX4HN7mDLgUrCQoJYNCPD7HJEP2vm52Cx\nwGu7z2J3OHj6jWLaOvv47K1jGZ0ml8AU4lpI8F/BvpN1NLX1sHBqOlHhIWaXI/pJT4pizsRUqurb\nefzlo5RUNDN9XBJLZo4yuzQhfIYE/2UMw2Dz/gqsFgtLCyVMvNHt83OwWiyUVDQzIjaM9asmyHZ9\nIa6DBP9ljpddpLq+g1kTU0iKkyl8vVFqQiQ3T0snOMjCg7dPJjpC1sqEuB6yJ+wym/eVAzIZm7f7\nwtJxrF2QQ0xkqNmlCOFzZMTfz9maVkoqmpmUM4Ks1BizyxEDCLJaJfSFGCIJ/n42uaZnWCmTsQkh\n/JgEv8uFpk4O6Qtkp8YwITvB7HKEEMJjJPhd/nKgEsNwTr0sR4gIIfzZgDt3lVJW4CkgH+gB7tda\nl/ZbXgg8DliAauA+rXWva1kKcAhYrLU+5Zny3aO1s5c9RTUkxYVTMD7Z7HKEEMKjBhvxrwVCtdZz\ngUdwhjwASikL8DSwTmu9ANgO5LiWhQC/Bjo8UbS7vXOoij6bg+WzsgiyykqQEMK/DZZy84DNAFrr\nfUBBv2V5QCPwsFJqJxCvtdauZY8BvwRq3FqtB/T02nnncDVR4cHMnzLS7HKEEMLjBgv+WKC13227\na/MPQBLbxQzDAAAL20lEQVQwF/g5sARYrJRapJRaB9Rrrbe4HufVG8z3HK+hvauPxTNHyYU7hBAB\nYbATuFqB/ge0W7XWDte/G4Ezl0b5SqnNONcIbgMMpdQSYBrwglJqjda6bqA3Sk4e/uPm7XYH2w5V\nERps5bPLxhMXHeax9zKjv+Hkz/35c28g/QWiwYJ/L7Aa2KCUmgMU9VtWBkQrpXJdO3wXAM9qrR+7\n9ACl1A7gq4OFPkB9fdt1F3+j9n9YR93FThZNz6C3q5f6rl6PvE9ycowp/Q0Xf+7Pn3sD6c/XDfVL\nbbDgfw1YqpTa67q9Xin1eSBaa/2MUurLwEuuHb17tdabhlSFCQzDYNO+CiwWWDYr0+xyhBBi2AwY\n/FprA3josrtP9Vu+A5g9wPMX3VB1HnRI11Ne20aBSiY1IdLscoQQYtgE5LGLnd19vLj1FMFBVu64\nOdfscoQQYlgFZPC/sqOUlo5e1swfTdoIGe0LIQJLwAV/SXkTu46dZ1RyNMtl6mUhRAAKqODv7bPz\nwuYSLBZYv2o8wUEB1b4QQgABFvxvvneOuqYulhZkkjNSLswthAhMARP8FXVtbPqggqS4cD69YIzZ\n5QghhGkCIvjtDgfPbyrBYRjct0LJ1AxCiIAWEMG/7WAV52rbuGlSGpNzEs0uRwghTOX3wX+huYvX\ndpURHRHC5xaPNbscIYQwnV8Hv2EY/N/mEnptDr6wZJxcnFsIIfDz4H/vRC3F55qYMiaR2RNTzS5H\nCCG8gt8Gf2tHLy9vP01YSBD3Ls+T6+gKIYSL3wb/y9tP09Ft446bx5AUF2F2OUII4TX8MviLShv4\n4GQdY9JjWTxjlNnlCCGEV/G74O/qsfHbv2iCrBbWrRyP1SqbeIQQoj+/C/7XdpVxsbWHVXOyGZUc\nbXY5Qgjhdfwq+EurW9h+qIqRiZHcNne02eUIIYRX8pvgt9md0zIYwJdWjCck2G9aE0IIt/KbdHz7\ng3KqGzpYND2DvMx4s8sRQgiv5RfBf76hg7feO0dCTBifkUspCiHEgHw++B2GwfObS7DZDb64LI/I\n8AGvHy+EEAHP54P/3SPVnKlqoWB8CtPHJZtdjhBCeD2fDv6Lrd1s2FlKZFgw9ywZZ3Y5QgjhE3w2\n+A3D4HdbTtHda+fuW8cSFx1mdklCCOETfDb4D+p6jp5pYEJ2AvPzR5pdjhBC+AyfDP72rj5e3KIJ\nCbZy3wolM28KIcR18Mngf2XHGVo7+1gzP4fUhEizyxFCCJ/ic8F/8txF9hTVkJUSzbLCTLPLEUII\nn+NTwd/TZ+e3mzUWC6xbNZ7gIJ8qXwghvMKAZzsppazAU0A+0APcr7Uu7be8EHgcsADVwH2AHXgG\nyAMM4EGtdbE7in1jz1kuNHexYlYWo9Ni3fGSQggRcAYbMq8FQrXWc4FHcIY8AEopC/A0sE5rvQDY\nDuQAtwEOrfV84J+B/3BHoeW1bfxlfyXJ8eGsWZDjjpcUQoiANFjwzwM2A2it9wEF/ZblAY3Aw0qp\nnUC8dnod+KrrMaOBphst0u5w8NymD3EYBvetGE9YSNCNvqQQQgSswYI/Fmjtd9vu2vwDkATMBX4O\nLAEWK6UWAWit7Uqp54EngZdutMgtByqpqGtn3pQ0Jo0ecaMvJ4QQAW2w4G8FYvo/XmvtcP27ETjj\nGuXbcK4Z/HWNQGu9DudawTNKqSFf7byuqZONu88SGxnC3bfKtAxCCHGjBpvKci+wGtiglJoDFPVb\nVgZEK6VyXTt8FwDPKqXuBUZprf8L6AIcrp8BJSfHfOI+wzB44o9F9NkcfOdzM8jJ8t3R/pX68yf+\n3J8/9wbSXyCyGIZx1YWuHbiXjuoBWA/MBKK11s+4Nu08ivOonr1a6++4RvfPA2lACPBfWus3B6nD\nqK9v+8Sdu4vO89zbJUzNTeSbd+b77Bm6yckxXKk/f+HP/flzbyD9+brk5JghheKAI36ttQE8dNnd\np/ot3wHMvuw5XcDdQymmv5b2Hl555wxhoUHcu1ymZRBCCHfx2jOgXtp2mo5uG3fenMuI2HCzyxFC\nCL/hlcF/5HQ9B0oukJsRy6IZGWaXI4QQfsXrgr+rx8bvtpwiyGph3coJWGUTjxBCuJXXBf8f3y2l\nqa2H2+aOJiMpyuxyhBDC73hV8J+uambH4WrSk6JYNSfb7HKEEMIveU3wG4bB85tKsADrVo4nJNhr\nShNCCL/iNelqdxjYHQarbspmbEac2eUIIYTfGuzM3WETHGTl0a/eZHYZQgjh97xmxC+EEGJ4SPAL\nIUSAkeAXQogAI8EvhBABRoJfCCECjAS/EEIEGAl+IYQIMBL8QggRYCT4hRAiwEjwCyFEgJHgF0KI\nACPBL4QQAUaCXwghAowEvxBCBBgJfiGECDAS/EIIEWAk+IUQIsBI8AshRICR4BdCiAAjwS+EEAFG\ngl8IIQJM8EALlVJW4CkgH+gB7tdal/ZbXgg8DliAauA+wAD+F8gGwoAfaq3f9Ej1QgghrttgI/61\nQKjWei7wCM6QB0ApZQGeBtZprRcA24Ec4B6gXmu9EFgB/MIThQshhBiawYJ/HrAZQGu9DyjotywP\naAQeVkrtBOK11hrYAHy/3+vb3FmwEEKIGzNY8McCrf1u212bfwCSgLnAz4ElwGKl1CKtdYfWul0p\nFYPzS+D/ubtoIYQQQzdY8LcCMf0fr7V2uP7dCJzRTjacawYFAEqpTOAd4Lda65fdXLMQQogbMODO\nXWAvsBrYoJSaAxT1W1YGRCulcl07fBcAzyqlUoEtwNe01juusQ5LcnLM4I/yYdKf7/Ln3kD6C0QW\nwzCuutC1A/fSUT0A64GZQLTW+hml1CLgUZxH9ezVWn9HKfUz4C5A93uplVrrbk80IIQQ4voMGPxC\nCCH8j5zAJYQQAUaCXwghAowEvxBCBBgJfiGECDCDHc7pUYPNBeSLlFIhXDZXEfAh8DzgAE4AX9da\n++xedaVUCnAIWIyzp+fxn97+EechzCE4pxvZi5/05/p7exbnWfcO4AHAjo/3p5SaDTyqtV6klBrL\nFfpRSj0AfAXnTAI/1Fr/2bSCr9Nl/U0DnsT5e+sB7tNaX7je/swe8V91LiAfdvlcRf+Ds69/ct1n\nAdaYWN8NcX2x/RrowNnLT/Gf3m4BbnJ9Hm8BxuBHvztgGRCltZ4P/Bvwn/h4f0qp7wLP4BxkwRU+\nj0qpNOAbOGcaWA78l1Iq1Ix6r9cV+nsC+Fut9SLgVeB7rnOnrqs/s4N/oLmAfNXlcxX1ATO01rtc\n923COcWFr3oM+CVQ47rtT70tA44rpTYCbwJvADP9qL8uIM51fk4c0Ivv93cGuANnyMOVP4+FOM8z\n6tNat7qek/+JV/JOl/f3Oa31pRNpQ3D+Tmdxnf2ZHfwDzQXkk64wV9E/8/H/z+04/+h8jlJqHc61\nmS2uuyx89IEEH+7NJRnnCYp3Ag8CL+Ff/e0FwoESnGttT+Lj/WmtX+XjE0H276cNZz+xQMsV7vd6\nl/enta4FUErNBb4O/DdD6M/skB1oLiCfddlcRb/Hub3xkhig2ZTCbtx6YKlSagcwDXgBZ1he4su9\nATQAW7TWNq31KaCbj/8B+Xp/38U5MlQ4f3+/xTlqvMTX+4OP/63F4uzn8pyJAZqGsyh3UkrdjXOt\ne5XWupEh9Gd28O8FVgFcYS4gn9RvrqLvaq2fd919RCl1s+vfK4FdV3qut9Na36y1vsW1ffEozgvv\nbPaH3lz24Nwvg1IqHYgEtvtRf1F8tIbdhPPgDr/4bPZzpX72AwuUUmFKqThgAs4dvz5HKfVFnCP9\nW7TW51x3X3d/ph7VA7yGcwS513V7vZnFuMk/4Rwlfl8pdWlb/7eAJ107XE4CfzSrODczgL8DnvGH\n3rTWf1ZKLVRK7cc5KPoacA4/6Q/n/pnnlFK7cY70/xHn0Vn+0N+lI5E+8Xl0HdXzJLAb5+/1n7TW\nvSbVOVSGazP4z4By4FWlFMBOrfUPrrc/matHCCECjNmbeoQQQgwzCX4hhAgwEvxCCBFgJPiFECLA\nSPALIUSAkeAXQogAI8EvhBABRoJfCCECzP8HOjw+KzH9K2gAAAAASUVORK5CYII=\n",
       "text": [
        "<matplotlib.figure.Figure at 0x11937d650>"
       ]
      }
     ],
     "prompt_number": 206
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "All labels combined"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "It would be interesting to combine all different label categories into a single task. One reason to do so is to investigate the possibility that the presence of a label from a particular category $A$ increases the likelihood of a label from category $B$."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "all_labels = list(read_labels(\"data/all_labels.txt\"))\n",
      "all_labels, texts = zip(*match_labels_documents(documents, all_labels))\n",
      "all_labels, texts = np.array(all_labels), np.array(texts)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 7
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = small_doc_experiment(texts, all_labels)\n",
      "score_df['combined']['small_doc'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.41 (+/- 0.02)\n"
       ]
      }
     ],
     "prompt_number": 177
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = big_doc_experiment(texts, all_labels)\n",
      "score_df['combined']['big_doc'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.36 (+/- 0.02)\n"
       ]
      }
     ],
     "prompt_number": 178
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = ml_experiment(texts, all_labels)\n",
      "score_df['combined']['ml'] = scores.mean()\n",
      "print 'Average MAP over 10 folds: %.2f (+/- %.2f)' % (scores.mean(), scores.std())"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Average MAP over 10 folds: 0.58 (+/- 0.01)\n"
       ]
      }
     ],
     "prompt_number": 179
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores = np.zeros(11)\n",
      "for i, k in enumerate(range(1, 102, 10)):\n",
      "    scores[i] = ml_experiment(texts, all_labels, num_neighbors=k).mean()\n",
      "sb.plt.plot(range(1, 102, 10), scores)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 208,
       "text": [
        "[<matplotlib.lines.Line2D at 0x119515590>]"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAECCAYAAAD5OrxGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHxRJREFUeJzt3XtwXGed5vFvt+7W3XY7vtvYiX+2sU1iOyHYTibJkHDJ\nBsIWs7OpAWYCuWyALSq1tUyS2aIKCnZmlgo74RImk2QgVM2EqswSFiaMlw0EMF5iyFW+6ZXji+LY\nst2Sdb+r++wfp2W1FUUtyS119znPp0olnXP6SL9Xl6dfveft90Q8z0NERMIjmusCRERkbin4RURC\nRsEvIhIyCn4RkZBR8IuIhIyCX0QkZIonO2hmUeBRYAswCNzlnDuadvx+4DNAPLXrHuAo8ASwDkgC\ndzvnXPZLFxGRmcjU478dKHXO7QAeAB4ed3wr8Enn3I2ptyPALUClc24X8BXga9kuWkREZi5T8O8E\ndgM45/YB28cd3wY8ZGZ7zOyB1L5+oNbMIkAtMJTFekVE5BJNOtQD1ABdadsJM4s655Kp7aeB7wDd\nwLNmdiv+E0U50AgsAG7LbskiInIpMvX4u4Dq9MenhT7AI8658865YeA54CrgL4G9zjkDrgSeMrPS\nbBYtIiIzl6nHvxe/x/6MmV0LNIweMLNaoMHMNgJ9wE3Ak8B1jP2X0A6UAEWTfRHP87xIJDKjBoiI\nhNiMgjMy2SJtqXH60Vk9AHfij+tXOeceN7M7gPvxZ/w875z7spnVAd8DFuKH/t85536YoQ4vHu+e\nSf0FIRarRu0rTEFuG6h9hS4Wq85+8M8hBX8BC3L7gtw2UPsK3UyDXy/gEhEJGQW/iEjIKPhFREIm\n06weEZGCkfQ8OroHiXf009o5QO/QKZpPd9LRM8gH37uSzWsW5LrEvKDgF8lDI4kk3X3DeJ6H54Hn\neSTh4m3v4m3P84MvOW7fRI8d3a4510tHRx9JD8pKolTNK6GqooTqilJKS6Lk4zTr3oFhWjsGiHf0\nE+/sJ94xQGtHP/GOftq6BhhJvH3CSjQS4dp3X5aDavOTgl8kj7x5tps9DS28ePAMvQMjOa2lpDia\nehIouegJ4cLHqff+x6VUVRRTUjzpS3amZHgkQWvnAK2dfrinh3xrxwB9gxN/X6oqSlixqIqFtRXE\n6ipYWFfOFavmU4LH/Jpyios0sj1KwS+SY30Dw7x46Cx7Xm+h+aw/9bCmspSr1y+iqChChAjRCEQi\nESKp9+O3IxG/VzvhNqntaORt51ZXldPXOwiRCEPDCXr6h+nuG6anf5ie/iG6+4Y519HPm+d6ptSW\nstIiqitKqHzbE0YJVfNK/fept77BET/QU8MyrR39xDsH6OgeZKJJ5qXFURbWVXD58lpidRXEastZ\nWJcK+dpyKsreHmdBn845Uwp+kRxIeh6uuZ09DS283BRneCRJNBLhyssXct2WJWxeu2BOeqhTDcbh\nkWTqyWCYnr4hui98PJz28dj+lrZemoeTGT9vukgE5leXYyvrUr32sWCP1ZZTU1mal0NPhUjBLzKH\nzncN8Nv9Lfy2oYXWzgEALps/j+u2LGHHpsXUVZXluMKJlRRHqa8uo7566vUNDifoTfsPort/iJ4L\nHw9TXlqUCnU/5DUcM3cU/CKzbHgkyWtvtLLn9dMcPH4eDygtibJz82Ku27KUK5bXBrInW1ZSRFlJ\nEfNrynNdioyj4BeZJW+d6+E3Dad58eBZevqHAVi7rIbrtizl6vWLJhyTFpkL+s0TyaK+gRH2HT7L\nbxtOc7zFHzuvnlfCB65Zwa4tS1m2sDLHFYoo+EUumed5uDc72NNwmpecf6E2EoEtaxdw3ZalvOfy\nublQKzJVCn6RGWrvHuS3+1vY29DCuY5+ABbVV6Qu1C6Z1oVQkbmk4BeZhpFEkteOtLKnoYUDx9vw\nPH9++Y5Ni7luyxLWragL5IVaCRYFv0gGSc+j+Uw3+w6d5f8dOHPhQu2apTXs2rKE9264TBdqpaDo\nt1VkAt19Qxw8fp6m0028fPgs3X1+2FdVlHDL1SvYtWUJy2NVOa5SZGYU/CL4vfoTLd3sP9bG/mNt\nHD/ddWHZgNqqUnZtXsJ7Ll+oC7USCAp+Ca2uviEOHjvP/uNtHDh2/sIQTjQS4YrltWxeu4Drt62k\nqiSicXsJFAW/hEYy6XG8petCr/5ES/eFXn1dVam/Rs6aBWxcPZ955f6fhhb5kiBS8EugdfUOceB4\nG/uPnefg8bFefVE0wroVdWxeu4DNaxawPFapXr2EhoJfAiWZ9Dh2uouGVK+++cxYb72+uozr3zPW\nq9dMHAkr/eZLwevsHeJAKugPHj9/4QYmRdEI61eO9eqXLVSvXgQU/JKnkkmPweEEA0MJhlLvB4dT\nb0P+9rmOPvYfPX/h5iUA82vK2L5+EZvXLGDDqnr16kUmoL8KyYqk59HaOcDA4MhF4Tw+rNO3B98h\n0IeGEwyNTO0mHkXRCBtW1bN5zQI2r5nPUvXqRTJS8MuMjSSSNDa3c+jXx/hdw2k6e4dm9HmKiyL+\n2u2lRVTPK6G8tPzCWu5lpUWUlxZRWuK/T99fW1nKFcvr1KsXmSb9xci0DAyNsP/YeV5pitNwtJX+\nwQTgv6L1mg2LqJ5X+raAfsft1Hu9IEpkbin4JaOuviFeO9LKK01xDp1oZyThD8MsqCln1+al3HTN\nShZWlVAUVYCLFAIFv0wo3tHPq01xXmmKc+RUJ17qlU7LY5VsXRdj67oYKxZVEYlE9CInkQIzafCb\nWRR4FNgCDAJ3OeeOph2/H/gMEE/tusc5d8TMHgRuA0qAbzvnnpqN4iV7PM/jrXgvr6TC/uS5HgAi\nwNrltWy9IsbWdQtZVD8vt4WKyCXL1OO/HSh1zu0ws/cCD6f2jdoKfNI59+roDjO7AXhf6pxK4ItZ\nrlmyJJn0eONUJ680xXn1SJx4xwDgX2zdvGYBW9ct5MorYtRWlua4UhHJpkzBvxPYDeCc22dm28cd\n3wY8ZGaLgeecc38DfADYb2Y/BmqA/5rlmuUSDI8kONzczitNcV470kpXarnh8tIirtmwiK3rYmxe\ns0AzZUQCLNNfdw3QlbadMLOoc250kvXTwHeAbuBZM7sVWACsBP4dsAb4CbA+q1XLtPQNjLD/WJs/\nE+dYG4ND/kycmnklXP+epWxdF2PDqnpKinVxViQMMgV/F1Cdtp0e+gCPOOe6AMzsOeAqoA1odM6N\nAE1mNmBmC51zrdksXCbX1TvEK0f88frDJ9pJJP2rs7G6crZe6Yf92qW1RKN6sZNI2GQK/r34F2mf\nMbNrgYbRA2ZWCzSY2UagD7gJeBIoAr4AfMPMlgKV+E8Gk4rFqjM9pKDNRfvauwb43YEW9r5+mgNH\nW0llPWuW1XLtpiW8b/MSVi2unpVXtgb55xfktoHaF0YRb3Se3gTMLMLYrB6AO/HH9aucc4+b2R3A\n/fgzfp53zn05dd7fAjcCUeBB59z/zVCHF+TpgLM53bGzZ5CXXJyXGs/RdLLjwvrya5fVcLX5Y/YL\n6ypm5WuPCvJ0ziC3DdS+QheLVc+oFzdp8M8hBf80dPQM8rKL84fGcxxJC/vLl9dytS1im8WYX1Oe\nta+XSZD/uILcNlD7Ct1Mg19TNwpEe/cgL7tzvNR4jiNvdeLhz7G/fHkt29cvYrstor66LNdlikgB\nUPDnsfbuQV5qPMcf3DmOpoX9FSvq2G4xtinsRWQGFPx55nzXwIUx+zdOdQJ+2K9bUcf29f4wTl2V\nwl5EZk7BnwfOdw2M9exP+S+biERg/cpU2K+LUauwF5EsUfDnSGtnPy81xnnJnePY6bGw37Cqnu3r\n/dk4WipBRGaDgn8OjYb9HxrPcbzl4rC/OhX2NQp7EZllCv458vvDZ3nsJwfxPIhGImxcPdazr5mn\nsBeRuaPgnwMnz3bzvZ81UlpSxJ/edDnb1sWoVtiLSI4o+GfZwNAIf/1PrzA4nOC+2zdx9fpFuS5J\nREJOyzHOIs/z+MH/cZw828P7ty1X6ItIXlDwz6Jfv3aaFw+exVbW8x9uujzX5YiIABrqmTUnznTx\nz883UVVRwhc/tZ3ISCLXJYmIAOrxz4regWEeffYAiYTH3bdt1H1qRSSvKPizLOl5PPHTQ7R2DnDb\nztVsXrMg1yWJiFxEwZ9lu/e9yetH29i4up6P7HxXrssREXkbBX8WuTfb+V+/PkpdVSn33PZu3dZQ\nRPKSgj9LOnsG+fv/fZAIEe67fZOWXhCRvKXgz4JEMsljPzlIZ+8Qf3LjWq5YXpfrkkRE3pGCPwt+\nvOc4jW92sHVdjFuuXpHrckREJqXgv0SvvdHKc79rZlFdBZ/+8AYiEY3ri0h+U/BfgnhHP0/89BAl\nxVE++7FNzCvX6+FEJP8p+GdoeCTJd398gL7BET5x8zpWXlad65JERKZEwT9DP/zlEU6c6Wbn5sVc\n956luS5HRGTKFPwz8OLBM7zwyimWxyr5xC2W63JERKZFwT9Np1p7eWq3o7y0iM9+bDNlJUW5LklE\nZFoU/NMwMDTCo8/uZ3A4wac/vIHF87X4mogUHgX/FHmex1O7HS1tfdy8fQXbdVMVESlQCv4p+tWr\np9h36Cxrl9XwJzeuzXU5IiIzpuCfguMtXTz9iyNUVZRw30c3UVykb5uIFC4lWAY9/WM3VbnnIxuZ\nX1Oe65JERC7JpC81NbMo8CiwBRgE7nLOHU07fj/wGSCe2nWvc64pdWwR8DLwx6P7Ck3S83jyXw/R\n1jXAR3auZtO7dFMVESl8mdYYuB0odc7tMLP3Ag+n9o3aCnzSOfdq+klmVgI8BvRms9i59m8vNvP6\n0TberZuqiEiAZBrq2QnsBnDO7QO2jzu+DXjIzPaY2QNp+78OfBdoyVahc62xuZ0f/eYY9dVl3P0R\n3VRFRIIjU/DXAF1p24nU8M+op4F7gZuAXWZ2q5n9BRB3zv089ZiCS8yOnkH+/icHiUYi3PfRTdTM\n001VRCQ4Mg31dAHpq49FnXPJtO1HnHNdAGb2HHAVcDPgmdn7gSuBp8zso865s5N9oVgsPxY5SySS\nfOOZ1+nqHeKuj27ifVctz8rnzZf2zZYgty/IbQO1L4wyBf9e4DbgGTO7FmgYPWBmtUCDmW0E+vB7\n/U86576a9pgX8C/4Thr6APF49wzKz75/+dVRDhxtY5vFeN/6WFbqisWq86Z9syHI7Qty20DtK3Qz\nfVLLFPzPAjeb2d7U9p1mdgdQ5Zx7PDWu/wL+jJ/nnXO7Z1RFnnjtSCs/e7GZRfUV3Pkh3VRFRIJp\n0uB3znnAfeN2N6Udfxp/nP+dzr/xkqqbQ/GOfp7419RNVW7XTVVEJLj0Ai5geCTBo8+mbqpyi26q\nIiLBpuAHnv7FGzSf7WbXliVct0U3VRGRYAt98P/u4Bl+9eoplseq+MTN63JdjojIrAt18Ps3VWmk\noqyIz31sE6W6qYqIhEBog39wOMGjz+5naDjJpz+8gct0UxURCYnQBn/D0TZa2vq48aplbDPdVEVE\nwiO0wX/oxHkAdmxenONKRETmVqiDv6KsmNWLNXVTRMIllMF/rqOfeMcA61fWURQN5bdAREIslKl3\nODXMs3H1/BxXIiIy90IZ/IdOtAOwcXV9jisREZl7oQv+pOdxuLmd+uoyFmsKp4iEUOiC/+TZHnr6\nh9m4ul6rb4pIKIUu+A81a3xfRMItfME/Or6/SuP7IhJOoQr+4ZEER052sCxWSW1VWa7LERHJiVAF\n/xunuhgaSbJxlYZ5RCS8QhX8hy7M39cwj4iEV8iCv52iaIR1K+pyXYqISM6EJvh7B4Y5caaLNUtr\nqCjT/XRFJLxCE/yNzR14nqZxioiEJvjH5u9rfF9Ewi08wX+inbLSIt61pCbXpYiI5FQogr+tc4Cz\n5/tYv6KO4qJQNFlE5B2FIgW1TIOIyJhQBP9hLcMsInJB4IPf8zwOnThPbWUpSxdW5rocEZGcC3zw\nn4r30tWnZZhFREYFPvgPNY8O82h8X0QEYNKXsJpZFHgU2AIMAnc5546mHb8f+AwQT+26BzgB/COw\nCigDvuqc+2nWK5+i0fV5NmgZZhERIHOP/3ag1Dm3A3gAeHjc8a3AJ51zN6bejgB/BsSdc9cDHwS+\nne2ip2okkcS92cGSBfOYX1OeqzJERPJKpuDfCewGcM7tA7aPO74NeMjM9pjZA6l9zwBfSvv8I1mq\nddqOne5icDih3r6ISJpMwV8DdKVtJ1LDP6OeBu4FbgJ2mdmtzrle51yPmVXjPwn8VVYrnoaxZZg1\nvi8iMirTMpVdQHXadtQ5l0zbfsQ51wVgZs8BVwHPmdkK4EfAd5xzP5xKIbFYdeYHTdORU11EI7Bz\n6wqqKkqy/vmnYzbal0+C3L4gtw3UvjDKFPx7gduAZ8zsWqBh9ICZ1QINZrYR6MPv9T9pZpcBPwc+\n65x7YaqFxOPd0619Uv2DI7jmdlYvqaG/Z4D+noGsfv7piMWqs96+fBLk9gW5baD2FbqZPqllCv5n\ngZvNbG9q+04zuwOocs49nhrXfwF/xs/zzrndZvYIUAt8ycxGx/o/5Jyb0+R1JztIep5erSsiMs6k\nwe+c84D7xu1uSjv+NP44f/o5XwC+kK0CZ+rC+L7urysicpHAvoDr8Il2SoujrF1Wm+tSRETySiCD\nv6NnkFOtvaxbUUdJcSCbKCIyY4FMxbHVODXMIyIyXiCDf2z+vi7sioiMF7jg9zyPQ83tVFWUsHxR\nVa7LERHJO4EL/jPn+2jvHmTj6nqiWoZZRORtAhf8hzS+LyIyqQAG/+j8fY3vi4hMJFDBn0gmaXyz\nnUV1FSysq8h1OSIieSlQwX+ipZv+wYRm84iITCJQwa9lmEVEMgtY8LcTAdZrfF9E5B0FJvgHhxK8\ncaqTlYurc772vohIPgtM8De91UEiqWWYRUQyCUzwa3xfRGRqAhT87RQXRblCyzCLiEwqEMHf1TfE\nyXM9XLG8ltKSolyXIyKS1wIR/I3No8s0aHxfRCSTQAS/xvdFRKau4IPf8zwOHm+nsryYVZfN7I7z\nIiJhUvDBH+/op61rgPWr6olGtQyziEgmBR/8WoZZRGR6AhD8WoZZRGQ6Cjr4k0mPw83tLKgpY1G9\nlmEWEZmKgg7+N8910zswwobV84noNosiIlNS0ME/Nr6vYR4Rkakq8OD3x/c3rNKFXRGRqSrY4B8a\nTtB0spPlsSpqK0tzXY6ISMEo2OB/41QnI4mkhnlERKapeLKDZhYFHgW2AIPAXc65o2nH7wc+A8RT\nu+4B3gC++07nZIvm74uIzEymHv/tQKlzbgfwAPDwuONbgU86525MvR0BPpbhnKw4dOI8RdEI61Zo\nGWYRkenIFPw7gd0Azrl9wPZxx7cBD5nZHjN7YIrnXLKe/mGaz3Szdlkt5aWT/tMiIiLjZAr+GqAr\nbTuRGv4Z9TRwL3ATsMvMbp3COZessbkdD03jFBGZiUzd5S4gfcnLqHMumbb9iHOuC8DMngOumsI5\nE4rFpr6y5vFfHwNg55XLp3VeLhVKnTMV5PYFuW2g9oVRpuDfC9wGPGNm1wINowfMrBZoMLONQB9+\nr/9JYN47nTOZeLx7ykW/3HiW8tIi6iqKpnVersRi1QVR50wFuX1BbhuofYVupk9qmYL/WeBmM9ub\n2r7TzO4Aqpxzj6fG9V/An73zvHNut5lFxp8zo8reQWtHP+fa+7ny8oUURQt2NqqISM5MGvzOOQ+4\nb9zuprTjT+OP82c6J2sO6TaLIiKXpOC6zLrNoojIpSmo4E96/jLMdVWlLFkwL9fliIgUpIIK/rfO\n9dDdN8xGLcMsIjJjBRX8WoZZROTSFVbwN2sZZhGRS1UwwT+SSNJ0soOlCyupry7LdTkiIgWrYIL/\n6KlOhoaTuqm6iMglKpjg1zLMIiLZUTjB33yeaCSCrazLdSkiIgWtIIK/b2CE46e7WbO0hooyLcMs\nInIpCiL43cl2kp6naZwiIllQEMGv8X0RkewpkOA/T1lJEWuW1uS6FBGRgpf3wd/ePUhLWx+2so7i\norwvV0Qk7+V9ko6uxrlB8/dFRLKiAIJf4/siItmU18HveR6Hms9TM6+EZbHKXJcjIhIIeR38p9v6\n6OwZYsPq+US1DLOISFbkdfBfuNuWxvdFRLImr4P/sMb3RUSyLm+DfySRpPHNdi6rr2BBbXmuyxER\nCYy8Df4TLd0MDCXU2xcRybK8Df4L4/tan0dEJKvyOvgjwHpd2BURyaq8DP6BoRGOnu5i9ZJqKstL\ncl2OiEig5GXwN53sIJH0NL4vIjIL8jL4LyzToGEeEZGsy9PgP09JcZTLl9fmuhQRkcDJu+Dv7Bnk\nrXgv65bXUlJclOtyREQCZ9Ib2JpZFHgU2AIMAnc5545O8Lh/ANqccw+mznkCWAckgbudc26qBR1u\n1qt1RURmU6Ye/+1AqXNuB/AA8PD4B5jZvcAmwEvtugWodM7tAr4CfG06BWkZZhGR2ZUp+HcCuwGc\nc/uA7ekHzWwHcA3wGDC6fGY/UGtmEaAWGJpqMaPLMFeWF7PisqqpniYiItOQKfhrgK607URqKAcz\nWwJ8Cfg8Y6EPsBcoBxrxnxC+NdVizrX3c75rUMswi4jMoknH+PFDvzptO+qcS6Y+/jiwEPgZsBiY\nZ2aNwDJgr3Pur8xsOfBLM9vknMvY89cyDSIisy9T8O8FbgOeMbNrgYbRA865b5HqzZvZnwPmnHvK\nzL7G2H8J7UAJkHF6TixWzRst3QBct3UFsQXBuuNWLFad+UEFLMjtC3LbQO0Lo0zB/yxws5ntTW3f\naWZ3AFXOucff4ZyvA98zsz34of+gc64/UyFnz3bxelOchbXlFCWTxOPdU21D3ovFqgPVnvGC3L4g\ntw3UvkI30ye1SYPfOecB943b3TTB455K+7gD+Nh0CxkcTjA4nGDn5iXTPVVERKYhU49/zlSUFfM/\n7ttBVYUWZRMRmU15E/wA9dVluS5BRCTw8m7JBhERmV0KfhGRkFHwi4iEjIJfRCRkFPwiIiGj4BcR\nCRkFv4hIyCj4RURCRsEvIhIyCn4RkZBR8IuIhIyCX0QkZBT8IiIho+AXEQkZBb+ISMgo+EVEQkbB\nLyISMgp+EZGQUfCLiISMgl9EJGQU/CIiIaPgFxEJGQW/iEjIKPhFREJGwS8iEjIKfhGRkFHwi4iE\nTPFkB80sCjwKbAEGgbucc0cneNw/AG3OuQdT2w8CtwElwLedc09lu3AREZmZTD3+24FS59wO4AHg\n4fEPMLN7gU2Al9q+AXhf6pwbgDVZrFdERC5RpuDfCewGcM7tA7anHzSzHcA1wGNAJLX7A8B+M/sx\n8FPgJ9ksWERELk2m4K8ButK2E6nhH8xsCfAl4POMhT7AQmAb8HHgPwH/lLVqRUTkkk06xo8f+tVp\n21HnXDL18cfxQ/5nwGJgnpk1Aq3AYefcCNBkZgNmttA515rl2kVEZAYyBf9e/Iu0z5jZtUDD6AHn\n3LeAbwGY2Z8D651zT5nZrcAXgG+Y2VKgEmjL8HUisVh1hocUNrWvcAW5baD2hVGm4H8WuNnM9qa2\n7zSzO4Aq59zj4x7rATjnnjOz683s9/hDSZ91znlZrVpERGYs4nnKZBGRMNELuEREQkbBLyISMgp+\nEZGQUfCLiIRMplk9s2qqawEVEjMrAf4RWAWUAV8FDgPfB5LAAeBzhTzTycwWAS8Df4zfpu8TnLZd\ntM4U/pTm7xOA9qX+3p4A1uG3524gQYG3z8zeC/yNc+5GM7ucCdpjZncD9wAjwFedc8/lrOBpGte+\nK4Fv4v/cBoFPOefOTbd9ue7xZ1wLqAD9GRB3zl0PfBD4Dn67HkrtiwAfzWF9lyT1xPYY0Ivflm8Q\nnLbdwNvXmQrMzw64Bah0zu0CvgL8dwq8fWb2ReBx/E4WTPD7aGaLgf8M7MBfUuavzaw0F/VO1wTt\n+zvg8865G4EfAX9pZpcxzfblOvgnXQuoQD2Dv5QF+N/fYWCrc+43qX3/Brw/F4VlydeB7wItqe0g\nte0W3r7O1LYAta8fqDWzCFALDFH47XsD+PeMLRsz0e/j1cBe59ywc64rdc6WOa90Zsa37z8650Zf\nSFuC/zO9hmm2L9fB/45rARUq51yvc67HzKrxnwT+Gxd/n3vw/+gKjpn9Bf5/Mz9P7Ypw8TpNBdu2\nlBgXrzP1zwSrfXuBcqAR/7+2b1Lg7XPO/Qh/eGNUenu68dtTA3ROsD/vjW+fc+4MXFgg83PA/2QG\n7ct1yE62FlDBMrMVwC+BHzjnnsYfbxxVDXTkpLBLdyf+K7lfAK4EnsIPy1GF3Dbw15n6uXNuxDnX\nBAxw8R9Qobfvi/g9Q8P/+f0Av9c4qtDbBxf/rdXgt2d8zlQD7XNZVDaZ2Z/i/9f9YedcGzNoX66D\nfy/wYYDxawEVqtR428+BLzrnvp/a/aqZ/VHq4w8Bv5no3HznnPsj59wNqfHF14BPAbuD0LaU3+Jf\nlyG1ztQ84BcBal8lY/9ht+NP7gjE72aaidrze+A6Myszs1pgA/6F34JjZp/A7+nf4Jw7kdo97fbl\ndFYPE6wFlMtisuQh/F7il8xsdKz/C8A3UxdcDgH/kqvisswD/gvweBDaNtE6U8AJAtI+/Osz3zOz\nPfg9/QfxZ2cFoX2jM5He9vuYmtXzTWAP/s/1IefcUI7qnCkvNQz+CNAM/MjMAH7lnPvydNuntXpE\nREIm10M9IiIyxxT8IiIho+AXEQkZBb+ISMgo+EVEQkbBLyISMgp+EZGQUfCLiITM/wdmREsFrAEE\nSQAAAABJRU5ErkJggg==\n",
       "text": [
        "<matplotlib.figure.Figure at 0x117529890>"
       ]
      }
     ],
     "prompt_number": 208
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Overview"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Finally I give an overview of the results"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "%matplotlib inline"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 183
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import seaborn as sb"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 184
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "score_df.plot(kind='bar', figsize=(14, 6))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 263,
       "text": [
        "<matplotlib.axes._subplots.AxesSubplot at 0x12419ef10>"
       ]
      },
      {
       "metadata": {},
       "output_type": "display_data",
       "png": "iVBORw0KGgoAAAANSUhEUgAAAy8AAAGVCAYAAAD3484KAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xmc1WXd//HXmYV9gBk4uAsucCkqalmimN1mWJamdpuK\npokBiWJZmbsSKRKSK4m5VprLLZVpudatpmJipcjtT71QKE00nWGAQQaGYeb8/pgZHWlgAM8y35nX\n8/Hw4fkuXOdzhpnrMW+u5ZvKZDJIkiRJUkdXVOgCJEmSJGljGF4kSZIkJYLhRZIkSVIiGF4kSZIk\nJYLhRZIkSVIiGF4kSZIkJULJhi6GEIqAWcAIoA4YF2Nc2Or6UcD5QAa4Ncb4sxzWKkmSJKkLa2/k\n5UigW4xxf+Bc4Ip1rl8JjAZGAd8PIfTLfomSJEmS1H54GQU8DBBjnAvss871eqA/0BNI0TQCI0mS\nJElZ11546QvUtDpuaJ5K1uIK4O/AS8DvY4yt75UkSZKkrNngmheagktZq+OiGGMjQAhhe2ASMBio\nBX4VQjg6xvjr9TW2dm1DpqSk+GOWLEmSJKmTS7V1sr3wMgc4HJgdQhgJzG91rQfQANTFGBtDCO/R\nNIVsvZYurd34ctWppdNlVFauKHQZkjoA+wNJLewP1CKdLmvzfHvh5V5gdAhhTvPx2BDCGKBPjPGm\nEMIvgWdCCKuB14FfZKleSZIkSfqIVCaTvzX2lZUrXNAvwH9ZkfQh+wNJLewP1CKdLmtz2pgPqZQk\nSZKUCIYXSZIkSYlgeJEkSZKUCIYXSZIkSYnQ3m5jkiRJkjZJrvaoanMNe5dieJEkSZKy7Kp75lG5\nbHVW2kr378F3j9krK20lneFFkiRJyrLKZav5d/WqvL1fQ0MDZ555GmvXrmXGjGvo06fPeu+tqalh\n7txnGD36i3mrL1tc8yJJkiQlXGVlJbW1tVx//S0bDC4Ar7++gKeffjJPlWWXIy+SJElSwv3kJ5fx\n1ltvMmPGZc1BZiUNDWsZP/40PvGJfTjxxGPYfvvBlJSUUlOznNdff43777+Xl16az8EHH8K+++7H\ns88+w2OP/ZHzz5/McccdxYgRe/Hmm29QXl7B1KmX09jYyIwZl7F48Vs0NjYyfvxE9t77k3n9nIYX\nSZIkKeHOOus8Jk8+n169evPpT+/I0UcfR1VVJRMnjmP27PtYvXo1J588nqFDh/HCC3/nvvt+y1e+\nchQvvTSfVOo/NwJ45523mTnzBtLpQUyc+E1eeeVlFix4lf79yznvvItZvnwZkyZN4Pbb78nr5zS8\nSJIkSQmXyTTtcPbGG//gkEOa1rIMHJimd+/eLF1aDcD22w/+4N6W+9tqA6Bfv/6k04MAGDRoC9as\nqWPRooXMn/8CL7/8EgCNjY3U1Cynb99+uftg6zC8SJIkSVmW7t+jIG0NGbIDL774AkOHBior3+P9\n91d8EC6KipqWuxcXF38QVLp160ZVVSUACxa8+kE7bQzGMHjwYAYNGsSJJ45l5cr3ufvuOygr67u5\nH2uzGF4kSZKkLCvE1sapVIqvf30s06b9iCeeeIy6utWcffYFFBcX0/oZMdtssy2LFr3O7Nl3c9hh\nRzJt2o949NGH2G67wa1b+4+2jzjiv5k+/VImTZpAbe1KvvrVr7U55SyXUm0NGeVKZeWK/L2ZOrR0\nuozKyhWFLkNSB2B/IKmF/YFapNNlbaYit0qWJEmSlAiGF0mSJEmJYHiRJEmSlAiGF0mSJEmJ4G5j\nkiRJUlblao+q/O7s1REZXiRJkqQsu+7Fm6laVZ2Vtgb2rOD0Pcdlpa2kM7xIkiRJWVa1qpr3apcU\nuoz/sGjR66xYsYI999ybyZPP56KLfkRJSXIigWteJEmSpC7i8cf/l3/8YxEAU6ZclqjgAo68SJIk\nSYm3cuX7TJ8+lfffX0FVVSVf/erXGDp0F2bOvJLGxkbS6TTf/e7ZPPTQH+jWrRsh7MJFF53LXXf9\nlqqqSqZN+xGNjY0AnHnmD9h556Ecd9xRjBixF2+++Qbl5RVMnXo5b731L6ZNm0JxcQmZTIbJky9l\n0KAt8vY5DS+SJElSwi1e/BYHH3wIn/3sQVRVVTFp0nh69uzJlCmXsf32Q3jggfuprl7Cl750OAMG\nDGTXXXcjlUqRyWS47rqrOeaY4znggAN57bUF/PjHl3DzzbfxzjtvM3PmDaTTg5g48Zu88srLxPgK\nw4fvwcSJZzB//jzef//9vIYXp41JkiRJCVdeXsFTTz3BJZdcxC9/eQsNDQ1UV1ez/fZDAPjyl7/C\nsGG7tPln33jjn+y11ycAGDp0GO+99y4A/fr1J50eBMCgQVtQX7+Gww47gj59+vD973+b3/zmfygp\nKc79h2vFkRdJkqROK1db9rZw6971GdizIq9t3X33Hey++x4ceeTRPP/833jmmacYODDNW2/9i223\n3Y4777yNbbfdnqKiog+mh7UYPHgH5s17vnnkJTJgwAAAUuv89WYyGZ566s/suefejB07nj/+8WF+\n9atfcv75k7P2WdtjeJEkSerEsrllbwu37m1fvr8+o0Z9hquvnsGTTz7BDjvsSO/evTnrrHOZNu1H\npFIpBg5Mc8wxx1NaWsJ1113L4MFDgBSpVIpJk85k+vRLufvuX7F27VrOPffi5lY/ml5SqRS77LIr\nU6f+kNLSUhobG/n2t7+X18+ZymRyncg/VFm5In9vpg4tnS6jsnJFocuQ1AHYH0i5lGHKs5dnfcve\nQb0GMHnk2WR75MX+QC3S6bI2v7lc8yJJkiQpEQwvkiRJkhLB8CJJkiQpEQwvkiRJkhLB3cYkSZKk\nrMrVHlVuTW14kSRJkrJs8dU/ob6yKittlaYHss2ZZ2WlraTbYHgJIRQBs4ARQB0wLsa4sPnaFsDd\nrW7fCzgnxnhjjmqVJEmSEqG+soo1775b0BrOOONbTJ16OX379vvg3GuvLWDOnCc5+eSNfw5NTU0N\nc+c+w+jRX8xFmZukvZGXI4FuMcb9Qwj7Alc0nyPG+C5wEEAIYT/gEuCmHNYqSZIkaROs+0zHoUOH\nMXTosE1q4/XXF/D0008mIryMAh4GiDHODSHss+4NIYQUcC1wfIzRh1BKkiRJebZ27Vouu2wK77yz\nmIaGRo499ngArr32CiorK+nRowfnn/9DFi16nfvu+y1TplzGY4/9iXvuuZOioiJGjNiLU0+dxNKl\nS5k6dTIrV75PJgMXXjiF2267lYULX+f++++lX79+3HHHbZSUlDBwYJopUy4jlcrfWpz2wktfoKbV\ncUMIoSjG2Njq3OHASzHG17JenSRJkqR23Xffbygvr+Diiy+htraWU045gW7dunHSSWP51KdGcu+9\nv+b223/OAQccCDRNBbv11hu55Zbb6d69O5dccjF//etc5sx5is985r844oiv8tJL83n55f/HN77x\nTX73u9/wla8cxUUXncsJJ5zEZz/7OR5++AFWrlxJnz598vY52wsvNUBZq+N1gwvACcDVG/Nm5eW9\nKCkp3oTy1Jml02Xt3ySpS7A/kHKjoaEhZ21XVPSmuDj7v9d1hv6goaGBf2a5zfa+3u+99zYHHrh/\n89evjGHDhvKXv/yF0aP/i9LSUg44YF+uueZZ+vfvRY8epaxcuYSammWcd953AaitraWmpop3313M\nSScdTzpdxkEHjQJg7ty59OhRSjpdxg9/eBE33HAD9933a3bccUeOOuqwDhVe5tA0sjI7hDASmN/G\nPfvEGP+yMW+2dGntJpanziqdLqOyckWhy5DUAdgfSLmUuxn91dUryfbWvZ2nP8hQmh6YtdZK0wPb\n/XoPGrQNTz75DHvuuS+1tSt59dXIFltsxRNPPMNee32Cxx9/mu2224Fly2pZvbqenj3LGThwEDNm\nzKS4uJg//OE+Bg8extZbR5555jkqKrZm3rznefbZZ9hvv1GsWrWGysoV/PzntzNmzFjKy8uZMeMy\nfvvb33PooYdl7bO2WF+IbS+83AuMDiHMaT4eG0IYA/SJMd4UQkgDy7NXpiRJkpR8+d7a+Igjvsr0\n6Zdy2mnjqKur45RTJvDAA/fz6KMPccstN1BW1pcLL/whMb4KQP/+/TnuuBOYNGk8DQ2NbLXV1owe\n/UVOPPEUpk2bwiOPPERRURHnnnsRpaWlLFr0OrNn382uu+7G2WefSa9evenVqxejRh2Y18+ZWncH\nglyqrFzhgn4BnelfViR9XPYHUi5lmPLs5bxXuySrrQ7qNYDJI8/GkZfk+etfn+WRRx7iwgunFLqU\nDUqny9r85irKdyGSJEmS8u/VV19h1qxrGTXqM4UuZbO1N21MkiRJUiewyy678vOf31noMj4WR14k\nSZIkJYLhRZIkSVIiOG1MkiRJyqpc7VGVvyfZd1SGF0mSJCnLHrjnRWqWrc5KW3379+DLx+yZlbaS\nzvAiSZIkZVnNstUsq85OeMm3F198gT59ythpp5254IIfMHXqjEKX9AHXvEiSJEn6wB/+cB9VVZUA\nHSq4gCMvkiRJUuKtXbuWGTMuY/Hit8hkMnzzm9/i6qtnsNden2DhwtfZfvshVFRU8OKLL1BaWsqM\nGdewatUqLrnkImpra2loWMv48afRu3cfnnvuL7z22gKGDNmB8eO/wf33P8KCBa9y9dU/oaioiG7d\nunPOORfQ2NjID394AVtssSWLF7/Frrvuxllnncv8+fP46U+vprS0lO7de3DppdPp1atXVj6n4UWS\nJEkdQIaGhgZc7L55fv/739G/fznnnXcxy5cv4/TTJ1BXt5pDDjmU3XcfwQknHM0ZZ3yP8eMnMmnS\nBP7xj0U88siDfPrTIzn66OOoqqpk4sRxzJ59H/vuuz+f//wX2GKLLUk1f9mmT5/KeeddzM47D+Xp\np//MzJlXMWnSmbz11ptcffUsunfvzjHHHEF19RKefvrPfP7zh/C1r43h6af/zIoVNYYXSZIkdS63\n/+yZrC1yb9FVFrsvXPg6//d/83j55ZcAaGxsYPny5QwbtgsAffqUMWTIjgCUlfVlzZo1vPnmP/nC\nFw4FYODANL1792bp0uo221+ypIqddx4KwIgRe/Ozn/0UgG222Y6ePXsCMGDAQNasqefEE0/htttu\n5TvfmUg6nWb48N2z9jkNL5IkSeoQkrzIfV19+/fIa1tDhgxhiy224MQTx1Jbu5K77voVDz74e1Kp\n9Y84DR68Ay+++AJDhwYqK99jxYoa+vbtRyqVah4F+9DAgWkWLnydnXbamXnznme77QYDtNF+hkcf\nfZBDDz2M00//Drff/nPuv/9exo4dv8mfuy2GF0mSJCnL8j3ac8QR/8306ZcyadIEamtXctRRX6Oo\nqHi996dScOKJY5k27Uc88cRj1NWt5pxzLqS4uJjhw3fnhhuuY+utt6Flut0551zAVVddTiaToaSk\nhHPPvYhMJtNGeEmx6667MX36pfTo0ZPi4iLOPvuCrH3OVCaTq3mF/6myckX+3kwdWjpdRmXlikKX\nIakDsD+QcinDlGcv573aJVltdaveaS7c9/tkdx1JhrtufC7rIy/9K3owZsK+dPY1L51NOl3W5l+Y\nIy+SJEnaJP2792Px1VdQX1mVtTZ7DtsZ2CNr7alzMrxIkiRpk9VXVrHm3Xez1l7pwAHQO2vNqZPy\nIZWSJEmSEsHwIkmSJCkRnDYmSZIkZZUP2swVw4skSZKUZZULf8XauuVZaaukez/SO309K20lneFF\nkiRJyrK1dcupr2v7afWFtGjR66xYsYI999ybefOep6ysLzvttDMXXPADpk6dUejy2uWaF0mSJKmL\nePzx/+Uf/1gEwAMP3E9VVSVAIoILOPIiSZIkJd6bb77BtGlTKC4uIZPJMHnypfz2t7OZP38ejY2N\nHHvs8eyxx5489NAf6NatG8OGBebO/QuvvbaAIUN2YMKEb3DffY8wadIEhg0LLFq0kJUrV3LJJdPZ\ncsst+cUvbubJJ5+gf/9y6upWM27cqRQXF/PTn15NaWkp3bv34NJLp9OrV6+cfk7DiyRJkpRwf/vb\ncwwfvgcTJ57B/PnzeOqpJ3jnnbeZNetm6urqOPXUscyceSNf+tLhDBgwkOHDd2fkyP35/Oe/wBZb\nbPlBO6lUiuHDd+fb3/4+N944iz/96WH23Xd/5s59hltuuZ01a9bwjW8cB8DTT/+Zz3/+EL72tTE8\n/fSfWbGixvAiSZIkacMOO+wI7rjjl3z/+9+mT5/eDB0aiPFVzjjjWwA0NDTwzjtvb1Rbw4YFAAYN\n2oLq6iW8+eY/2XXX3UilUnTv3p0QdiWVSnHiiadw22238p3vTCSdTjN8+O45+3wtDC+SJElSlpV0\n75fXtp566s/suefejB07nj/+8WFuvPF6Pv3pffnBD85n7dq13H77z9lmm20pKiqisbERaBplaWho\naKO1j27JvMMOO/LrX/8PmUyG+vp6XnstkslkePTRBzn00MM4/fTvcPvtP+f+++9l7Njx2fjI62V4\nkSRJkrIs31sb77LLrkyd+kNKS0tpbGxk6tTpPPLIQ5x++nhWrarlwAMPolevXoSwC9dddy2DBw9h\n+PDdueGG69h6621Y3zNkUqkUO+64M/vtN4oJE06mf//+lJSUUFpayq677sb06ZfSo0dPiouLOPvs\nC3L+OVOZTK4eovOfKitX5O/N1KGl02VUVq4odBmSOgD7AymXMkx59nLeq12S1VZ3rdiZw2YvZM27\n72atzd67DefJ3geyrHp11toE6F/RgzET9sUHPG6+pUuX8sQT/8tRRx3NmjVrOOmkY7n22p8xaNAW\nOXvPdLqszb8wR14kSZIkrVf//v155ZX/x4MP3g+kOPzwI3MaXDbE8CJJkiRpvVKpFOefP7nQZQA+\npFKSJElSQhheJEmSJCWC08YkSZKkrMrVHlVuOmB4kSRJkrLsFwsWU11Xn5W2KrqXcvKwbbLSVtJt\nMLyEEIqAWcAIoA4YF2Nc2Or6p4AraIqBi4GTYoxrcleuJEmS1PFV19VTtXptocvYZA8++HuWL1/O\nmDEffU7N5Mnnc9FFP6KkZPPGPpYtW8ZFF53DzJk3fKz62lvzciTQLca4P3AuTUEFgBBCCrgRODnG\n+Bngf4EdPlY1kiRJkgomlWp7atqUKZdtdnDJpvYqGAU8DBBjnBtC2KfVtWHAEuB7IYTdgQdijDE3\nZUqSJElan7q61Vx22RTeffdd6uvr+fa3v8999/2Gd95ZTENDI8ceewIHHzyaSZMmMHRoYNGihfTq\n1ZMRI/bmuef+wvvvr+DKK68jk8nw3HN/4dln51BbW8spp0xgv/1GcfTRh3Pnnb9hxozL6NatG++8\n8w5LllRxwQWTGTZsFx577E/cc8+dFBUVMWLEXpx66iSqq5cwZcpFNDY2sOWWW2Xlc7Y38tIXqGl1\n3NA8lQxgILA/MBP4PHBwCOGgrFQlSZIkaaP97ne/Yeutt+VnP7uVKVMuY968v1NeXsH119/K1VfP\n4qabrmf58mWkUimGD9+Na66ZxZo19fTs2YOrrrqOIUN2ZN68v5NKpSgvr+Caa67n8suv4sorLyeT\nyXwwIpNKpdhyy6258sqZHH30sdx//73U1NRw6603cs011zNr1s1UVr7HX/86l9tuu5XRow9h5swb\nOOSQL2blc7Y38lIDlLU6LooxNja/XgK83jLaEkJ4GNgHeHx9jZWX96KkpPhjlKvOJJ0ua/8mSV2C\n/YGUGw0NDYUuoUOoqOhNcXH+fgfNxde9vc9QWfkOBx54IOl0Gen0cO6/fzYHHjiquX8tI4Sh1NYu\npbS0mJEjP0k6XcbAgeXstdfupNNlDBo0gB49ikmlenDAAfs1t1NGv35llJSspagoRTpdRo8epXzq\nU3uRTpcxdOgQXnvtZVauXEJNzTLOO++7ANTW1lJTU8W//72YE088nnS6jIMPPpC77rrtY/f37YWX\nOcDhwOwQwkhgfqtri4A+IYSdmhfxfwa4eUONLV1a+3Fq1Xrlaju+Ftnfli+dLqOyckXW25WUPPYH\nUi7l+neEZKiuXkl+txnOUNG9NGutVXQvbfczbLHFtsyd+3dGjPg0ixe/xYMPPkhDQ4o999yX2tqV\nvPLKq/To0Z/6+gaWLq2lsnIFq1fXs2xZ0+tVq9ZQU7OKNWvW8MILf+dzn/sSlZXvsXJlLQ0NpTQ2\nZj74M8uXr6KycgXLltWyenU9PXuWM3DgIGbMmElxcTF/+MN9DB48jK23fpU///kZBgzYhuef/xv1\n9Q0b3d+vL+S0F17uBUaHEOY0H48NIYwB+sQYbwohfBO4s3nx/pwY40MbVY2y7roXb6ZqVXVW2xzY\ns4LT9xyX1TYlSZK6gnxvbXzEEV9l2rQfMWnSBDKZDD/5ybX85jf3cNpp46irq+OUUyZQXl6+UW3V\n1CznO9+ZyKpVqzjnnAubz34YnFpPIQPo378/xx13ApMmjaehoZGtttqa0aO/yMknj+OSSy7mscf+\nyODBQ9a7GcCmSGUy+UvklZUrjP85kWHKs5fzXu2SrLY6qNcAJo88G0deJOWS/YGUS7n5HWHXip05\nbPZC1rz7btba7L3bcJ7sfSDLqldnrU2A/hU9GDNhX3zAY7Kk02Vt/oW1t2BfkiRJkjoEw4skSZKk\nRCj8k2YkSV1YpnlXnlzNKnaaiCR1JoYXSVJB3f6zZ6hZlt057n379+DLx+yZ1TYlSYVneJEkFVTN\nstVZX6ArSeqcDC8qgFxOE3GKiCQpiXI1ddKNXnMnl19bf59ZH8OLCiLb00ScIiJJSrqr7plHZZan\nUA7bri9s3KM9tBl+sWAx1XX1WWuvontp3p8PkzSGFxWE00QkSfqoymWr+Xf1qqy2ObBf9y4fXoqK\nUuRmlCRDdV09VavX5qBtrY/hRZIkSZ1W77JuVC68g7V1y7Pabvc+2wLO+sg3w4skSZI6tbV1y6mv\nq85qmyXd+mW1PW0cH1IpSZIkKREML5IkSZISwWljktTpuH2nJKlzMrxIUid03Ys3U7Uqe/O7B/as\n4PQ9x2WtPUmSNofhRZI6oapV1bxXu6TQZUiSlFWueZEkSZKUCIYXSZIkSYlgeJEkSZKUCIYXSZIk\nSYlgeJEkSZKUCIYXSZIkSYngVsl5lasHx+XygXSSJElSx2B4ybOr7plH5bLVWW1z2HZ9oTyrTUqS\nJEkdjuElzyqXrebf1auy2ubAft0NL5IkSer0XPMiSZIkKREML5IkSZISwfAiSZIkKREML5IkSZIS\nwfAiSZIkKRHcbUySCsZnP0mStCkML5JUQD77SZKkjWd4kaQC8tlPkiRtPNe8SJIkSUoEw4skSZKk\nRNjgtLEQQhEwCxgB1AHjYowLW13/LvBNoLL51LdijAtyVKskSZKkLqy9NS9HAt1ijPuHEPYFrmg+\n1+ITwIkxxhdyVaAkSZIkQfvTxkYBDwPEGOcC+6xz/ZPA+SGEp0II5+agPkmSJEkC2g8vfYGaVscN\nzVPJWtwFfAv4HHBACOHLWa5PkiRJkoD2p43VAGWtjotijI2tjq+JMdYAhBAeAPYGHlhfY+XlvSgp\nKd7cWhOvoaGh0CVssoqK3hQXZ/fvLFdfh1zUKuVS0vqEJPUHYJ+gZElaf6Dcse/asPbCyxzgcGB2\nCGEkML/lQgihHzA/hDAcqKVp9OWWDTW2dGntx6s28ZL31Ovq6pVAKsut5ubrkJtapVxKVp+QpP4A\n7BOUNMnqD5Q79l1N0umyNs+3F17uBUaHEOY0H48NIYwB+sQYb2pe5/I4TTuR/SnG+HC2CpYkSZKk\n1jYYXmKMGWDiOqcXtLp+F03rXiRJkiQpp3xIpSRJkqREMLxIkiRJSgTDiyRJkqREaG/Bvrqw4lQR\nudn9xB1VJEmStOkML1qv/t37sfjqK6ivrMpquz2H7QzskdU2JUmS1PkZXrRB9ZVVrHn33ay2WTpw\nAPTOapOSJEnqAlzzIkmSJCkRDC+SJEmSEsHwIkmSJCkRXPOiTqGoKEVudzFL5bBtSZIkbQzDizqF\n3mXdqFx4B2vrlme13ZLu/Ujv9PWstilJkqTNY3hRp7G2bjn1ddWFLkNSB5Db0VhHYiWpUAwvkqRO\nJxejsY7ESlLhGV4kSZ2So7GS1Pm425gkSZKkRDC8SJIkSUoEw4skSZKkRDC8SJIkSUoEF+xLktpV\nnCoiN1sP5/LhspKkzsbwIklqV//u/Vh89RXUV1Zltd2ew3YG9shqm5KkzsvwIknaKPWVVax5992s\ntlk6cAD0zmqTkqROzDUvkiRJkhLB8CJJkiQpEQwvkiRJkhLB8CJJkiQpEQwvkiRJkhLB8CJJkiQp\nEQwvkiRJkhLB8CJJkiQpEQwvkiRJkhLB8CJJkiQpEQwvkiRJkhLB8CJJkiQpEQwvkiRJkhKhZEMX\nQwhFwCxgBFAHjIsxLmzjvhuBJTHG83JSpVQoqSIgk8s3yGHbkiRJncsGwwtwJNAtxrh/CGFf4Irm\ncx8IIXwL2B14IicVSgVUUlrGLxa8TXVdfVbbreheysnDtslqm5IkSZ1de+FlFPAwQIxxbghhn9YX\nQwj7A58GbgB2yUmFUoFV19VTtXptocuQJEnq8toLL32BmlbHDSGEohhjYwhhK+Bi4Cjg2I15s/Ly\nXpSUFG9epZ1AQ0NDoUtQB1JR0Zvi4q778yD7hCTy51a5Yn+gFvYzG9ZeeKkBylodF8UYG5tfHw0M\nBB4EtgR6hRBeiTHetr7Gli6t/Ti1dgK5XDuhpKmuXolrXro6+4Sk8edWuWN/oCb2M03S6bI2z7cX\nXuYAhwOzQwgjgfktF2KMM4GZACGEbwC7bCi4SJIkSdLH0V54uRcYHUKY03w8NoQwBugTY7xpnXv9\nJwNJkiRJObPB8BJjzAAT1zm9oI37fpnNoiRJkiRpXT6kUpIkSVIiGF4kSZIkJYLhRZIkSVIiGF4k\nSZIkJYLhRZIkSVIiGF4kSZIkJYLhRZIkSVIiGF4kSZIkJYLhRZIkSVIiGF4kSZIkJYLhRZIkSVIi\nlBS6AEmSEiFVBGRy+QY5bFuSOgfDiyRJG6GktIxfLHib6rr6rLZb0b2Uk4dtk9U2JamzMrxIkrSR\nquvqqVp9+aFqAAAQUklEQVS9ttBlSFKX5ZoXSZIkSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKUCIYX\nSZIkSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKU\nCIYXSZIkSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKUCIYXSZIkSYlQsqGL\nIYQiYBYwAqgDxsUYF7a6/t/AOUAGuCPGeG0Oa5UkSZLUhbU38nIk0C3GuD9wLnBFy4UQQjEwDTgY\n2A84LYRQkatCJUmSJHVt7YWXUcDDADHGucA+LRdijA3ALjHGFUAaKAbW5KhOSZIkSV1ce+GlL1DT\n6riheSoZADHGxhDCV4EXgMeB2uyXKEmSJEntrHmhKbiUtTouijE2tr4hxvjbEMK9wC+Ak5r/36by\n8l6UlBRvXqWdQENDQ6FLUAdSUdGb4uKu+/Mg+wR9yP5A9gdqYX+wYe2FlznA4cDsEMJIYH7LhRBC\nX+D3wOgY45oQwkpggz95S5d29YGZTKELUAdSXb0SSBW6DBWUfYKa2B/I/kAt7A+apNNlbZ5vL7zc\nC4wOIcxpPh4bQhgD9Ikx3hRC+BXwZAihHngR+FW2CpYkSZKk1jYYXmKMGWDiOqcXtLp+E3BTDuqS\nJEmSpI/wIZWSJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmS\nEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+S\nJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkR\nDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmS\nJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRSjZ0MYRQBMwCRgB1wLgY48JW18cA3wHW\nAv8HnBZjzOSuXEmSJEldVXsjL0cC3WKM+wPnAle0XAgh9AQuAf4rxngA0A84LFeFSpIkSera2gsv\no4CHAWKMc4F9Wl1bDewXY1zdfFwCrMp6hZIkSZJE++GlL1DT6riheSoZMcZMjLESIIRwBtA7xvin\n3JQpSZIkqavb4JoXmoJLWavjohhjY8tBc5C5HNgZ+O/23qy8vBclJcWbU2en0NDQUOgS1IFUVPSm\nuLjr/jzIPkEfsj+Q/YFa2B9sWHvhZQ5wODA7hDASmL/O9Rtomj521MYs1F+6tHaziuw83MtAH6qu\nXgmkCl2GCso+QU3sD2R/oBb2B03S6bI2z7cXXu4FRocQ5jQfj23eYawP8DfgFOBJ4LEQAsA1Mcbf\nZaViSZIkSWplg+GleTRl4jqnF7R67ZiWJEmSpLzwIZWSJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmS\nJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHw\nIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmS\nEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+S\nJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkRDC+SJEmSEsHwIkmSJCkR\nSjbmphBCETALGAHUAeNijAvXuacX8EfglBhjzHahkiRJkrq2jR15ORLoFmPcHzgXuKL1xRDCPsCT\nwA5AJqsVSpIkSRIbH15GAQ8DxBjnAvusc70bTQHHERdJkiRJObGx4aUvUNPquKF5KhkAMcZnYoxv\nZbUySZIkSWplo9a80BRcylodF8UYGzf1zcrLe1FSUrypf6zTaGhoKHQJ6kAqKnpTXNx1fx5kn6AP\n2R/I/kAt7A82bGPDyxzgcGB2CGEkMH9z3mzp0trN+WOdiMuB9KHq6pVAqtBlqKDsE9TE/kD2B2ph\nf9AknS5r8/zGhpd7gdEhhDnNx2NDCGOAPjHGm7JQnyRJkiRt0EaFlxhjBpi4zukFbdx3UDaKkiRJ\nkqR1+ZBKSZIkSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKUCIYXSZIkSYlg\neJEkSZKUCIYXSZIkSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKUCIYXSZIk\nSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKUCIYX\nSZIkSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKU\nCIYXSZIkSYlgeJEkSZKUCIYXSZIkSYlgeJEkSZKUCCUbuhhCKAJmASOAOmBcjHFhq+uHAxcBa4Fb\nY4w357BWSZIkSV1YeyMvRwLdYoz7A+cCV7RcCCGUAlcCo4HPAhNCCINyVagkSZKkrq298DIKeBgg\nxjgX2KfVtV2B12OMy2OM9cDTwIE5qVKSJElSl7fBaWNAX6Cm1XFDCKEoxtjYfG15q2srgH4bauyT\nn9y9zfN///tLXeb+mpVr+Nr3ftnm/Xf/5KQ2zx931m0bvH9OSYrG4joaM40AnHTzpDbvv23cT9s8\nv777Z540jZtWrYWGxo+cv2/M19u8/4i7ftXm+XXvL6kop2/PHkz5yYlt3j/5rNvbPL+h+/v07U5J\n949++x167OVt3v/Q/5zd5vk27y8q4czZT7Z5/6zjD23z/Gl3PtTu/cUpuKO06cevI31/en/+769Z\nuYaGxswH59v7eV9XW/dX9O3O9eNnfdAntLa5/UN5j36Upgd+cH5jf97bu//Rz4yib88e/3H+4/YP\nxSUpSkvW8NDdZ7V5/2b1D0UlrMp0o+HDv66N+nlvra37K7qX8slP7tHm/YX+/vT+/N6f7t/0s7C5\nvw+0dX9F3+5071nxkfOb+vtAW/f/qaiEo045rc37N7d/SD31OHWpm8k094mb8/tAW86ZciylJWtg\nnT5xk34faOP+4m5lVDSWfqzfB9a9v6J76X+c7yjfn/m+/80332jzeiqTybR5ASCEcAXwbIxxdvPx\nv2KM2zW/3gP4cYzxy83HVwJPxxh/u94GJUmSJGkztTdtbA7wJYAQwkhgfqtrrwJDQwjlIYRuNE0Z\n+0tOqpQkSZLU5bU38pLiw93GAMYCnwT6xBhvCiEcBlxMUwi6JcZ4fY7rlSRJktRFbTC8SJIkSVJH\n4UMqJUmSJCWC4UWSJElSIhheJEmSJCWC4UWSJElSIhheJEkFE0L4zyeySZK0Hu42prxo3nZ72xjj\nv0IIn4ox/rXQNUkqjBDCBGBYjPGsEMLDwJ0xxrYfHS6p0woh/Bto+UW05f8pIBNj3LowVamjc+RF\n+fIz4Ljm1yeEEK4pZDGSCmoicF7z68OB0wpYi6QCiTFuGWPcCjgVeI6mh6G/2Pyf1KaSQhegLuMT\nMcZvAcQYzwwhPFXogiQVzNrm/1peNxawFkmFNwOYACxrPnZakNbL8KJ8yYQQBsYYq0II5UBxoQuS\nVDD3AU+FEJ4DPgHcX+B6JBXWSzHGJwpdhJLBNS/KixDCYcBMYCnQHzgtxvhwYauSVCghhL2AALwa\nY3SKiNSFhRC+QdN00leaT2VijKcUsCR1YIYX5U0IoQTYGngnxlhf6HokFUYIYTvgSmA3IALfjTH+\ns6BFSSqYEMLzwHRgefOpTIzxkQKWpA7MaWPKixDCQcAtQA3QP4QwIcb4aIHLklQYNwGzgKeAz9LU\nNxxc0IokFdI7Mcb/KXQRSgbDi/LlUuCAGOPbIYRtgHsBw4vUNfWIMbasc/ldCOF7Ba1GUqGtbt42\nfR5Ni/UzMcbzC1yTOii3Sla+rI0xvg0QY1wMrCpwPZIKpziEMAIghLAH7iwkdXW/B+6mac3LqzRN\nJ5Xa5MiL8mVFCOEM4EngQKC6wPVIKpxvA7eGELYC3gbGF7geSQUUY/xFoWtQcrhgX3kRQugPXAjs\nQtO/rFwWY1xa2KokSZKUJI68KKdCCIP5cErIzFaXymjaNllSFxFC+L/ml8VAN6ASGAhUxxj3LVhh\nkqTEMLwo137a/P/tgD7AX4G9gSrggEIVJSn/Yox7AIQQfg78OMYYQwg7AVMKW5kkKSlcsK+cijEe\nHmM8HPgXsHuMcQwwgqYtkyV1TTvFGCNAjHEhMKSw5UiSksKRF+XLVjHG1c2v1wKDClmMpIKqCiFc\nAvwNGAW8UeB6JEkJ4ciL8uWBEMKTIYSrgKeB2YUuSFLBfJ2mNW9fAhYDpwCEEHoUsihJUsfnbmPK\nmxDCJ4AAvBxjfLH53MgY47OFrUxSRxBCeDzGeFCh65AkdVxOG1PexBifB55f5/Q0wF9WJEmS1C6n\njUmSJElKBMOLJEmSpEQwvEiSJElKBMOLJKmjeLnQBUiSOjYX7KvQ7ix0AZLyK4Tw+Dqn6oE3gakF\nKEeSlCCGF+VUCOHfQFv7cWdijFvHGG/Kd02SCu6fwByanvk0EjgceBa4GTi4cGVJkjo6w4tyKsa4\nZaFrkNThDI4xjm1+/WoI4esxxptDCCcWtCpJUodneFFOhRDuWs+lTIzx+LwWI6mj6BZC+CLwDLA/\nUBJC2AnoVdiyJEkdneFFuXZDq9dtTR+T1PWcDMwArgJeAk6hafrY9wpYkyQpAVKZjL9PKvdCCAOA\nLwClQArYKsY4rbBVScqnEEJpjLE+hNCNpn6gRSbGuKZQdUmSksORF+XLvTRtgzoCWAXEwpYjqQBu\nA8bQ9POfAgYAVTSNyu5YwLokSQnhc16UL6kY46nAq8AhwLYFrkdSnsUYxzS/PJ2mwLKIpm2Sx673\nD0mS1IrhRflSH0LoCfQBGoFBBa5HUuFMBvaNMe4JHAT8uMD1SJISwvCifJkFnAk8CvwLeKOw5Ugq\noPdjjO8BxBgXA7UFrkeSlBAu2FfehRD6xhhrCl2HpPwKIXy/+eUXgPeAJ2naZWxAjPGIghUmSUoM\nF+wrL0IIpwLfAno0H2dijMMLW5WkPFtC01qXO2hasJ+hKcD4r2iSpI3iyIvyIoTwCnAosKzlXIxx\n2fr/hCRJkvRRjrwoX14E3ooxri10IZIkSUomw4vy5TFgUQhhYfNxJsb4uUIWJEmSpGQxvChfTgW+\nBiwvdCGSJElKJsOL8uVfwN9ijA2FLkSSJEnJZHhRvvQAXgwhvETTzkKZGOPxBa5JkiRJCWJ4Ub7c\nDvQD1gLnANcWthxJkiQlTVGhC1CXMQF4GTgEuAA4srDlSJIkKWkML8qXRuApoF+M8S7AtS+SJEna\nJIYX5UspMB14MoRwENCtwPVIkiQpYQwvypexwEKaAkwa+EZhy5EkSVLSpDKZTKFrkCRJkqR2OfIi\nSZIkKREML5IkSZISwfAiSZIkKREML5IkSZIS4f8DFlhxXKDsGpkAAAAASUVORK5CYII=\n",
       "text": [
        "<matplotlib.figure.Figure at 0x112693810>"
       ]
      }
     ],
     "prompt_number": 263
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As a final experiment, I want to investigate whether certain combinations of labelsets help to improve the performance on individual labelsets. For example, does the performance of the emotions labeling go up if we combine it with the good/misfortune labelset. The code below computes all combinations."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from itertools import chain, combinations\n",
      "\n",
      "def ml_experiment_per_category(X, y, categories, num_neighbors=101):\n",
      "    kf = KFold(len(y), n_folds=10, shuffle=True, random_state=1)\n",
      "    rank_scores = np.zeros((10, len(categories)))\n",
      "    for i, (train, test) in enumerate(kf):\n",
      "        # separate dataset into training (or index) and test set\n",
      "        X_train, X_test, y_train, y_test = X[train], X[test], y[train], y[test]\n",
      "        # make a set of all unique labels\n",
      "        labels = set(flatten(list(y_train)))\n",
      "        # initialize the ML-IR System\n",
      "        model = ML_IRSystem(num_neighbors=num_neighbors, smooth=1., k1=1.2, b=0.75, cutoff=0)\n",
      "        # Fit the model with unigrams, remove all stopwords and hapaxes.\n",
      "        model.fit_raw(X_train, y_train, ngram_range=(1, 1), stop_words='english', min_df=2)\n",
      "        # compute the ranking\n",
      "        ranking = model.rank(X_test, raw=True)\n",
      "        ranking = [sorted(ranks, key=ranks.__getitem__, reverse=True) for ranks in ranking]\n",
      "        # compute the average precision per category\n",
      "        for c, category in enumerate(categories):\n",
      "            ranking_category = [[label for label in rankedlist if label.startswith(category)]\n",
      "                                for rankedlist in ranking]\n",
      "            ranking_category, y_category = zip(*[(r, y_) for r, y_ in zip(ranking_category, y_test)\n",
      "                                                 if any(l.startswith(category) for l in y_)])\n",
      "            rank_scores[i, c] = mean_average_precision(ranking_category, y_category)\n",
      "    return rank_scores\n",
      "\n",
      "all_labels = list(read_labels(\"data/all_labels.txt\"))\n",
      "\n",
      "def is_of_type(label):\n",
      "    return lambda x: x.startswith(label)\n",
      "\n",
      "label_types = (\"act\", \"set\", \"emo\", \"obj\", (\"gf\", \"mf\"))\n",
      "scores = []\n",
      "for i, labelgroup in enumerate(chain.from_iterable(\n",
      "    combinations(label_types, r) for r in range(len(label_types)+1))):\n",
      "    if len(labelgroup) > 0:\n",
      "        labelset = ((doc, [label for label in labels if \n",
      "                           any(map(lambda l: is_of_type(l)(label), labelgroup))])\n",
      "                    for doc, labels in all_labels)\n",
      "        labelset = [(doc, labels) for doc, labels in labelset if labels]\n",
      "        labelset, texts = zip(*match_labels_documents(documents, labelset))\n",
      "        labelset, texts = np.array(labelset), np.array(texts)\n",
      "        scores.append((ml_experiment_per_category(\n",
      "            texts, labelset, labelgroup).mean(axis=0), '-'.join(map(str, labelgroup))))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 43
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "scores_per_category = defaultdict(list)\n",
      "for scoring, labels in scores:\n",
      "    for i, label in enumerate(labels.split('-')):\n",
      "        scores_per_category[label].append((scoring[i], labels))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 44
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As can be seen from the results below, combining different label groups does not really improve the performance of the individual labelsets..."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for category, groupscores in scores_per_category.items():\n",
      "    print 'CATEGORY:', category\n",
      "    for (score, labelgroup) in groupscores:\n",
      "        print '   MAP: %.2f for %s in combination with labels from %s' % (\n",
      "            score, category, labelgroup)\n",
      "    print"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "CATEGORY: ('gf', 'mf')\n",
        "   MAP: 0.59 for ('gf', 'mf') in combination with labels from ('gf', 'mf')\n",
        "   MAP: 0.59 for ('gf', 'mf') in combination with labels from act-('gf', 'mf')\n",
        "   MAP: 0.60 for ('gf', 'mf') in combination with labels from set-('gf', 'mf')\n",
        "   MAP: 0.59 for ('gf', 'mf') in combination with labels from emo-('gf', 'mf')\n",
        "   MAP: 0.61 for ('gf', 'mf') in combination with labels from obj-('gf', 'mf')\n",
        "   MAP: 0.60 for ('gf', 'mf') in combination with labels from act-set-('gf', 'mf')\n",
        "   MAP: 0.59 for ('gf', 'mf') in combination with labels from act-emo-('gf', 'mf')\n",
        "   MAP: 0.60 for ('gf', 'mf') in combination with labels from act-obj-('gf', 'mf')\n",
        "   MAP: 0.60 for ('gf', 'mf') in combination with labels from set-emo-('gf', 'mf')\n",
        "   MAP: 0.60 for ('gf', 'mf') in combination with labels from set-obj-('gf', 'mf')\n",
        "   MAP: 0.60 for ('gf', 'mf') in combination with labels from emo-obj-('gf', 'mf')\n",
        "   MAP: 0.60 for ('gf', 'mf') in combination with labels from act-set-emo-('gf', 'mf')\n",
        "   MAP: 0.60 for ('gf', 'mf') in combination with labels from act-set-obj-('gf', 'mf')\n",
        "   MAP: 0.60 for ('gf', 'mf') in combination with labels from act-emo-obj-('gf', 'mf')\n",
        "   MAP: 0.60 for ('gf', 'mf') in combination with labels from set-emo-obj-('gf', 'mf')\n",
        "   MAP: 0.60 for ('gf', 'mf') in combination with labels from act-set-emo-obj-('gf', 'mf')\n",
        "\n",
        "CATEGORY: emo\n",
        "   MAP: 0.67 for emo in combination with labels from emo\n",
        "   MAP: 0.68 for emo in combination with labels from act-emo\n",
        "   MAP: 0.67 for emo in combination with labels from set-emo\n",
        "   MAP: 0.68 for emo in combination with labels from emo-obj\n",
        "   MAP: 0.67 for emo in combination with labels from emo-('gf', 'mf')\n",
        "   MAP: 0.67 for emo in combination with labels from act-set-emo\n",
        "   MAP: 0.67 for emo in combination with labels from act-emo-obj\n",
        "   MAP: 0.68 for emo in combination with labels from act-emo-('gf', 'mf')\n",
        "   MAP: 0.67 for emo in combination with labels from set-emo-obj\n",
        "   MAP: 0.67 for emo in combination with labels from set-emo-('gf', 'mf')\n",
        "   MAP: 0.68 for emo in combination with labels from emo-obj-('gf', 'mf')\n",
        "   MAP: 0.67 for emo in combination with labels from act-set-emo-obj\n",
        "   MAP: 0.67 for emo in combination with labels from act-set-emo-('gf', 'mf')\n",
        "   MAP: 0.67 for emo in combination with labels from act-emo-obj-('gf', 'mf')\n",
        "   MAP: 0.67 for emo in combination with labels from set-emo-obj-('gf', 'mf')\n",
        "   MAP: 0.67 for emo in combination with labels from act-set-emo-obj-('gf', 'mf')\n",
        "\n",
        "CATEGORY: set\n",
        "   MAP: 0.57 for set in combination with labels from set\n",
        "   MAP: 0.57 for set in combination with labels from act-set\n",
        "   MAP: 0.57 for set in combination with labels from set-emo\n",
        "   MAP: 0.57 for set in combination with labels from set-obj\n",
        "   MAP: 0.57 for set in combination with labels from set-('gf', 'mf')\n",
        "   MAP: 0.57 for set in combination with labels from act-set-emo\n",
        "   MAP: 0.57 for set in combination with labels from act-set-obj\n",
        "   MAP: 0.57 for set in combination with labels from act-set-('gf', 'mf')\n",
        "   MAP: 0.57 for set in combination with labels from set-emo-obj\n",
        "   MAP: 0.57 for set in combination with labels from set-emo-('gf', 'mf')\n",
        "   MAP: 0.57 for set in combination with labels from set-obj-('gf', 'mf')\n",
        "   MAP: 0.57 for set in combination with labels from act-set-emo-obj\n",
        "   MAP: 0.57 for set in combination with labels from act-set-emo-('gf', 'mf')\n",
        "   MAP: 0.57 for set in combination with labels from act-set-obj-('gf', 'mf')\n",
        "   MAP: 0.57 for set in combination with labels from set-emo-obj-('gf', 'mf')\n",
        "   MAP: 0.57 for set in combination with labels from act-set-emo-obj-('gf', 'mf')\n",
        "\n",
        "CATEGORY: obj\n",
        "   MAP: 0.66 for obj in combination with labels from obj\n",
        "   MAP: 0.66 for obj in combination with labels from act-obj\n",
        "   MAP: 0.66 for obj in combination with labels from set-obj\n",
        "   MAP: 0.65 for obj in combination with labels from emo-obj\n",
        "   MAP: 0.66 for obj in combination with labels from obj-('gf', 'mf')\n",
        "   MAP: 0.66 for obj in combination with labels from act-set-obj\n",
        "   MAP: 0.66 for obj in combination with labels from act-emo-obj\n",
        "   MAP: 0.66 for obj in combination with labels from act-obj-('gf', 'mf')\n",
        "   MAP: 0.66 for obj in combination with labels from set-emo-obj\n",
        "   MAP: 0.66 for obj in combination with labels from set-obj-('gf', 'mf')\n",
        "   MAP: 0.65 for obj in combination with labels from emo-obj-('gf', 'mf')\n",
        "   MAP: 0.66 for obj in combination with labels from act-set-emo-obj\n",
        "   MAP: 0.66 for obj in combination with labels from act-set-obj-('gf', 'mf')\n",
        "   MAP: 0.66 for obj in combination with labels from act-emo-obj-('gf', 'mf')\n",
        "   MAP: 0.66 for obj in combination with labels from set-emo-obj-('gf', 'mf')\n",
        "   MAP: 0.66 for obj in combination with labels from act-set-emo-obj-('gf', 'mf')\n",
        "\n",
        "CATEGORY: act\n",
        "   MAP: 0.73 for act in combination with labels from act\n",
        "   MAP: 0.73 for act in combination with labels from act-set\n",
        "   MAP: 0.72 for act in combination with labels from act-emo\n",
        "   MAP: 0.73 for act in combination with labels from act-obj\n",
        "   MAP: 0.72 for act in combination with labels from act-('gf', 'mf')\n",
        "   MAP: 0.73 for act in combination with labels from act-set-emo\n",
        "   MAP: 0.73 for act in combination with labels from act-set-obj\n",
        "   MAP: 0.73 for act in combination with labels from act-set-('gf', 'mf')\n",
        "   MAP: 0.73 for act in combination with labels from act-emo-obj\n",
        "   MAP: 0.73 for act in combination with labels from act-emo-('gf', 'mf')\n",
        "   MAP: 0.73 for act in combination with labels from act-obj-('gf', 'mf')\n",
        "   MAP: 0.73 for act in combination with labels from act-set-emo-obj\n",
        "   MAP: 0.73 for act in combination with labels from act-set-emo-('gf', 'mf')\n",
        "   MAP: 0.73 for act in combination with labels from act-set-obj-('gf', 'mf')\n",
        "   MAP: 0.73 for act in combination with labels from act-emo-obj-('gf', 'mf')\n",
        "   MAP: 0.73 for act in combination with labels from act-set-emo-obj-('gf', 'mf')\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 45
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}