MulticlassLDF.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:1b3492af97916660f0d45ff003f162b1c66af4cf6e7cebce626206dd49da489d"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# Classification with Label Dependent Features\n",
      "\n",
      "In the [MulticlassClassification]() notebook, we saw how to build classifiers whose output is one of K classes. In those examples, each class was treated as an atomic unit: class 1 had no more to do with class 2 than it did with class 65. In some cases, you want to associate features with the different classes (or labels): this gives rise to **label dependent feature** models. (These are also useful to [zero-shot learning](#zeroshot), which we'll see later in this notebook.)\n",
      "\n",
      "## Formalism and Data Format\n",
      "\n",
      "A natural problem where LDF arises is in search ranking-like problems. Here, the input to the system is a query (like \"vowpal wabbit ldf howto\") and the output is one of the billion-or-so documents on the web. It doesn't make a lot of sense to give each document on the web a unique id and then try to learn a mapping from inputs (queries) to outputs (documents), treating all documents independently. Instead, what you *want* to do is associate a feature vector with a *pair* $(x,y)$, where $x$ is the query and $y$ is a hypothetical returned document. This feature vector might do things like count how many of the query terms are present in the corresponding document.\n",
      "\n",
      "The associated LDF learning problem is: given some $x$ (query) and a set of possible retreived documents $(y_1, y_2, \\dots, y_K)$, we want to compute $\\arg\\min_k f( \\varphi(x,y_k) )$, where $\\varphi(x,y_k)$ is the features you extract for that pair, and $f$ is the learned \"cost\" function. *[Note: sometimes people talk about a \"score\" function that we try to maximize rather than a \"cost\" function that we try to minimize. Here, we'll talk only about costs, so lower is always better.]*\n",
      "\n",
      "The basic data format (we'll see briefer alternatives later) for LDF is something like the following:\n",
      "\n",
      "    1:1 | features for pair (x1,y1) ...\n",
      "    2:0 | features for pair (x1,y2) ...  <-- this is the correct \"label\"\n",
      "    3:1 | features for pair (x1,y3) ...\n",
      "    4:1 | features for pair (x1,y4) ...\n",
      "    <blank line>\n",
      "    1:0 | features for pair (x2,y1) ...  <-- this is the correct \"label\"\n",
      "    2:1 | features for pair (x2,y2) ...\n",
      "    5:1 | features for pair (x2,y5) ...\n",
      "    <blank line>\n",
      "    \n",
      "And so on. Here, each **block** of data is separated by a blank line. A single block corresponds to a single training example, and each line in that block corresponds to a valid prediction for that example. The \"label\" portion consists of a class identifier (1, 2, 3, 4, 5, ...) and a cost (:1 means \"cost=1\" and :0 means \"cost=0\"). After that, any features that make sense can be written out.\n",
      "\n",
      "A few notes:\n",
      "\n",
      "1. Costs can be any non-negative values, and there can be multiple labels with a cost of zero.\n",
      "1. The actual labels (1, 2, 3, 4, 5, ...) are completely ignored by the learning algorithm. Everything you want to encode *must* be encoded in the features. If each line in a single block has exactly the same features, it will be impossible to differentiate them. This also means that label ids needn't be consistent across blocks.\n",
      "1. You don't have to use consecutive label ids (as in the second block above).\n",
      "\n",
      "## LDF for Quizbowl, Step 1\n",
      "\n",
      "I'm going to assume you've already gone through the [MulticlassClassification]() notebook and have downloaded the QuizBowl data. If not, do so now. Here is some code copied from before for loading the data."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import csv, sys, re, random, os\n",
      "\n",
      "def sanify(s): return s.replace(':',';').replace('|','/')   # replace : with ; and | with /\n",
      "def tokenize(s): return re.sub('([^A-Za-z0-9 ]+)', ' \\\\1 ', s).split()  # add space around anything not alphanum\n",
      "\n",
      "def readQuizBowlData(filename):\n",
      "    train,dev,test = [],[],[]\n",
      "    data = csv.reader( open(filename, 'r').readlines() )\n",
      "    header = data.next()\n",
      "    if header != ['Question ID', 'Fold', 'Category', 'Answer', 'Text']:\n",
      "        raise Exception('data improperly formatted')\n",
      "    for item in iter(data):\n",
      "        y = item[3]        \n",
      "        x = sanify(' '.join(tokenize(item[4].replace('|||',''))))\n",
      "        if   item[1] == 'train': train.append( (x,y) )\n",
      "        elif item[1] == 'dev'  :   dev.append( (x,y) )\n",
      "        elif item[1] == 'test' :  test.append( (x,y) )\n",
      "    return train,dev,test\n",
      "\n",
      "def makeLabelIDs(train, outputFile):\n",
      "    labelIds = { label: k+1 for k,label in enumerate(set([y for x,y in train])) }\n",
      "    labelIds['***UNKNOWN***'] = len(labelIds)+1\n",
      "    with open(outputFile, 'w') as h:\n",
      "        for label,k in labelIds.iteritems():\n",
      "            print >>h, '%d\\t%s' % (k,label)\n",
      "    return labelIds\n",
      "\n",
      "train,dev,test = readQuizBowlData('data/questions/questions.csv')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 12
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "\n",
      "We're going to start off by basically replicating the experiment there, but using a very wasteful LDF format. In particular, for each quizbowl example, we'll generate a single block. This block will have one line *for every class*. For class 10, for instance, we'll replace every features \"f\" with \"f_10\". This will be horribly bloated, but we'll fix that later.\n",
      "\n",
      "In order to actually make this not take huge amounts of time and disk space, we'll cut down the train/dev/test significantly (to the point that they're toy data)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "random.seed(9876)\n",
      "random.shuffle(train)\n",
      "train_small = train[:10]\n",
      "labelIds_small = makeLabelIDs(train_small, 'data/quizbowl.ldf.labels')\n",
      "print 'total of %d labels, including one for unknown' % len(labelIds_small)\n",
      "\n",
      "def writeVWFile(filename, data, labelIds):\n",
      "    unknownId = labelIds['***UNKNOWN***']\n",
      "    with open(filename,'w') as h:\n",
      "        for x,y in data:\n",
      "            trueLabel = labelIds.get(y, unknownId)\n",
      "            feats     = x.split()\n",
      "            # generate a single block by going over EVERY POSSIBLE LABEL\n",
      "            for k in range(1, unknownId+1):           # unknownId is the last label\n",
      "                cost = 0. if trueLabel == k else 1.   # how bad is it to predict k?\n",
      "                x_k  = ' '.join(['%s_%d' % (f,k) for f in feats])  # example for label k\n",
      "                print >>h, '%d:%g |q %s' % (k, cost, x_k)\n",
      "            print >>h, ''   # blank line to separate blocks\n",
      "            \n",
      "writeVWFile('data/quizbowl.ldf.tr', train_small, labelIds_small)\n",
      "!wc data/quizbowl.ldf.tr\n",
      "!head -n12 data/quizbowl.ldf.tr"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "total of 11 labels, including one for unknown\n",
        "   120  14212 105193 data/quizbowl.ldf.tr\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1:1 |q Within_1 a_1 constant_1 it_1 '_1 s_1 the_1 Laplacian_1 of_1 the_1 function_1 \"_1 one_1 over_1 distance_1 \"_1 and_1 is_1 also_1 the_1 charge_1 density_1 of_1 a_1 point_1 charge_1 ._1 Actually_1 an_1 operator_1 which_1 projects_1 out_1 a_1 function_1 '_1 s_1 value_1 at_1 the_1 origin_1 ,_1 it_1 '_1 s_1 not_1 really_1 a_1 function_1 at_1 all_1 ._1 For_1 ten_1 points_1 ,_1 name_1 this_1 mathematical_1 entity_1 ,_1 which_1 is_1 zero_1 everywhere_1 except_1 the_1 origin_1 and_1 is_1 designated_1 by_1 a_1 Greek_1 letter_1 ._1\r\n",
        "2:1 |q Within_2 a_2 constant_2 it_2 '_2 s_2 the_2 Laplacian_2 of_2 the_2 function_2 \"_2 one_2 over_2 distance_2 \"_2 and_2 is_2 also_2 the_2 charge_2 density_2 of_2 a_2 point_2 charge_2 ._2 Actually_2 an_2 operator_2 which_2 projects_2 out_2 a_2 function_2 '_2 s_2 value_2 at_2 the_2 origin_2 ,_2 it_2 '_2 s_2 not_2 really_2 a_2 function_2 at_2 all_2 ._2 For_2 ten_2 points_2 ,_2 name_2 this_2 mathematical_2 entity_2 ,_2 which_2 is_2 zero_2 everywhere_2 except_2 the_2 origin_2 and_2 is_2 designated_2 by_2 a_2 Greek_2 letter_2 ._2\r\n",
        "3:1 |q Within_3 a_3 constant_3 it_3 '_3 s_3 the_3 Laplacian_3 of_3 the_3 function_3 \"_3 one_3 over_3 distance_3 \"_3 and_3 is_3 also_3 the_3 charge_3 density_3 of_3 a_3 point_3 charge_3 ._3 Actually_3 an_3 operator_3 which_3 projects_3 out_3 a_3 function_3 '_3 s_3 value_3 at_3 the_3 origin_3 ,_3 it_3 '_3 s_3 not_3 really_3 a_3 function_3 at_3 all_3 ._3 For_3 ten_3 points_3 ,_3 name_3 this_3 mathematical_3 entity_3 ,_3 which_3 is_3 zero_3 everywhere_3 except_3 the_3 origin_3 and_3 is_3 designated_3 by_3 a_3 Greek_3 letter_3 ._3\r\n",
        "4:1 |q Within_4 a_4 constant_4 it_4 '_4 s_4 the_4 Laplacian_4 of_4 the_4 function_4 \"_4 one_4 over_4 distance_4 \"_4 and_4 is_4 also_4 the_4 charge_4 density_4 of_4 a_4 point_4 charge_4 ._4 Actually_4 an_4 operator_4 which_4 projects_4 out_4 a_4 function_4 '_4 s_4 value_4 at_4 the_4 origin_4 ,_4 it_4 '_4 s_4 not_4 really_4 a_4 function_4 at_4 all_4 ._4 For_4 ten_4 points_4 ,_4 name_4 this_4 mathematical_4 entity_4 ,_4 which_4 is_4 zero_4 everywhere_4 except_4 the_4 origin_4 and_4 is_4 designated_4 by_4 a_4 Greek_4 letter_4 ._4\r\n",
        "5:1 |q Within_5 a_5 constant_5 it_5 '_5 s_5 the_5 Laplacian_5 of_5 the_5 function_5 \"_5 one_5 over_5 distance_5 \"_5 and_5 is_5 also_5 the_5 charge_5 density_5 of_5 a_5 point_5 charge_5 ._5 Actually_5 an_5 operator_5 which_5 projects_5 out_5 a_5 function_5 '_5 s_5 value_5 at_5 the_5 origin_5 ,_5 it_5 '_5 s_5 not_5 really_5 a_5 function_5 at_5 all_5 ._5 For_5 ten_5 points_5 ,_5 name_5 this_5 mathematical_5 entity_5 ,_5 which_5 is_5 zero_5 everywhere_5 except_5 the_5 origin_5 and_5 is_5 designated_5 by_5 a_5 Greek_5 letter_5 ._5\r\n",
        "6:1 |q Within_6 a_6 constant_6 it_6 '_6 s_6 the_6 Laplacian_6 of_6 the_6 function_6 \"_6 one_6 over_6 distance_6 \"_6 and_6 is_6 also_6 the_6 charge_6 density_6 of_6 a_6 point_6 charge_6 ._6 Actually_6 an_6 operator_6 which_6 projects_6 out_6 a_6 function_6 '_6 s_6 value_6 at_6 the_6 origin_6 ,_6 it_6 '_6 s_6 not_6 really_6 a_6 function_6 at_6 all_6 ._6 For_6 ten_6 points_6 ,_6 name_6 this_6 mathematical_6 entity_6 ,_6 which_6 is_6 zero_6 everywhere_6 except_6 the_6 origin_6 and_6 is_6 designated_6 by_6 a_6 Greek_6 letter_6 ._6\r\n",
        "7:1 |q Within_7 a_7 constant_7 it_7 '_7 s_7 the_7 Laplacian_7 of_7 the_7 function_7 \"_7 one_7 over_7 distance_7 \"_7 and_7 is_7 also_7 the_7 charge_7 density_7 of_7 a_7 point_7 charge_7 ._7 Actually_7 an_7 operator_7 which_7 projects_7 out_7 a_7 function_7 '_7 s_7 value_7 at_7 the_7 origin_7 ,_7 it_7 '_7 s_7 not_7 really_7 a_7 function_7 at_7 all_7 ._7 For_7 ten_7 points_7 ,_7 name_7 this_7 mathematical_7 entity_7 ,_7 which_7 is_7 zero_7 everywhere_7 except_7 the_7 origin_7 and_7 is_7 designated_7 by_7 a_7 Greek_7 letter_7 ._7\r\n",
        "8:1 |q Within_8 a_8 constant_8 it_8 '_8 s_8 the_8 Laplacian_8 of_8 the_8 function_8 \"_8 one_8 over_8 distance_8 \"_8 and_8 is_8 also_8 the_8 charge_8 density_8 of_8 a_8 point_8 charge_8 ._8 Actually_8 an_8 operator_8 which_8 projects_8 out_8 a_8 function_8 '_8 s_8 value_8 at_8 the_8 origin_8 ,_8 it_8 '_8 s_8 not_8 really_8 a_8 function_8 at_8 all_8 ._8 For_8 ten_8 points_8 ,_8 name_8 this_8 mathematical_8 entity_8 ,_8 which_8 is_8 zero_8 everywhere_8 except_8 the_8 origin_8 and_8 is_8 designated_8 by_8 a_8 Greek_8 letter_8 ._8\r\n",
        "9:1 |q Within_9 a_9 constant_9 it_9 '_9 s_9 the_9 Laplacian_9 of_9 the_9 function_9 \"_9 one_9 over_9 distance_9 \"_9 and_9 is_9 also_9 the_9 charge_9 density_9 of_9 a_9 point_9 charge_9 ._9 Actually_9 an_9 operator_9 which_9 projects_9 out_9 a_9 function_9 '_9 s_9 value_9 at_9 the_9 origin_9 ,_9 it_9 '_9 s_9 not_9 really_9 a_9 function_9 at_9 all_9 ._9 For_9 ten_9 points_9 ,_9 name_9 this_9 mathematical_9 entity_9 ,_9 which_9 is_9 zero_9 everywhere_9 except_9 the_9 origin_9 and_9 is_9 designated_9 by_9 a_9 Greek_9 letter_9 ._9\r\n",
        "10:0 |q Within_10 a_10 constant_10 it_10 '_10 s_10 the_10 Laplacian_10 of_10 the_10 function_10 \"_10 one_10 over_10 distance_10 \"_10 and_10 is_10 also_10 the_10 charge_10 density_10 of_10 a_10 point_10 charge_10 ._10 Actually_10 an_10 operator_10 which_10 projects_10 out_10 a_10 function_10 '_10 s_10 value_10 at_10 the_10 origin_10 ,_10 it_10 '_10 s_10 not_10 really_10 a_10 function_10 at_10 all_10 ._10 For_10 ten_10 points_10 ,_10 name_10 this_10 mathematical_10 entity_10 ,_10 which_10 is_10 zero_10 everywhere_10 except_10 the_10 origin_10 and_10 is_10 designated_10 by_10 a_10 Greek_10 letter_10 ._10\r\n",
        "11:1 |q Within_11 a_11 constant_11 it_11 '_11 s_11 the_11 Laplacian_11 of_11 the_11 function_11 \"_11 one_11 over_11 distance_11 \"_11 and_11 is_11 also_11 the_11 charge_11 density_11 of_11 a_11 point_11 charge_11 ._11 Actually_11 an_11 operator_11 which_11 projects_11 out_11 a_11 function_11 '_11 s_11 value_11 at_11 the_11 origin_11 ,_11 it_11 '_11 s_11 not_11 really_11 a_11 function_11 at_11 all_11 ._11 For_11 ten_11 points_11 ,_11 name_11 this_11 mathematical_11 entity_11 ,_11 which_11 is_11 zero_11 everywhere_11 except_11 the_11 origin_11 and_11 is_11 designated_11 by_11 a_11 Greek_11 letter_11 ._11\r\n",
        "\r\n"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "So in this silly data, we have 11 labels, so all blocks has length 11. One of these (in this case, label 8) is the correct label and has a cost of zero.\n",
      "\n",
      "Before we worry about shrinking this file, let's train and test a model on it. The main thing is the `--csoaa_ldf multiline` argument, which says we want to run cost-sensitive one against all training, where data is provided in multiline format."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "!vw -k -c --csoaa_ldf multiline -d data/quizbowl.ldf.tr --passes 10 --holdout_off -f data/quizbowl.ldf.model\n",
      "!vw -i data/quizbowl.ldf.model -t -d data/quizbowl.ldf.tr -r data/quizbowl.ldf.tr.raw\n",
      "!head -n24 data/quizbowl.ldf.tr.raw"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "final_regressor = data/quizbowl.ldf.model\r\n",
        "Num weight bits = 18\r\n",
        "learning rate = 0.5\r\n",
        "initial_t = 0\r\n",
        "power_t = 0.5\r\n",
        "decay_learning_rate = 1\r\n",
        "creating cache_file = data/quizbowl.ldf.tr.cache\r\n",
        "Reading datafile = data/quizbowl.ldf.tr\r\n",
        "num sources = 1\r\n",
        "average  since         example        example  current  current  current\r\n",
        "loss     last          counter         weight    label  predict features\r\n",
        "1.000000 1.000000            1            1.0    known        1      847\r\n",
        "0.500000 0.000000            2            2.0    known        0     1408\r\n",
        "0.750000 1.000000            4            4.0    known        0     1243\r\n",
        "0.875000 1.000000            8            8.0    known        0     1166\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.625000 0.375000           16           16.0    known        0     1386\r\n",
        "0.312500 0.000000           32           32.0    known        1     1408\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "0.156250 0.000000           64           64.0    known        0     1243\r\n",
        "\r\n",
        "finished run\r\n",
        "number of examples per pass = 10\r\n",
        "passes used = 10\r\n",
        "weighted example sum = 100.000000\r\n",
        "weighted label sum = 0.000000\r\n",
        "average loss = 0.100000\r\n",
        "total feature number = 141020\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "only testing\r\n",
        "raw predictions = data/quizbowl.ldf.tr.raw\r\n",
        "Num weight bits = 18\r\n",
        "learning rate = 0.5\r\n",
        "initial_t = 0\r\n",
        "power_t = 0.5\r\n",
        "using no cache\r\n",
        "Reading datafile = data/quizbowl.ldf.tr\r\n",
        "num sources = 1\r\n",
        "average  since         example        example  current  current  current\r\n",
        "loss     last          counter         weight    label  predict features\r\n",
        "0.000000 0.000000            1            1.0    known        0      847\r\n",
        "0.000000 0.000000            2            2.0    known        1     1408\r\n",
        "0.000000 0.000000            4            4.0    known        0     1243\r\n",
        "0.000000 0.000000            8            8.0    known        0     1166\r\n",
        "\r\n",
        "finished run\r\n",
        "number of examples per pass = 10\r\n",
        "passes used = 1\r\n",
        "weighted example sum = 10.000000\r\n",
        "weighted label sum = 0.000000\r\n",
        "average loss = 0.000000\r\n",
        "total feature number = 14102\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1:1.59138\r\n",
        "2:1.53712\r\n",
        "3:1.5944\r\n",
        "4:1.63682\r\n",
        "5:1.55016\r\n",
        "6:1.48501\r\n",
        "7:1.33151\r\n",
        "8:1.3858\r\n",
        "9:1.44881\r\n",
        "10:-0.202838\r\n",
        "11:1.6381\r\n",
        "\r\n",
        "1:-0.0452115\r\n",
        "2:1.05496\r\n",
        "3:1.07181\r\n",
        "4:1.01842\r\n",
        "5:1.14079\r\n",
        "6:1.2917\r\n",
        "7:1.00388\r\n",
        "8:1.12249\r\n",
        "9:1.32976\r\n",
        "10:1.89733\r\n",
        "11:1.30963\r\n",
        "\r\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Here, we've trained a model and made predictions on the training data. The numbers printed in the raw file are the expected costs.\n",
      "\n",
      "Now, there's a lot of stuff going on here that's silly. We'll address these in order:\n",
      "\n",
      "1. The features in the above example are really a cross product between the input $x$ and the label $y_k$. But `vw` has built in support for cross-produces with `-q`, so let's just use that. After fixing that problem, the same input $x$ gets repeated over and over again. That's really redundant.\n",
      "1. Now, the same features (or in this case, single feature) gets repeated over and over again for the label. That's also redundant.\n",
      "1. Finally, we'd like more interesting features on the label.\n",
      "\n",
      "## Separating out the input and output using quadratic features: shared features\n",
      "\n",
      "Hopefully it's clear that a training example like:\n",
      "\n",
      "    1:1 |q If_1 it_1 were_1 always_1 true_1 ,_1 fractional_1 distillation_1 would_1 ...\n",
      "    \n",
      "is exactly the same as:\n",
      "\n",
      "    1:1 |l 1 |q If it were always true , fractional distillation would ...\n",
      "\n",
      "if `-q lq` is added to the command line. This simplifies the data, but then the blocks will look like:\n",
      "\n",
      "    1:1 |l 1 |q If it were always true , fractional distillation would ...\n",
      "    2:1 |l 2 |q If it were always true , fractional distillation would ...\n",
      "    3:1 |l 3 |q If it were always true , fractional distillation would ...\n",
      "    4:1 |l 4 |q If it were always true , fractional distillation would ...\n",
      "    ...\n",
      "    \n",
      "where the \"`q`\" part will be the same in every example. We can fix this problem by having \"`shared`\" features for blocks. This looks like the following:\n",
      "\n",
      "    ***shared*** |q If it were always true , fractional distillation would ...\n",
      "    1:1 |l 1\n",
      "    2:1 |l 2\n",
      "    3:1 |l 3\n",
      "    4:1 |l 4\n",
      "    ...\n",
      "\n",
      "What this says is: pretend that the `shared` line appears on every example within this block. We can fix our code to output data that looks like this, which will be much smaller."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def writeVWFile(filename, data, labelIds):\n",
      "    unknownId = labelIds['***UNKNOWN***']\n",
      "    with open(filename,'w') as h:\n",
      "        for x,y in data:\n",
      "            print >>h, '***shared*** |q %s' % x\n",
      "            trueLabel = labelIds.get(y, unknownId)\n",
      "            # generate a single block by going over EVERY POSSIBLE LABEL\n",
      "            for k in range(1, unknownId+1):           # unknownId is the last label\n",
      "                cost = 0. if trueLabel == k else 1.   # how bad is it to predict k?\n",
      "                print >>h, '%d:%g |l %d' % (k, cost, k)\n",
      "            print >>h, ''   # blank line to separate blocks\n",
      "            \n",
      "writeVWFile('data/quizbowl.ldf.tr', train_small, labelIds_small)\n",
      "!head -n13 data/quizbowl.ldf.tr\n",
      "!vw -k -c --csoaa_ldf multiline -d data/quizbowl.ldf.tr --passes 10 --holdout_off -f data/quizbowl.ldf.model -q lq\n",
      "!vw -i data/quizbowl.ldf.model -t -d data/quizbowl.ldf.tr -r data/quizbowl.ldf.tr.raw\n",
      "!head -n24 data/quizbowl.ldf.tr.raw"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "***shared*** |q Within a constant it ' s the Laplacian of the function \" one over distance \" and is also the charge density of a point charge . Actually an operator which projects out a function ' s value at the origin , it ' s not really a function at all . For ten points , name this mathematical entity , which is zero everywhere except the origin and is designated by a Greek letter .\r\n",
        "1:1 |l 1\r\n",
        "2:1 |l 2\r\n",
        "3:1 |l 3\r\n",
        "4:1 |l 4\r\n",
        "5:1 |l 5\r\n",
        "6:1 |l 6\r\n",
        "7:1 |l 7\r\n",
        "8:1 |l 8\r\n",
        "9:1 |l 9\r\n",
        "10:0 |l 10\r\n",
        "11:1 |l 11\r\n",
        "\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "creating quadratic features for pairs: lq \r\n",
        "final_regressor = data/quizbowl.ldf.model\r\n",
        "Num weight bits = 18\r\n",
        "learning rate = 0.5\r\n",
        "initial_t = 0\r\n",
        "power_t = 0.5\r\n",
        "decay_learning_rate = 1\r\n",
        "creating cache_file = data/quizbowl.ldf.tr.cache\r\n",
        "Reading datafile = data/quizbowl.ldf.tr\r\n",
        "num sources = 1\r\n",
        "average  since         example        example  current  current  current\r\n",
        "loss     last          counter         weight    label  predict features\r\n",
        "1.000000 1.000000            1            1.0    known        1      858\r\n",
        "0.500000 0.000000            2            2.0    known        0     1419\r\n",
        "0.750000 1.000000            4            4.0    known        0     1254\r\n",
        "0.875000 1.000000            8            8.0    known        0     1177\r\n",
        "0.625000 0.375000           16           16.0    known        0     1397\r\n",
        "0.312500 0.000000           32           32.0    known        1     1419\r\n",
        "0.156250 0.000000           64           64.0    known        0     1254\r\n",
        "\r\n",
        "finished run\r\n",
        "number of examples per pass = 10\r\n",
        "passes used = 10\r\n",
        "weighted example sum = 100.000000\r\n",
        "weighted label sum = 0.000000\r\n",
        "average loss = 0.100000\r\n",
        "total feature number = 142120\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "creating quadratic features for pairs: lq \r\n",
        "only testing\r\n",
        "raw predictions = data/quizbowl.ldf.tr.raw\r\n",
        "Num weight bits = 18\r\n",
        "learning rate = 0.5\r\n",
        "initial_t = 0\r\n",
        "power_t = 0.5\r\n",
        "using no cache\r\n",
        "Reading datafile = data/quizbowl.ldf.tr\r\n",
        "num sources = 1\r\n",
        "average  since         example        example  current  current  current\r\n",
        "loss     last          counter         weight    label  predict features\r\n",
        "0.000000 0.000000            1            1.0    known        0      858\r\n",
        "0.000000 0.000000            2            2.0    known        1     1419\r\n",
        "0.000000 0.000000            4            4.0    known        0     1254\r\n",
        "0.000000 0.000000            8            8.0    known        0     1177\r\n",
        "\r\n",
        "finished run\r\n",
        "number of examples per pass = 10\r\n",
        "passes used = 1\r\n",
        "weighted example sum = 10.000000\r\n",
        "weighted label sum = 0.000000\r\n",
        "average loss = 0.000000\r\n",
        "total feature number = 14212\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1:1.41836\r\n",
        "2:1.0625\r\n",
        "3:1.06238\r\n",
        "4:1.07689\r\n",
        "5:1.1484\r\n",
        "6:1.14225\r\n",
        "7:1.07524\r\n",
        "8:1.08765\r\n",
        "9:1.01596\r\n",
        "10:-0.0424166\r\n",
        "11:1.48031\r\n",
        "\r\n",
        "1:-0.178392\r\n",
        "2:1.71998\r\n",
        "3:1.3529\r\n",
        "4:1.09386\r\n",
        "5:1.41592\r\n",
        "6:1.11247\r\n",
        "7:1.11531\r\n",
        "8:1.17376\r\n",
        "9:1.12266\r\n",
        "10:1.56086\r\n",
        "11:1.25449\r\n",
        "\r\n"
       ]
      }
     ],
     "prompt_number": 15
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Note that the output is similar but not identical. It's not identical because there are extra features (the `-q` continues to include the linear terms, which are not in the original formulation).\n",
      "\n",
      "## Headers with label-specific features\n",
      "\n",
      "In some cases, you really want to be able to define whatever arbitrary features you want for $\\varphi(x,y_k)$. But in many cases, $\\varphi(x,y_k)$ is just the outer product of some features of $x$ (that might be written as `shared`) features and some features of $y_k$. In the case above, the features of $y_k$ is just the singleton feature `k`, but (as we'll see below) you often want more. We can specify all of these in a header as follows:\n",
      "\n",
      "    label:1 |l 1\n",
      "    label:2 |l 2\n",
      "    ...\n",
      "    label:11 |l 11\n",
      "    <blank line>\n",
      "    \n",
      "This batch of label definitions basically says \"any time you see an example with, eg, label 2, add \"`|l 2`\" to it. We can then generate the equivalent data much more simply as:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def writeVWFile(filename, data, labelIds):\n",
      "    unknownId = labelIds['***UNKNOWN***']\n",
      "    with open(filename,'w') as h:\n",
      "        for k in range(1, unknownId+1):\n",
      "            print >>h, '***label***:%d |l %d' % (k,k)\n",
      "        print >>h, ''\n",
      "        \n",
      "        for x,y in data:\n",
      "            print >>h, '***shared*** |q %s' % x\n",
      "            trueLabel = labelIds.get(y, unknownId)\n",
      "            # generate a single block by going over EVERY POSSIBLE LABEL\n",
      "            for k in range(1, unknownId+1):           # unknownId is the last label\n",
      "                cost = 0. if trueLabel == k else 1.   # how bad is it to predict k?\n",
      "                print >>h, '%d:%g | bias' % (k, cost)\n",
      "            print >>h, ''   # blank line to separate blocks\n",
      "            \n",
      "writeVWFile('data/quizbowl.ldf.tr', train_small, labelIds_small)\n",
      "!echo -n \"\\nTRAINING DATA:\\n\\n\"\n",
      "!head -n20 data/quizbowl.ldf.tr\n",
      "!echo -n \"\\nTRAINING:\\n\\n\"\n",
      "!vw -k -c --csoaa_ldf multiline -d data/quizbowl.ldf.tr --passes 10 --holdout_off -f data/quizbowl.ldf.model -q lq\n",
      "!echo -n \"\\nTESTING:\\n\\n\"\n",
      "!vw -i data/quizbowl.ldf.model -t -d data/quizbowl.ldf.tr -r data/quizbowl.ldf.tr.raw\n",
      "!echo -n \"\\nOUTPUT (HEAD):\\n\\n\"\n",
      "!head -n24 data/quizbowl.ldf.tr.raw"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r\n",
        "TRAINING DATA:\r\n",
        "\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "***label***:1 |l 1\r\n",
        "***label***:2 |l 2\r\n",
        "***label***:3 |l 3\r\n",
        "***label***:4 |l 4\r\n",
        "***label***:5 |l 5\r\n",
        "***label***:6 |l 6\r\n",
        "***label***:7 |l 7\r\n",
        "***label***:8 |l 8\r\n",
        "***label***:9 |l 9\r\n",
        "***label***:10 |l 10\r\n",
        "***label***:11 |l 11\r\n",
        "\r\n",
        "***shared*** |q Within a constant it ' s the Laplacian of the function \" one over distance \" and is also the charge density of a point charge . Actually an operator which projects out a function ' s value at the origin , it ' s not really a function at all . For ten points , name this mathematical entity , which is zero everywhere except the origin and is designated by a Greek letter .\r\n",
        "1:1 | bias\r\n",
        "2:1 | bias\r\n",
        "3:1 | bias\r\n",
        "4:1 | bias\r\n",
        "5:1 | bias\r\n",
        "6:1 | bias\r\n",
        "7:1 | bias\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r\n",
        "TRAINING:\r\n",
        "\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "creating quadratic features for pairs: lq \r\n",
        "final_regressor = data/quizbowl.ldf.model\r\n",
        "Num weight bits = 18\r\n",
        "learning rate = 0.5\r\n",
        "initial_t = 0\r\n",
        "power_t = 0.5\r\n",
        "decay_learning_rate = 1\r\n",
        "creating cache_file = data/quizbowl.ldf.tr.cache\r\n",
        "Reading datafile = data/quizbowl.ldf.tr\r\n",
        "num sources = 1\r\n",
        "average  since         example        example  current  current  current\r\n",
        "loss     last          counter         weight    label  predict features\r\n",
        "1.000000 1.000000            1            1.0    known        1      880\r\n",
        "0.500000 0.000000            2            2.0    known        0     1441\r\n",
        "0.750000 1.000000            4            4.0    known        0     1276\r\n",
        "0.875000 1.000000            8            8.0    known        0     1199\r\n",
        "0.625000 0.375000           16           16.0    known        0     1419\r\n",
        "0.312500 0.000000           32           32.0    known        1     1441\r\n",
        "0.156250 0.000000           64           64.0    known        0     1276\r\n",
        "\r\n",
        "finished run\r\n",
        "number of examples per pass = 10\r\n",
        "passes used = 10\r\n",
        "weighted example sum = 100.000000\r\n",
        "weighted label sum = 0.000000\r\n",
        "average loss = 0.100000\r\n",
        "total feature number = 144320\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r\n",
        "TESTING:\r\n",
        "\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "creating quadratic features for pairs: lq \r\n",
        "only testing\r\n",
        "raw predictions = data/quizbowl.ldf.tr.raw\r\n",
        "Num weight bits = 18\r\n",
        "learning rate = 0.5\r\n",
        "initial_t = 0\r\n",
        "power_t = 0.5\r\n",
        "using no cache\r\n",
        "Reading datafile = data/quizbowl.ldf.tr\r\n",
        "num sources = 1\r\n",
        "average  since         example        example  current  current  current\r\n",
        "loss     last          counter         weight    label  predict features\r\n",
        "0.000000 0.000000            1            1.0    known        0      869\r\n",
        "0.000000 0.000000            2            2.0    known        1     1430\r\n",
        "0.000000 0.000000            4            4.0    known        0     1265\r\n",
        "0.000000 0.000000            8            8.0    known        0     1188\r\n",
        "\r\n",
        "finished run\r\n",
        "number of examples per pass = 10\r\n",
        "passes used = 1\r\n",
        "weighted example sum = 10.000000\r\n",
        "weighted label sum = 0.000000\r\n",
        "average loss = 0.000000\r\n",
        "total feature number = 14322\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r\n",
        "OUTPUT (HEAD):\r\n",
        "\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1:1.41235\r\n",
        "2:1.05971\r\n",
        "3:1.06102\r\n",
        "4:1.07412\r\n",
        "5:1.14742\r\n",
        "6:1.14116\r\n",
        "7:1.07265\r\n",
        "8:1.08606\r\n",
        "9:1.01476\r\n",
        "10:-0.0439742\r\n",
        "11:1.47786\r\n",
        "\r\n",
        "1:-0.178582\r\n",
        "2:1.71463\r\n",
        "3:1.34784\r\n",
        "4:1.08935\r\n",
        "5:1.41376\r\n",
        "6:1.11073\r\n",
        "7:1.11116\r\n",
        "8:1.1721\r\n",
        "9:1.12157\r\n",
        "10:1.55313\r\n",
        "11:1.25278\r\n",
        "\r\n"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Note that in the above we had to add a \"bias\" feature to each example. Technically this should be unnecessary, but for some reason there's an arguable bug in vw where it ignores these examples with no features; until that bug is fixed, you have to have at least *something* written as a feature for each example.\n",
      "\n",
      "## Actually putting useful features in the label-dependent features\n",
      "\n",
      "Right now we're not actually sharing any useful information in the label dependent features, *but we can*! For the QuizBowl data, each label corresponds to a Wikipedia page, so we'll make the labels depend on the words that appear in that page."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from collections import Counter\n",
      "\n",
      "def readWikipedia(labelIds):\n",
      "    wiki = {}\n",
      "    directory = 'data' + os.sep + 'questions' + os.sep + 'wiki' + os.sep\n",
      "    for labelName in labelIds.iterkeys():\n",
      "        try:\n",
      "            with open(directory + labelName + '.txt', 'r') as h:\n",
      "                bow = Counter()\n",
      "                for l in h.readlines():\n",
      "                    if not l.startswith('== '): continue\n",
      "                    for w in tokenize(l.lower())[1:-1]:\n",
      "                        bow[w] += 1\n",
      "                wiki[labelName] = ' '.join(['%s:%d' % (sanify(w), c) for (w,c) in bow.iteritems()])\n",
      "        except IOError:\n",
      "            print 'warning: could not read wikipedia page for %s' % labelName\n",
      "            wiki[labelName] = 'unknown'\n",
      "    return wiki\n",
      "\n",
      "def writeVWFile(filename, data, labelIds, wikipedia):\n",
      "    unknownId = labelIds['***UNKNOWN***']\n",
      "    with open(filename,'w') as h:\n",
      "        for labelName, k in labelIds.iteritems():\n",
      "            print >>h, '***label***:%d |l label%d %s' % (k, k, wikipedia[labelName])\n",
      "        print >>h, ''\n",
      "        \n",
      "        for x,y in data:\n",
      "            print >>h, '***shared*** |q %s' % x\n",
      "            trueLabel = labelIds.get(y, unknownId)\n",
      "            # generate a single block by going over EVERY POSSIBLE LABEL\n",
      "            for k in range(1, unknownId+1):           # unknownId is the last label\n",
      "                cost = 0. if trueLabel == k else 1.   # how bad is it to predict k?\n",
      "                print >>h, '%d:%g | bias' % (k, cost)\n",
      "            print >>h, ''   # blank line to separate blocks\n",
      "            \n",
      "labelIds = makeLabelIDs(train, 'data/quizbowl.ldf.labels')\n",
      "print 'total of %d labels, including one for unknown' % len(labelIds)\n",
      "\n",
      "wikipedia = readWikipedia(labelIds_small)\n",
      "\n",
      "writeVWFile('data/quizbowl.ldf.tr', train_small, labelIds_small, wikipedia)\n",
      "!echo -n \"\\nTRAINING DATA:\\n\\n\"\n",
      "!head -n20 data/quizbowl.ldf.tr\n",
      "!echo -n \"\\nTRAINING:\\n\\n\"\n",
      "!vw -k -c --csoaa_ldf multiline -d data/quizbowl.ldf.tr --passes 10 --holdout_off -f data/quizbowl.ldf.model -q lq\n",
      "!echo -n \"\\nTESTING:\\n\\n\"\n",
      "!vw -i data/quizbowl.ldf.model -t -d data/quizbowl.ldf.tr -r data/quizbowl.ldf.tr.raw\n",
      "!echo -n \"\\nOUTPUT (HEAD):\\n\\n\"\n",
      "!head -n24 data/quizbowl.ldf.tr.raw"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "total of 2321 labels, including one for unknown\n",
        "warning: could not read wikipedia page for ***UNKNOWN***\n",
        "\r\n",
        "TRAINING DATA:\r\n",
        "\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "***label***:2 |l label2 opium:2 see:1 also:1 second:1 references:1 war:2 first:1\r\n",
        "***label***:4 |l label4 theory:1 paintings:1 color:1 external:1 influence:1 links:1 also:1 see:1 references:1 drawings:1 biography:1 exhibitions:1\r\n",
        "***label***:5 |l label5 and:1 also:1 industrial:1 safety:1 links:1 compounds:1 characteristics:1 notes:1 storage:1 citations:1 applications:1 see:1 production:1 role:1 external:1 references:1 of:1 biological:1 precautions:1 o2:1 history:1\r\n",
        "***label***:3 |l label3 and:1 activities:1 life:2 writings:1 bibliography:1 family:1 service:1 notes:1 later:1 links:1 early:1 legacy:1 references:1 reading:1 in:1 navy:1 further:1 the:1 works:1 religious:1 external:1\r\n",
        "***label***:6 |l label6 links:1 also:1 rankings:1 demographics:1 see:1 culture:1 etymology:1 history:1 references:1 external:1 international:1 politics:1 economy:1 notes:1 geography:1\r\n",
        "***label***:11 |l label11 unknown\r\n",
        "***label***:7 |l label7 and:1 colleges:1 links:1 people:1 culture:1 references:1 politics:1 research:1 around:1 universities:1 relations:1 sports:1 international:1 geography:1 economy:1 transportation:1 scientific:1 demographics:1 munich:1 famous:1 external:1 subdivisions:1 architecture:1 institutions:1 history:1\r\n",
        "***label***:1 |l label1 and:3 life:1 modern:1 letters:1 links:1 of:1 abelard:1 philosophy:1 translations:1 theology:1 cultural:1 citations:1 music:1 external:1 editions:1 further:1 references:2 the:1 reading:1 heloise:1 notes:1\r\n",
        "***label***:8 |l label8 and:1 death:1 links:1 notes:1 work:1 also:1 see:1 legacy:1 references:1 external:1 further:1 reading:1 biography:1 reputation:1\r\n",
        "***label***:9 |l label9 literary:1 life:1 links:1 career:1 private:1 years:1 early:1 reputation:1 references:1 external:1 works:1\r\n",
        "***label***:10 |l label10 links:1 sokhotski:1 overview:1 see:1 references:1 derivatives:1 also:1 distributional:1 -:1 transform:1 to:1 kronecker:1 function:1 relationship:1 theorem:1 applications:1 external:1 delta:2 representations:1 comb:1 properties:1 plemelj:1 dirac:1 of:1 notes:1 definitions:1 the:2 fourier:1 history:1\r\n",
        "\r\n",
        "***shared*** |q Within a constant it ' s the Laplacian of the function \" one over distance \" and is also the charge density of a point charge . Actually an operator which projects out a function ' s value at the origin , it ' s not really a function at all . For ten points , name this mathematical entity , which is zero everywhere except the origin and is designated by a Greek letter .\r\n",
        "1:1 | bias\r\n",
        "2:1 | bias\r\n",
        "3:1 | bias\r\n",
        "4:1 | bias\r\n",
        "5:1 | bias\r\n",
        "6:1 | bias\r\n",
        "7:1 | bias\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r\n",
        "TRAINING:\r\n",
        "\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "creating quadratic features for pairs: lq \r\n",
        "final_regressor = data/quizbowl.ldf.model\r\n",
        "Num weight bits = 18\r\n",
        "learning rate = 0.5\r\n",
        "initial_t = 0\r\n",
        "power_t = 0.5\r\n",
        "decay_learning_rate = 1\r\n",
        "creating cache_file = data/quizbowl.ldf.tr.cache\r\n",
        "Reading datafile = data/quizbowl.ldf.tr\r\n",
        "num sources = 1\r\n",
        "average  since         example        example  current  current  current\r\n",
        "loss     last          counter         weight    label  predict features\r\n",
        "1.000000 1.000000            1            1.0    known        1     1234\r\n",
        "0.500000 0.000000            2            2.0    known        0     1795\r\n",
        "0.750000 1.000000            4            4.0    known        0     1630\r\n",
        "0.875000 1.000000            8            8.0    known        0     1553\r\n",
        "0.625000 0.375000           16           16.0    known        0     1773\r\n",
        "0.312500 0.000000           32           32.0    known        1     1795\r\n",
        "0.156250 0.000000           64           64.0    known        0     1630\r\n",
        "\r\n",
        "finished run\r\n",
        "number of examples per pass = 10\r\n",
        "passes used = 10\r\n",
        "weighted example sum = 100.000000\r\n",
        "weighted label sum = 0.000000\r\n",
        "average loss = 0.100000\r\n",
        "total feature number = 179720\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r\n",
        "TESTING:\r\n",
        "\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "creating quadratic features for pairs: lq \r\n",
        "only testing\r\n",
        "raw predictions = data/quizbowl.ldf.tr.raw\r\n",
        "Num weight bits = 18\r\n",
        "learning rate = 0.5\r\n",
        "initial_t = 0\r\n",
        "power_t = 0.5\r\n",
        "using no cache\r\n",
        "Reading datafile = data/quizbowl.ldf.tr\r\n",
        "num sources = 1\r\n",
        "average  since         example        example  current  current  current\r\n",
        "loss     last          counter         weight    label  predict features\r\n",
        "0.000000 0.000000            1            1.0    known        0     1046\r\n",
        "0.000000 0.000000            2            2.0    known        1     1607\r\n",
        "0.000000 0.000000            4            4.0    known        0     1442\r\n",
        "0.000000 0.000000            8            8.0    known        0     1365\r\n",
        "\r\n",
        "finished run\r\n",
        "number of examples per pass = 10\r\n",
        "passes used = 1\r\n",
        "weighted example sum = 10.000000\r\n",
        "weighted label sum = 0.000000\r\n",
        "average loss = 0.000000\r\n",
        "total feature number = 16092\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\r\n",
        "OUTPUT (HEAD):\r\n",
        "\r\n"
       ]
      },
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "1:1.60651\r\n",
        "2:1.55012\r\n",
        "3:1.27702\r\n",
        "4:1.30667\r\n",
        "5:1.10201\r\n",
        "6:1.17644\r\n",
        "7:1.12054\r\n",
        "8:1.09035\r\n",
        "9:1.24678\r\n",
        "10:-0.164604\r\n",
        "11:1.52328\r\n",
        "\r\n",
        "1:-0.112427\r\n",
        "2:1.21939\r\n",
        "3:1.41709\r\n",
        "4:1.15276\r\n",
        "5:1.51793\r\n",
        "6:1.23505\r\n",
        "7:1.12692\r\n",
        "8:1.1945\r\n",
        "9:1.19273\r\n",
        "10:1.9022\r\n",
        "11:1.27931\r\n",
        "\r\n"
       ]
      }
     ],
     "prompt_number": 17
    }
   ],
   "metadata": {}
  }
 ]
}