magicPCAv2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using PCA to visualize the MtG universe\n",
    "\n",
    "In this notebook, we're going to scrape Magic the Gathering's <a href=\"http://gatherer.wizards.com/Pages/Default.aspx\" target=\"_blank\">Gatherer</a> card database and then perform <a href=\"http://en.wikipedia.org/wiki/Principal_component_analysis\" target=\"_blank\">principal components analysis</a> to visualize hidden relationships between cards. Our goal will be to see how much card-to-card variation can be simplified and then plotted in two-dimensions and again what those card groupings look like.\n",
    "\n",
    "<img src=\"http://thisisinfamous.com/wp-content/uploads/2014/01/Magic-The-Gathering-Duels-of-the-Planeswalkers-2012.jpg\" style=\"max-height:300px\">\n",
    "\n",
    "This data set is very high-dimensional -- there are over a 100 unique mechanics in the game and the game state has many different elements (hand, battlefield, mana pool, etc.). Being able to translate the <a href=\"http://mtgsalvation.gamepedia.com/Magic:_The_Gathering_statistics_and_trivia?cookieSetup=true\" target=\"_blank\">13,000 unique card texts</a> into structured data is also a challenge NLP-related task."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Warning: This notebook is long...so, for the impatient:\n",
    "\n",
    "Here is what we will be working towards, *a programmatic mapping of every Magic card ever made *across two psuedo-axes:\n",
    "\n",
    "<img src=\"./pca2.png\">\n",
    "\n",
    "We will show that while Magic cards can differ in thousands of ways, they can be roughly categorized based on two simple measures: how \"creature-y\" are they? and how much do they related to the board or non-board state?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementation details\n",
    "\n",
    "Pretty baller, right? We will interpret and grok this graph later, but for now, let's do this...\n",
    "\n",
    "*LEERRRROOYYY JENNNKIINNNNNSSS*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Outline\n",
    "Here's a breakdown of the four steps that we'll go through to accomplish this task:\n",
    "1. Scrape + clean the data using `requests`, `web` from `pattern`, and `pandas`\n",
    "- Extract features from the data using `fuzzywuzzy` and domain knowledge\n",
    "- Perform and analyze PCA using `sklearn`\n",
    "- Visualize + interpret results using the `plotly` graphing library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*First some boring imports and settings (feel free to skip over)*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 248,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# boring imports\n",
    "\n",
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pylab as plt\n",
    "\n",
    "import requests\n",
    "from pattern import web\n",
    "requests.packages.urllib3.disable_warnings()\n",
    "\n",
    "import re, string\n",
    "from sets import Set\n",
    "from collections import Counter\n",
    "from fuzzywuzzy import fuzz\n",
    "\n",
    "database = {}\n",
    "pd.set_option('display.max_rows', 10)\n",
    "\n",
    "# Silly helper functions\n",
    "\n",
    "def isInt(s):\n",
    "    try: \n",
    "        int(s)\n",
    "        return True\n",
    "    except ValueError:\n",
    "        return False\n",
    "\n",
    "def anyIntOrColor(l):\n",
    "    for val in l:\n",
    "        if isInt(val) | (val in ['Black', 'Red', 'Green', 'Blue', 'White']) : return True\n",
    "    return False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (1) -- Scrape baby, scrape\n",
    "\n",
    "Our first order of business is scraping the data from the Gatherer database using `requests` and `web` from `pattern`. In it's simplest form, every Magic card has a name, text, type, mana cost, and power/toughness (if it's a creature). An example is Hypnotic Specter, a powerful creature in the early days of Magic:\n",
    "\n",
    "<img src=\"./hyppy.png\" style=\"margin: auto; display:block; height: 300px;\">\n",
    "\n",
    "To scrape the relevant card features, we will construct card URLs using the card's `multiverse_id` and on the page we load will look for unique HTML elements that correspond to each of the features we will to obtain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# grabCard scrapes:\n",
    "\n",
    "# name, types, text (lowered, alphanumeritized), mana cost,\n",
    "# cmc, power and toughness, and rarity.\n",
    "\n",
    "# and adds it to the global card database\n",
    "\n",
    "def grabCard(multiverse_id):\n",
    "    xml = \"http://gatherer.wizards.com/Pages/Card/Details.aspx?multiverseid=\" + str(multiverse_id)\n",
    "    dom = web.Element(requests.get(xml).text)\n",
    "    \n",
    "    # card name, card type\n",
    "    cardName = dom('div.cardImage img')[0].attributes['alt'] if dom('div .cardImage img') else ''\n",
    "    cardType = [element.strip() for element in \\\n",
    "                dom('div#ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_typeRow div.value')[0].content.split(u'\\u2014')]\n",
    "    \n",
    "    # extract, parse, clean text into a list\n",
    "    cardText = []\n",
    "    pattern = re.compile('[\\W_]+')\n",
    "    for line in dom('div.cardtextbox'):\n",
    "        for element in line:\n",
    "            cardText.append(element)\n",
    "    \n",
    "    for i in xrange(len(cardText)):\n",
    "        if cardText[i].type == 'element' and cardText[i].tag == 'img':\n",
    "            cardText[i] = cardText[i].attributes['alt']\n",
    "        else:\n",
    "            cardText[i] = str(cardText[i]).strip().lower()\n",
    "        pattern.sub('', cardText[i]) \n",
    "    \n",
    "    # really gross, needs to be cleaned\n",
    "    \n",
    "    fullCardText = ''\n",
    "    \n",
    "    for line in cardText:\n",
    "        fullCardText += line\n",
    "\n",
    "    # mana symbols\n",
    "    manaCost = [element.attributes['alt'] for element in dom('div#ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_manaRow div.value img')]\n",
    "    cmc = int(dom('div#ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_cmcRow div.value')[0].content.strip()) \\\n",
    "            if dom('div#ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_cmcRow div.value') else np.nan\n",
    "    \n",
    "    # rarity\n",
    "    rarity = dom('div #ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_rarityRow div.value span')[0].content.lower()\n",
    "    \n",
    "    # p/t\n",
    "    power = np.nan\n",
    "    power = [_.strip() for _ in dom('div#ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_ptRow div.value')[0].content.split(' / ')][0] \\\n",
    "                if dom('div#ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_ptRow div.value') else np.nan\n",
    "    power = float(power) if power != '*' and power != np.nan else np.nan\n",
    "    toughness = [_.strip() for _ in dom('div#ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_ptRow div.value')[0].content.split(' / ')][1] \\\n",
    "                    if dom('div#ctl00_ctl00_ctl00_MainContent_SubContent_SubContent_ptRow div.value') else np.nan\n",
    "    toughness = float(toughness) if (toughness != '*' and toughness != '7-*' and toughness != '1+*' and toughness != np.nan) else np.nan\n",
    "      \n",
    "    # add data\n",
    "    database[cardName] = {\n",
    "                            'cardType' : cardType,\n",
    "                            'cardText' : cardText,\n",
    "                            'manaCost' : manaCost,\n",
    "                            'cmc' : cmc,\n",
    "                            'rarity': rarity,\n",
    "                            'power' : power,\n",
    "                            'toughness' : toughness,\n",
    "                            'fullCardText' : fullCardText\n",
    "                         }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perform the scraping\n",
    "\n",
    "We'll iterate through a range of `multiverse_id`s to scrape a desired amount of cards. Note that it takes around 1 minute/500 `multiverse_id`s. Given that there are 13k+ cards (and multiple versions of each -- see below), we'll limit our scraping to ~500 cards from the very first Magic set: <a href=\"http://en.wikipedia.org/wiki/Limited_Edition_%28Magic:_The_Gathering%29\" target=\"_blank\">Alpha</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Grabbed 1\n",
      "Grabbed 2\n",
      "Grabbed 3\n",
      "Grabbed 4\n",
      "Grabbed 5\n",
      "Grabbed 6\n",
      "Grabbed 7\n",
      "Grabbed 8\n",
      "Grabbed 9\n",
      "Grabbed 10\n",
      "Grabbed 11\n",
      "Grabbed 12\n",
      "Grabbed 13\n",
      "Grabbed 14\n",
      "Grabbed 15\n",
      "Grabbed 16\n",
      "Grabbed 17\n",
      "Grabbed 18\n",
      "Grabbed 19\n",
      "Grabbed 20\n",
      "Grabbed 21\n",
      "Grabbed 22\n",
      "Grabbed 23\n",
      "Grabbed 24\n",
      "Grabbed 25\n",
      "Grabbed 26\n",
      "Grabbed 27\n",
      "Grabbed 28\n",
      "Grabbed 29\n",
      "Grabbed 30\n",
      "Grabbed 31\n",
      "Grabbed 32\n",
      "Grabbed 33\n",
      "Grabbed 34\n",
      "Grabbed 35\n",
      "Grabbed 36\n",
      "Grabbed 37\n",
      "Grabbed 38\n",
      "Grabbed 39\n",
      "Grabbed 40\n",
      "Grabbed 41\n",
      "Grabbed 42\n",
      "Grabbed 43\n",
      "Grabbed 44\n",
      "Grabbed 45\n",
      "Grabbed 46\n",
      "Grabbed 47\n",
      "Grabbed 48\n",
      "Grabbed 49\n",
      "Grabbed 50\n",
      "Grabbed 51\n",
      "Grabbed 52\n",
      "Grabbed 53\n",
      "Grabbed 54\n",
      "Grabbed 55\n",
      "Grabbed 56\n",
      "Grabbed 57\n",
      "Grabbed 58\n",
      "Grabbed 59\n",
      "Grabbed 60\n",
      "Grabbed 61\n",
      "Grabbed 62\n",
      "Grabbed 63\n",
      "Grabbed 64\n",
      "Grabbed 65\n",
      "Grabbed 66\n",
      "Grabbed 67\n",
      "Grabbed 68\n",
      "Grabbed 69\n",
      "Grabbed 70\n",
      "Grabbed 71\n",
      "Grabbed 72\n",
      "Grabbed 73\n",
      "Grabbed 74\n",
      "Grabbed 75\n",
      "Grabbed 76\n",
      "Grabbed 77\n",
      "Grabbed 78\n",
      "Grabbed 79\n",
      "Grabbed 80\n",
      "Grabbed 81\n",
      "Grabbed 82\n",
      "Grabbed 83\n",
      "Grabbed 84\n",
      "Grabbed 85\n",
      "Grabbed 86\n",
      "Grabbed 87\n",
      "Grabbed 88\n",
      "Grabbed 89\n",
      "Grabbed 90\n",
      "Grabbed 91\n",
      "Grabbed 92\n",
      "Grabbed 93\n",
      "Grabbed 94\n",
      "Grabbed 95\n",
      "Grabbed 96\n",
      "Grabbed 97\n",
      "Grabbed 98\n",
      "Grabbed 99\n",
      "Grabbed 100\n",
      "Grabbed 101\n",
      "Grabbed 102\n",
      "Grabbed 103\n",
      "Grabbed 104\n",
      "Grabbed 105\n",
      "Grabbed 106\n",
      "Grabbed 107\n",
      "Grabbed 108\n",
      "Grabbed 109\n",
      "Grabbed 110\n",
      "Grabbed 111\n",
      "Grabbed 112\n",
      "Grabbed 113\n",
      "Grabbed 114\n",
      "Grabbed 115\n",
      "Grabbed 116\n",
      "Grabbed 117\n",
      "Grabbed 118\n",
      "Grabbed 119\n",
      "Grabbed 120\n",
      "Grabbed 121\n",
      "Grabbed 122\n",
      "Grabbed 123\n",
      "Grabbed 124\n",
      "Grabbed 125\n",
      "Grabbed 126\n",
      "Grabbed 127\n",
      "Grabbed 128\n",
      "Grabbed 129\n",
      "Grabbed 130\n",
      "Grabbed 131\n",
      "Grabbed 132\n",
      "Grabbed 133\n",
      "Grabbed 134\n",
      "Grabbed 135\n",
      "Grabbed 136\n",
      "Grabbed 137\n",
      "Grabbed 138\n",
      "Grabbed 139\n",
      "Grabbed 140\n",
      "Grabbed 141\n",
      "Grabbed 142\n",
      "Grabbed 143\n",
      "Grabbed 144\n",
      "Grabbed 145\n",
      "Grabbed 146\n",
      "Grabbed 147\n",
      "Grabbed 148\n",
      "Grabbed 149\n",
      "Grabbed 150\n",
      "Grabbed 151\n",
      "Grabbed 152\n",
      "Grabbed 153\n",
      "Grabbed 154\n",
      "Grabbed 155\n",
      "Grabbed 156\n",
      "Grabbed 157\n",
      "Grabbed 158\n",
      "Grabbed 159\n",
      "Grabbed 160\n",
      "Grabbed 161\n",
      "Grabbed 162\n",
      "Grabbed 163\n",
      "Grabbed 164\n",
      "Grabbed 165\n",
      "Grabbed 166\n",
      "Grabbed 167\n",
      "Grabbed 168\n",
      "Grabbed 169\n",
      "Grabbed 170\n",
      "Grabbed 171\n",
      "Grabbed 172\n",
      "Grabbed 173\n",
      "Grabbed 174\n",
      "Grabbed 175\n",
      "Grabbed 176\n",
      "Grabbed 177\n",
      "Grabbed 178\n",
      "Grabbed 179\n",
      "Grabbed 180\n",
      "Grabbed 181\n",
      "Grabbed 182\n",
      "Grabbed 183\n",
      "Grabbed 184\n",
      "Grabbed 185\n",
      "Grabbed 186\n",
      "Grabbed 187\n",
      "Grabbed 188\n",
      "Grabbed 189\n",
      "Grabbed 190\n",
      "Grabbed 191\n",
      "Grabbed 192\n",
      "Grabbed 193\n",
      "Grabbed 194\n",
      "Grabbed 195\n",
      "Grabbed 196\n",
      "Grabbed 197\n",
      "Grabbed 198\n",
      "Grabbed 199\n",
      "Grabbed 200\n",
      "Grabbed 201\n",
      "Grabbed 202\n",
      "Grabbed 203\n",
      "Grabbed 204\n",
      "Grabbed 205\n",
      "Grabbed 206\n",
      "Grabbed 207\n",
      "Grabbed 208\n",
      "Grabbed 209\n",
      "Grabbed 210\n",
      "Grabbed 211\n",
      "Grabbed 212\n",
      "Grabbed 213\n",
      "Grabbed 214\n",
      "Grabbed 215\n",
      "Grabbed 216\n",
      "Grabbed 217\n",
      "Grabbed 218\n",
      "Grabbed 219\n",
      "Grabbed 220\n",
      "Grabbed 221\n",
      "Grabbed 222\n",
      "Grabbed 223\n",
      "Grabbed 224\n",
      "Grabbed 225\n",
      "Grabbed 226\n",
      "Grabbed 227\n",
      "Grabbed 228\n",
      "Grabbed 229\n",
      "Grabbed 230\n",
      "Grabbed 231\n",
      "Grabbed 232\n",
      "Grabbed 233\n",
      "Grabbed 234\n",
      "Grabbed 235\n",
      "Grabbed 236\n",
      "Grabbed 237\n",
      "Grabbed 238\n",
      "Grabbed 239\n",
      "Grabbed 240\n",
      "Grabbed 241\n",
      "Grabbed 242\n",
      "Grabbed 243\n",
      "Grabbed 244\n",
      "Grabbed 245\n",
      "Grabbed 246\n",
      "Grabbed 247\n",
      "Grabbed 248\n",
      "Grabbed 249\n",
      "Grabbed 250\n",
      "Grabbed 251\n",
      "Grabbed 252\n",
      "Grabbed 253\n",
      "Grabbed 254\n",
      "Grabbed 255\n",
      "Grabbed 256\n",
      "Grabbed 257\n",
      "Grabbed 258\n",
      "Grabbed 259\n",
      "Grabbed 260\n",
      "Grabbed 261\n",
      "Grabbed 262\n",
      "Grabbed 263\n",
      "Grabbed 264\n",
      "Grabbed 265\n",
      "Grabbed 266\n",
      "Grabbed 267\n",
      "Grabbed 268\n",
      "Grabbed 269\n",
      "Grabbed 270\n",
      "Grabbed 271\n",
      "Grabbed 272\n",
      "Grabbed 273\n",
      "Grabbed 274\n",
      "Grabbed 275\n",
      "Grabbed 276\n",
      "Grabbed 277\n",
      "Grabbed 278\n",
      "Grabbed 279\n",
      "Grabbed 280\n",
      "Grabbed 281\n",
      "Grabbed 282\n",
      "Grabbed 283\n",
      "Grabbed 284\n",
      "Grabbed 285\n",
      "Grabbed 286\n",
      "Grabbed 287\n",
      "Grabbed 288\n",
      "Grabbed 289\n",
      "Grabbed 290\n",
      "Grabbed 291\n",
      "Grabbed 292\n",
      "Grabbed 293\n",
      "Grabbed 294\n",
      "Grabbed 295\n",
      "Grabbed 296\n",
      "Grabbed 297\n",
      "Grabbed 298\n",
      "Grabbed 299\n",
      "Grabbed 300\n",
      "Grabbed 301\n",
      "Grabbed 302\n",
      "Grabbed 303\n",
      "Grabbed 304\n",
      "Grabbed 305\n",
      "Grabbed 306\n",
      "Grabbed 307\n",
      "Grabbed 308\n",
      "Grabbed 309\n",
      "Grabbed 310\n",
      "Grabbed 311\n",
      "Grabbed 312\n",
      "Grabbed 313\n",
      "Grabbed 314\n",
      "Grabbed 315\n",
      "Grabbed 316\n",
      "Grabbed 317\n",
      "Grabbed 318\n",
      "Grabbed 319\n",
      "Grabbed 320\n",
      "Grabbed 321\n",
      "Grabbed 322\n",
      "Grabbed 323\n",
      "Grabbed 324\n",
      "Grabbed 325\n",
      "Grabbed 326\n",
      "Grabbed 327\n",
      "Grabbed 328\n",
      "Grabbed 329\n",
      "Grabbed 330\n",
      "Grabbed 331\n",
      "Grabbed 332\n",
      "Grabbed 333\n",
      "Grabbed 334\n",
      "Grabbed 335\n",
      "Grabbed 336\n",
      "Grabbed 337\n",
      "Grabbed 338\n",
      "Grabbed 339\n",
      "Grabbed 340\n",
      "Grabbed 341\n",
      "Grabbed 342\n",
      "Grabbed 343\n",
      "Grabbed 344\n",
      "Grabbed 345\n",
      "Grabbed 346\n",
      "Grabbed 347\n",
      "Grabbed 348\n",
      "Grabbed 349\n",
      "Grabbed 350\n",
      "Grabbed 351\n",
      "Grabbed 352\n",
      "Grabbed 353\n",
      "Grabbed 354\n",
      "Grabbed 355\n",
      "Grabbed 356\n",
      "Grabbed 357\n",
      "Grabbed 358\n",
      "Grabbed 359\n",
      "Grabbed 360\n",
      "Grabbed 361\n",
      "Grabbed 362\n",
      "Grabbed 363\n",
      "Grabbed 364\n",
      "Grabbed 365\n",
      "Grabbed 366\n",
      "Grabbed 367\n",
      "Grabbed 368\n",
      "Grabbed 369\n",
      "Grabbed 370\n",
      "Grabbed 371\n",
      "Grabbed 372\n",
      "Grabbed 373\n",
      "Grabbed 374\n",
      "Grabbed 375\n",
      "Grabbed 376\n",
      "Grabbed 377\n",
      "Grabbed 378\n",
      "Grabbed 379\n",
      "Grabbed 380\n",
      "Grabbed 381\n",
      "Grabbed 382\n",
      "Grabbed 383\n",
      "Grabbed 384\n",
      "Grabbed 385\n",
      "Grabbed 386\n",
      "Grabbed 387\n",
      "Grabbed 388\n",
      "Grabbed 389\n",
      "Grabbed 390\n",
      "Grabbed 391\n",
      "Grabbed 392\n",
      "Grabbed 393\n",
      "Grabbed 394\n",
      "Grabbed 395\n",
      "Grabbed 396\n",
      "Grabbed 397\n",
      "Grabbed 398\n",
      "Grabbed 399\n",
      "Grabbed 400\n",
      "Grabbed 401\n",
      "Grabbed 402\n",
      "Grabbed 403\n",
      "Grabbed 404\n",
      "Grabbed 405\n",
      "Grabbed 406\n",
      "Grabbed 407\n",
      "Grabbed 408\n",
      "Grabbed 409\n",
      "Grabbed 410\n",
      "Grabbed 411\n",
      "Grabbed 412\n",
      "Grabbed 413\n",
      "Grabbed 414\n",
      "Grabbed 415\n",
      "Grabbed 416\n",
      "Grabbed 417\n",
      "Grabbed 418\n",
      "Grabbed 419\n",
      "Grabbed 420\n",
      "Grabbed 421\n",
      "Grabbed 422\n",
      "Grabbed 423\n",
      "Grabbed 424\n",
      "Grabbed 425\n",
      "Grabbed 426\n",
      "Grabbed 427\n",
      "Grabbed 428\n",
      "Grabbed 429\n",
      "Grabbed 430\n",
      "Grabbed 431\n",
      "Grabbed 432\n",
      "Grabbed 433\n",
      "Grabbed 434\n",
      "Grabbed 435\n",
      "Grabbed 436\n",
      "Grabbed 437\n",
      "Grabbed 438\n",
      "Grabbed 439\n",
      "Grabbed 440\n",
      "Grabbed 441\n",
      "Grabbed 442\n",
      "Grabbed 443\n",
      "Grabbed 444\n",
      "Grabbed 445\n",
      "Grabbed 446\n",
      "Grabbed 447\n",
      "Grabbed 448\n",
      "Grabbed 449\n",
      "Grabbed 450\n",
      "Grabbed 451\n",
      "Grabbed 452\n",
      "Grabbed 453\n",
      "Grabbed 454\n",
      "Grabbed 455\n",
      "Grabbed 456\n",
      "Grabbed 457\n",
      "Grabbed 458\n",
      "Grabbed 459\n",
      "Grabbed 460\n",
      "Grabbed 461\n",
      "Grabbed 462\n",
      "Grabbed 463\n",
      "Grabbed 464\n",
      "Grabbed 465\n",
      "Grabbed 466\n",
      "Grabbed 467\n",
      "Grabbed 468\n",
      "Grabbed 469\n",
      "Grabbed 470\n",
      "Grabbed 471\n",
      "Grabbed 472\n",
      "Grabbed 473\n",
      "Grabbed 474\n",
      "Grabbed 475\n",
      "Grabbed 476\n",
      "Grabbed 477\n",
      "Grabbed 478\n",
      "Grabbed 479\n",
      "Grabbed 480\n",
      "Grabbed 481\n",
      "Grabbed 482\n",
      "Grabbed 483\n",
      "Grabbed 484\n",
      "Grabbed 485\n",
      "Grabbed 486\n",
      "Grabbed 487\n",
      "Grabbed 488\n",
      "Grabbed 489\n",
      "Grabbed 490\n",
      "Grabbed 491\n",
      "Grabbed 492\n",
      "Grabbed 493\n",
      "Grabbed 494\n",
      "Grabbed 495\n",
      "Grabbed 496\n",
      "Grabbed 497\n",
      "Grabbed 498\n",
      "Grabbed 499\n",
      "Done!\n"
     ]
    }
   ],
   "source": [
    "cardsToScrape = 500\n",
    "\n",
    "for i in xrange(1, cardsToScrape):\n",
    "    if (i % 1 == 0): print \"Grabbed \" + str(i)\n",
    "    grabCard(i)\n",
    "\n",
    "print \"Done!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, we now have roughly `cardsToScrape` cards and associated values in a local `dict` using the `cardName` as the key. (Note that we have less than `cardsToScrape` as we're iterating over `multiverse_id`s and some ids don't actually match to a card page.)\n",
    "\n",
    "### Note for potential future work\n",
    "\n",
    "*There are other aspects represented on the Gatherer database such as set and community ratings but we leave this to future work. Annoyingly, for cards in multiple sets, the card will have a different page (and subsequently different set of ratings) for each set; though this would require more work, it'd be super interesting if you could predict a card's community interest (# ratings) and favorability (average rating).*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making the data usable\n",
    "\n",
    "We'll now put this into a `pandas` dataframe for cleaning, variable creation and initial analysis/spot checking/understanding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 251,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>cardType</th>\n",
       "      <th>fullCardText</th>\n",
       "      <th>toughness</th>\n",
       "      <th>power</th>\n",
       "      <th>cardText</th>\n",
       "      <th>manaCost</th>\n",
       "      <th>cmc</th>\n",
       "      <th>rarity</th>\n",
       "      <th>cardName</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Air Elemental</th>\n",
       "      <td>[Creature, Elemental]</td>\n",
       "      <td>flying</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>[flying]</td>\n",
       "      <td>[3, Blue, Blue]</td>\n",
       "      <td>5</td>\n",
       "      <td>uncommon</td>\n",
       "      <td>Air Elemental</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ancestral Recall</th>\n",
       "      <td>[Instant]</td>\n",
       "      <td>target player draws three cards.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[target player draws three cards.]</td>\n",
       "      <td>[Blue]</td>\n",
       "      <td>1</td>\n",
       "      <td>rare</td>\n",
       "      <td>Ancestral Recall</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Animate Artifact</th>\n",
       "      <td>[Enchantment, Aura]</td>\n",
       "      <td>enchant artifactas long as enchanted artifact ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[enchant artifact, as long as enchanted artifa...</td>\n",
       "      <td>[3, Blue]</td>\n",
       "      <td>4</td>\n",
       "      <td>uncommon</td>\n",
       "      <td>Animate Artifact</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Animate Dead</th>\n",
       "      <td>[Enchantment, Aura]</td>\n",
       "      <td>enchant creature card in a graveyardwhen anima...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[enchant creature card in a graveyard, when an...</td>\n",
       "      <td>[1, Black]</td>\n",
       "      <td>2</td>\n",
       "      <td>uncommon</td>\n",
       "      <td>Animate Dead</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Animate Wall</th>\n",
       "      <td>[Enchantment, Aura]</td>\n",
       "      <td>enchant wallenchanted wall can attack as thoug...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[enchant wall, enchanted wall can attack as th...</td>\n",
       "      <td>[White]</td>\n",
       "      <td>1</td>\n",
       "      <td>rare</td>\n",
       "      <td>Animate Wall</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Winter Orb</th>\n",
       "      <td>[Artifact]</td>\n",
       "      <td>players can't untap more than one land during ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[players can't untap more than one land during...</td>\n",
       "      <td>[2]</td>\n",
       "      <td>2</td>\n",
       "      <td>rare</td>\n",
       "      <td>Winter Orb</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wooden Sphere</th>\n",
       "      <td>[Artifact]</td>\n",
       "      <td>whenever a player casts a green spell, you may...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[whenever a player casts a green spell, you ma...</td>\n",
       "      <td>[1]</td>\n",
       "      <td>1</td>\n",
       "      <td>uncommon</td>\n",
       "      <td>Wooden Sphere</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Word of Command</th>\n",
       "      <td>[Instant]</td>\n",
       "      <td>look at target opponent's hand and choose a ca...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[look at target opponent's hand and choose a c...</td>\n",
       "      <td>[Black, Black]</td>\n",
       "      <td>2</td>\n",
       "      <td>rare</td>\n",
       "      <td>Word of Command</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Wrath of God</th>\n",
       "      <td>[Sorcery]</td>\n",
       "      <td>destroy all creatures. they can't be regenerated.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[destroy all creatures. they can't be regenera...</td>\n",
       "      <td>[2, White, White]</td>\n",
       "      <td>4</td>\n",
       "      <td>rare</td>\n",
       "      <td>Wrath of God</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Zombie Master</th>\n",
       "      <td>[Creature, Zombie]</td>\n",
       "      <td>other zombie creatures have swampwalk.other zo...</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>[other zombie creatures have swampwalk., other...</td>\n",
       "      <td>[1, Black, Black]</td>\n",
       "      <td>3</td>\n",
       "      <td>rare</td>\n",
       "      <td>Zombie Master</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>290 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                               cardType  \\\n",
       "Air Elemental     [Creature, Elemental]   \n",
       "Ancestral Recall              [Instant]   \n",
       "Animate Artifact    [Enchantment, Aura]   \n",
       "Animate Dead        [Enchantment, Aura]   \n",
       "Animate Wall        [Enchantment, Aura]   \n",
       "...                                 ...   \n",
       "Winter Orb                   [Artifact]   \n",
       "Wooden Sphere                [Artifact]   \n",
       "Word of Command               [Instant]   \n",
       "Wrath of God                  [Sorcery]   \n",
       "Zombie Master        [Creature, Zombie]   \n",
       "\n",
       "                                                       fullCardText  \\\n",
       "Air Elemental                                                flying   \n",
       "Ancestral Recall                   target player draws three cards.   \n",
       "Animate Artifact  enchant artifactas long as enchanted artifact ...   \n",
       "Animate Dead      enchant creature card in a graveyardwhen anima...   \n",
       "Animate Wall      enchant wallenchanted wall can attack as thoug...   \n",
       "...                                                             ...   \n",
       "Winter Orb        players can't untap more than one land during ...   \n",
       "Wooden Sphere     whenever a player casts a green spell, you may...   \n",
       "Word of Command   look at target opponent's hand and choose a ca...   \n",
       "Wrath of God      destroy all creatures. they can't be regenerated.   \n",
       "Zombie Master     other zombie creatures have swampwalk.other zo...   \n",
       "\n",
       "                  toughness  power  \\\n",
       "Air Elemental             4      4   \n",
       "Ancestral Recall        NaN    NaN   \n",
       "Animate Artifact        NaN    NaN   \n",
       "Animate Dead            NaN    NaN   \n",
       "Animate Wall            NaN    NaN   \n",
       "...                     ...    ...   \n",
       "Winter Orb              NaN    NaN   \n",
       "Wooden Sphere           NaN    NaN   \n",
       "Word of Command         NaN    NaN   \n",
       "Wrath of God            NaN    NaN   \n",
       "Zombie Master             3      2   \n",
       "\n",
       "                                                           cardText  \\\n",
       "Air Elemental                                              [flying]   \n",
       "Ancestral Recall                 [target player draws three cards.]   \n",
       "Animate Artifact  [enchant artifact, as long as enchanted artifa...   \n",
       "Animate Dead      [enchant creature card in a graveyard, when an...   \n",
       "Animate Wall      [enchant wall, enchanted wall can attack as th...   \n",
       "...                                                             ...   \n",
       "Winter Orb        [players can't untap more than one land during...   \n",
       "Wooden Sphere     [whenever a player casts a green spell, you ma...   \n",
       "Word of Command   [look at target opponent's hand and choose a c...   \n",
       "Wrath of God      [destroy all creatures. they can't be regenera...   \n",
       "Zombie Master     [other zombie creatures have swampwalk., other...   \n",
       "\n",
       "                           manaCost  cmc    rarity          cardName  \n",
       "Air Elemental       [3, Blue, Blue]    5  uncommon     Air Elemental  \n",
       "Ancestral Recall             [Blue]    1      rare  Ancestral Recall  \n",
       "Animate Artifact          [3, Blue]    4  uncommon  Animate Artifact  \n",
       "Animate Dead             [1, Black]    2  uncommon      Animate Dead  \n",
       "Animate Wall                [White]    1      rare      Animate Wall  \n",
       "...                             ...  ...       ...               ...  \n",
       "Winter Orb                      [2]    2      rare        Winter Orb  \n",
       "Wooden Sphere                   [1]    1  uncommon     Wooden Sphere  \n",
       "Word of Command      [Black, Black]    2      rare   Word of Command  \n",
       "Wrath of God      [2, White, White]    4      rare      Wrath of God  \n",
       "Zombie Master     [1, Black, Black]    3      rare     Zombie Master  \n",
       "\n",
       "[290 rows x 9 columns]"
      ]
     },
     "execution_count": 251,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# START\n",
    "\n",
    "data = pd.DataFrame.from_dict(database, orient='index')\n",
    "data['cardName'] = data.index\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (2) -- Feature extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "source": [
    "Based on our domain knowledge, we're going to extract four main types of features for each card:\n",
    "1. **Mana cost** and **mana amounts** of a card\n",
    "2. **Categorical features** -- type (i.e. Artifact, Creature, etc.) and rarity (i.e. Common, Uncommon, etc.)\n",
    "3. **Text features** based on the card's text (i.e. \"When this creature enters the battlefield...\")\n",
    "4. **Functional features** -- having a Tap ability, being a mana generator, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 252,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Which features do we want to use?\n",
    "# All enabled by default, mana and categorical features required\n",
    "\n",
    "textFeatures = True\n",
    "automatedTextFeatures = False\n",
    "functionalFeatures = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (2.1) -- Mana features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 253,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create mana features\n",
    "\n",
    "colorlessMana = []\n",
    "colorless = []\n",
    "\n",
    "for row in data['manaCost']:\n",
    "    found = 0\n",
    "    for val in row:\n",
    "        if isInt(val):\n",
    "            colorlessMana.append(float(val))\n",
    "            found = 1\n",
    "    if found == 0:\n",
    "        colorlessMana.append(0)\n",
    "\n",
    "data['colorlessMana'] = colorlessMana \n",
    "data['Variable Colorless'] = [1 if 'Variable Colorless' in text else 0 for text in data['manaCost']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 254,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Count mana symbols\n",
    "\n",
    "manaSymbols = []\n",
    "\n",
    "manaSymbols = ['Blue', 'Black', 'Red', 'Green', 'White']\n",
    "manaVars = ['mana_' + _ for _ in manaSymbols]\n",
    "\n",
    "for i in xrange(len(manaSymbols)):\n",
    "    data[manaVars[i]] = [text.count(manaSymbols[i]) for text in data['manaCost']]\n",
    "    data[manaSymbols[i]] = [1 if text.count(manaSymbols[i]) > 0 else 0 for text in data['manaCost']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 255,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Artifact</th>\n",
       "      <th>Black</th>\n",
       "      <th>Blue</th>\n",
       "      <th>Green</th>\n",
       "      <th>Red</th>\n",
       "      <th>Variable Colorless</th>\n",
       "      <th>White</th>\n",
       "      <th>cmc</th>\n",
       "      <th>colorlessMana</th>\n",
       "      <th>mana_Black</th>\n",
       "      <th>mana_Blue</th>\n",
       "      <th>mana_Green</th>\n",
       "      <th>mana_Red</th>\n",
       "      <th>mana_White</th>\n",
       "      <th>power</th>\n",
       "      <th>toughness</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>color</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">Artifact</th>\n",
       "      <th>count</th>\n",
       "      <td>61</td>\n",
       "      <td>61</td>\n",
       "      <td>61</td>\n",
       "      <td>61</td>\n",
       "      <td>61</td>\n",
       "      <td>61</td>\n",
       "      <td>61</td>\n",
       "      <td>47.000000</td>\n",
       "      <td>61.000000</td>\n",
       "      <td>61</td>\n",
       "      <td>61</td>\n",
       "      <td>61</td>\n",
       "      <td>61</td>\n",
       "      <td>61</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.361702</td>\n",
       "      <td>1.819672</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.400000</td>\n",
       "      <td>5.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.673652</td>\n",
       "      <td>1.774901</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2.302173</td>\n",
       "      <td>1.414214</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">White</th>\n",
       "      <th>min</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1.500000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>4.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>6.000000</td>\n",
       "      <td>6.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>48 rows × 16 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                Artifact  Black  Blue  Green  Red  Variable Colorless  White  \\\n",
       "color                                                                          \n",
       "Artifact count        61     61    61     61   61                  61     61   \n",
       "         mean          1      0     0      0    0                   0      0   \n",
       "         std           0      0     0      0    0                   0      0   \n",
       "         min           1      0     0      0    0                   0      0   \n",
       "         25%           1      0     0      0    0                   0      0   \n",
       "...                  ...    ...   ...    ...  ...                 ...    ...   \n",
       "White    min           0      0     0      0    0                   0      1   \n",
       "         25%           0      0     0      0    0                   0      1   \n",
       "         50%           0      0     0      0    0                   0      1   \n",
       "         75%           0      0     0      0    0                   0      1   \n",
       "         max           0      0     0      0    0                   1      1   \n",
       "\n",
       "                      cmc  colorlessMana  mana_Black  mana_Blue  mana_Green  \\\n",
       "color                                                                         \n",
       "Artifact count  47.000000      61.000000          61         61          61   \n",
       "         mean    2.361702       1.819672           0          0           0   \n",
       "         std     1.673652       1.774901           0          0           0   \n",
       "         min     0.000000       0.000000           0          0           0   \n",
       "         25%     1.000000       0.000000           0          0           0   \n",
       "...                   ...            ...         ...        ...         ...   \n",
       "White    min     1.000000       0.000000           0          0           0   \n",
       "         25%     1.000000       0.000000           0          0           0   \n",
       "         50%     2.000000       0.000000           0          0           0   \n",
       "         75%     3.000000       2.000000           0          0           0   \n",
       "         max     6.000000       3.000000           0          0           0   \n",
       "\n",
       "                mana_Red  mana_White     power  toughness  \n",
       "color                                                      \n",
       "Artifact count        61          61  5.000000   5.000000  \n",
       "         mean          0           0  2.400000   5.000000  \n",
       "         std           0           0  2.302173   1.414214  \n",
       "         min           0           0  0.000000   3.000000  \n",
       "         25%           0           0  0.000000   4.000000  \n",
       "...                  ...         ...       ...        ...  \n",
       "White    min           0           1  1.000000   1.000000  \n",
       "         25%           0           1  1.500000   1.000000  \n",
       "         50%           0           1  2.000000   2.000000  \n",
       "         75%           0           2  3.000000   4.500000  \n",
       "         max           0           3  6.000000   6.000000  \n",
       "\n",
       "[48 rows x 16 columns]"
      ]
     },
     "execution_count": 255,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find color (ignores multicolor)\n",
    "\n",
    "def isColorless(l):\n",
    "    for val in l:\n",
    "        if val in manaSymbols: return False\n",
    "    return True\n",
    "\n",
    "data['Artifact'] = [1 if isColorless(x) else 0 for x in data['manaCost']]\n",
    "\n",
    "def findColor(l):\n",
    "    for val in l:\n",
    "        if not isInt(val) and val != 'Variable Colorless': return val\n",
    "    return 'Artifact'\n",
    "\n",
    "data['color'] = [findColor(l) for l in data['manaCost']]\n",
    "\n",
    "data.groupby(data['color']).describe().to_csv('colorSummary.csv')\n",
    "data.groupby(data['color']).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (2b) -- Categorical features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 256,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>toughness</th>\n",
       "      <th>power</th>\n",
       "      <th>cmc</th>\n",
       "      <th>colorlessMana</th>\n",
       "      <th>Variable Colorless</th>\n",
       "      <th>mana_Blue</th>\n",
       "      <th>Blue</th>\n",
       "      <th>mana_Black</th>\n",
       "      <th>Black</th>\n",
       "      <th>mana_Red</th>\n",
       "      <th>...</th>\n",
       "      <th>Artifact</th>\n",
       "      <th>Creature</th>\n",
       "      <th>Enchantment</th>\n",
       "      <th>Instant</th>\n",
       "      <th>Land</th>\n",
       "      <th>Sorcery</th>\n",
       "      <th>basic land</th>\n",
       "      <th>common</th>\n",
       "      <th>rare</th>\n",
       "      <th>uncommon</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Primary Type</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">Artifact</th>\n",
       "      <th>count</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>43.00</td>\n",
       "      <td>43.00</td>\n",
       "      <td>43</td>\n",
       "      <td>...</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>43</td>\n",
       "      <td>43.00</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>43.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>2.116279</td>\n",
       "      <td>2.116279</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.604651</td>\n",
       "      <td>0.395349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.499354</td>\n",
       "      <td>1.499354</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.494712</td>\n",
       "      <td>0.494712</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">Sorcery</th>\n",
       "      <th>min</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.250000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.75</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>3.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>48 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                    toughness  power        cmc  colorlessMana  \\\n",
       "Primary Type                                                     \n",
       "Artifact     count          1      1  43.000000      43.000000   \n",
       "             mean           6      3   2.116279       2.116279   \n",
       "             std          NaN    NaN   1.499354       1.499354   \n",
       "             min            6      3   0.000000       0.000000   \n",
       "             25%            6      3   1.000000       1.000000   \n",
       "...                       ...    ...        ...            ...   \n",
       "Sorcery      min          NaN    NaN   1.000000       0.000000   \n",
       "             25%          NaN    NaN   1.250000       0.000000   \n",
       "             50%          NaN    NaN   2.000000       1.000000   \n",
       "             75%          NaN    NaN   3.000000       2.000000   \n",
       "             max          NaN    NaN   4.000000       3.000000   \n",
       "\n",
       "                    Variable Colorless  mana_Blue  Blue  mana_Black  Black  \\\n",
       "Primary Type                                                                 \n",
       "Artifact     count                  43         43    43       43.00  43.00   \n",
       "             mean                    0          0     0        0.00   0.00   \n",
       "             std                     0          0     0        0.00   0.00   \n",
       "             min                     0          0     0        0.00   0.00   \n",
       "             25%                     0          0     0        0.00   0.00   \n",
       "...                                ...        ...   ...         ...    ...   \n",
       "Sorcery      min                     0          0     0        0.00   0.00   \n",
       "             25%                     0          0     0        0.00   0.00   \n",
       "             50%                     0          0     0        0.00   0.00   \n",
       "             75%                     1          0     0        0.75   0.75   \n",
       "             max                     1          3     1        3.00   1.00   \n",
       "\n",
       "                    mana_Red    ...      Artifact  Creature  Enchantment  \\\n",
       "Primary Type                    ...                                        \n",
       "Artifact     count        43    ...            43        43           43   \n",
       "             mean          0    ...             1         0            0   \n",
       "             std           0    ...             0         0            0   \n",
       "             min           0    ...             1         0            0   \n",
       "             25%           0    ...             1         0            0   \n",
       "...                      ...    ...           ...       ...          ...   \n",
       "Sorcery      min           0    ...             0         0            0   \n",
       "             25%           0    ...             0         0            0   \n",
       "             50%           0    ...             0         0            0   \n",
       "             75%           0    ...             0         0            0   \n",
       "             max           1    ...             0         0            0   \n",
       "\n",
       "                    Instant  Land  Sorcery  basic land  common       rare  \\\n",
       "Primary Type                                                                \n",
       "Artifact     count       43    43       43          43   43.00  43.000000   \n",
       "             mean         0     0        0           0    0.00   0.604651   \n",
       "             std          0     0        0           0    0.00   0.494712   \n",
       "             min          0     0        0           0    0.00   0.000000   \n",
       "             25%          0     0        0           0    0.00   0.000000   \n",
       "...                     ...   ...      ...         ...     ...        ...   \n",
       "Sorcery      min          0     0        1           0    0.00   0.000000   \n",
       "             25%          0     0        1           0    0.00   0.000000   \n",
       "             50%          0     0        1           0    0.00   0.000000   \n",
       "             75%          0     0        1           0    0.75   1.000000   \n",
       "             max          0     0        1           0    1.00   1.000000   \n",
       "\n",
       "                     uncommon  \n",
       "Primary Type                   \n",
       "Artifact     count  43.000000  \n",
       "             mean    0.395349  \n",
       "             std     0.494712  \n",
       "             min     0.000000  \n",
       "             25%     0.000000  \n",
       "...                       ...  \n",
       "Sorcery      min     0.000000  \n",
       "             25%     0.000000  \n",
       "             50%     0.000000  \n",
       "             75%     0.750000  \n",
       "             max     1.000000  \n",
       "\n",
       "[48 rows x 26 columns]"
      ]
     },
     "execution_count": 256,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create categorical features\n",
    "    \n",
    "primaryTypes = [cardType[0] for cardType in data['cardType']]\n",
    "\n",
    "for i in xrange(len(primaryTypes)):\n",
    "    if primaryTypes[i] == u'Basic Land':\n",
    "        primaryTypes[i] = u'Land'\n",
    "    if primaryTypes[i] == u'Artifact Creature':\n",
    "        primaryTypes[i] = u'Creature'\n",
    "\n",
    "data['Primary Type'] = primaryTypes\n",
    "            \n",
    "data = pd.concat([data, pd.get_dummies(data['Primary Type'])], axis=1)\n",
    "data = pd.concat([data, pd.get_dummies(data['rarity'])], axis=1)\n",
    "\n",
    "data.groupby(data['rarity']).describe().to_csv('byRarity.csv')\n",
    "data.groupby(data['rarity']).describe()\n",
    "    \n",
    "data.groupby(data['Primary Type']).describe().to_csv('byType.csv')\n",
    "data.groupby(data['Primary Type']).describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (2c) -- Text features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A helper function from `fuzzywuzzy` to find partial word matches in card text boxes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 257,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def partialMatch(s, l, threshold=95):\n",
    "    fuzzVals = [fuzz.partial_ratio(s, x) for x in l]\n",
    "    if not fuzzVals: fuzzVals = [0]\n",
    "    return max(fuzzVals) >= threshold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on domain knowledge, we'll fuzzy match if certain important words are in a card's text box that will give us a hint of what the card does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 258,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Create text-based features\n",
    "\n",
    "if textFeatures:\n",
    "\n",
    "    data['Damage'] = [1 if partialMatch('damage', l) else 0 for l in data['cardText']]\n",
    "    data['Hand'] = [1 if partialMatch('hand', l) else 0 for l in data['cardText']]\n",
    "    data['Draw'] = [1 if partialMatch('draw', l, 80) else 0 for l in data['cardText']]\n",
    "    data['Upkeep'] = [1 if partialMatch('draw', l, 80) else 0 for l in data['cardText']]\n",
    "    data['Library'] = [1 if partialMatch('library', l) else 0 for l in data['cardText']]\n",
    "    data['Sacrifice'] = [1 if partialMatch('sacrifice', l) else 0 for l in data['cardText']]\n",
    "    data['Destroy'] = [1 if partialMatch('destroy', l) else 0 for l in data['cardText']]\n",
    "    data['Discard'] = [1 if partialMatch('discard', l) else 0 for l in data['cardText']]\n",
    "    data['Prevent'] = [1 if partialMatch('prevent', l) else 0 for l in data['cardText']]\n",
    "    data['Life'] = [1 if partialMatch('life', l) else 0 for l in data['cardText']]\n",
    "    data['Attack'] = [1 if partialMatch('attack', l) else 0 for l in data['cardText']]\n",
    "    data['Block'] = [1 if partialMatch('block', l) else 0 for l in data['cardText']]\n",
    "    data['Search'] = [1 if partialMatch('search', l) else 0 for l in data['cardText']]\n",
    "    data['Choose'] = [1 if partialMatch('choose', l) else 0 for l in data['cardText']]\n",
    "    data['Copy'] = [1 if partialMatch('copy', l) else 0 for l in data['cardText']]\n",
    "    data['Change'] = [1 if partialMatch('change', l) else 0 for l in data['cardText']]\n",
    "    data['Turn'] = [1 if partialMatch('turn', l) else 0 for l in data['cardText']]\n",
    "    data['End of turn'] = [1 if partialMatch('end of turn', l, 80) else 0 for l in data['cardText']]\n",
    "    data['Beginning of turn'] = [1 if partialMatch('beginning of turn', l, 80) else 0 for l in data['cardText']]\n",
    "    data['Spell ref'] = [1 if partialMatch('spell', l) else 0 for l in data['cardText']]\n",
    "    data['Creature ref'] = [1 if partialMatch('creature', l) else 0 for l in data['cardText']]\n",
    "    data['Land'] = [1 if partialMatch('land', l) else 0 for l in data['cardText']]\n",
    "    data['Mana'] = [1 if partialMatch('mana', l) else 0 for l in data['cardText']]\n",
    "    data['Battlefield'] = [1 if partialMatch('battlefield', l) else 0 for l in data['cardText']]\n",
    "    data['Blue ref'] = [1 if partialMatch('blue', l) else 0 for l in data['cardText']]\n",
    "    data['Black ref'] = [1 if partialMatch('black', l) else 0 for l in data['cardText']]\n",
    "    data['Green ref'] = [1 if partialMatch('green', l) else 0 for l in data['cardText']]\n",
    "    data['Red ref'] = [1 if partialMatch('red', l) else 0 for l in data['cardText']]\n",
    "    data['White ref'] = [1 if partialMatch('white', l) else 0 for l in data['cardText']]\n",
    "    data['Colorless ref'] = [1 if partialMatch('colorless', l) else 0 for l in data['cardText']]\n",
    "    \n",
    "    # 3% to 15% sparsity, which is actually pretty good for text\n",
    "    data.describe().to_csv(\"features.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 259,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "### Alternate/supplemental automatic text featurization\n",
    "\n",
    "if automatedTextFeatures:\n",
    "\n",
    "    from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "    from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "    # try n-grams, binary\n",
    "\n",
    "    # vectorizer = TfidfVectorizer(min_df=1, max_features=30)\n",
    "    # X = vectorizer.fit_transform(data['fullCardText'])\n",
    "    # print vectorizer.get_feature_names()\n",
    "    # print X.toarray()\n",
    "\n",
    "    # vectorizer = TfidfVectorizer(min_df=1, max_features=30, binary=True)\n",
    "    # X = vectorizer.fit_transform(data['fullCardText'])\n",
    "    # print vectorizer.get_feature_names()\n",
    "    # vectorizer.idf_\n",
    "\n",
    "    # vectorizer = TfidfVectorizer(min_df=1, max_features=30, ngram_range=(2,3))\n",
    "    # X = vectorizer.fit_transform(data['fullCardText'])\n",
    "    # print vectorizer.get_feature_names()\n",
    "    # vectorizer.idf_\n",
    "\n",
    "    # vectorizer = TfidfVectorizer(min_df=1, max_features=30, ngram_range=(4,6), analyzer='char')\n",
    "    # X = vectorizer.fit_transform(data['fullCardText'])\n",
    "    # print vectorizer.get_feature_names()\n",
    "    # vectorizer.idf_\n",
    "\n",
    "    # for (i, feature) in enumerate(vectorizer.get_feature_names()):\n",
    "    #     data[feature] = X.toarray()[:, i]\n",
    "\n",
    "    # MOST USEFUL THING IN THE END WAS JUST high-level TEXT SUMMARY\n",
    "    \n",
    "    vectorizer = CountVectorizer(min_df = 1, max_features= 50, ngram_range=(2, 3))\n",
    "    X = vectorizer.fit_transform(data['fullCardText'])\n",
    "    for item in sorted(vectorizer.vocabulary_.items(), key=lambda h: h[1], reverse=True):\n",
    "        print item\n",
    "\n",
    "    # data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (2d) -- Functional features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 260,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 4. Special functional features\n",
    "\n",
    "def isBuff(str, l):\n",
    "    found = 0\n",
    "    for val in l:\n",
    "        if str in val:\n",
    "            found += 1\n",
    "    if found > 0: return True\n",
    "    else: return False\n",
    "\n",
    "if functionalFeatures:\n",
    "\n",
    "    data['Untap'] = [1 if partialMatch('untap', l) else 0 for l in data['cardText']]\n",
    "    data['All'] = [1 if partialMatch('all', l) | partialMatch('any', l) else 0 for l in data['cardText']]\n",
    "\n",
    "    data['Tap ability'] = [1 if 'Tap' in x else 0 for x in data['cardText']]\n",
    "    data['Mana symbol'] = [1 if anyIntOrColor(x) else 0 for x in data['cardText']]\n",
    "    data['Mana related'] = [1 if partialMatch('add mana', l) | partialMatch('your mana pool', l) \\\n",
    "                                  else 0 for l in data['cardText']]\n",
    "\n",
    "    data['Buff'] = [1 if isBuff('+', l) else 0 for l in data['cardText']]\n",
    "    data['Debuff'] = [1 if isBuff('-', l) else 0 for l in data['cardText']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Note\n",
    "\n",
    "Some of this might have been able to be done automatically, especially the **text features**, which could have been done by finding the most common words referred to in text boxes. Again, I leave this to future work and am really curious about what the literature on automatic feature creation says about this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (3) -- Perform PCA\n",
    "\n",
    "Surprisingly, the PCA itself is the easiest part of this entire thing. We'll use `sklearn` to perform a 10-component PCA to see how much of the entire data's dimensional variation can be reduced to 10 dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 261,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.cluster import KMeans\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.preprocessing import scale\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "numericData = data.copy()\n",
    "# scale to mean 0, variance 1\n",
    "numericData_std = scale(numericData.fillna(0).select_dtypes(include=['float64', 'int64']))\n",
    "\n",
    "pca = PCA(n_components=10)\n",
    "Y_pca = pca.fit_transform(numericData_std)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###So, how well did we do?\n",
    "\n",
    "Well, based on the explained variance vector below it doesn't look like we did very well. The first two principal components only combined for **14% of the total variance** in the data; though, of note, is that the first 10 factors do account for **46% of the total variance**. Considering we're working with 62 features though, this is pretty decent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 262,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Variance explained by each factor:\n",
      "[0.081, 0.06, 0.052, 0.049, 0.042, 0.04, 0.038, 0.036, 0.034, 0.031]\n",
      "\n",
      "Variance explained by all 10 factors:\n",
      "0.463\n",
      "\n",
      "Num features:\n",
      "62\n",
      "\n",
      "Explained variance in 2-d: 0.141403885393\n"
     ]
    }
   ],
   "source": [
    "# Analysis of PCA effectiveness\n",
    "\n",
    "print\n",
    "print \"Variance explained by each factor:\"\n",
    "print [round(x, 3) for x in pca.explained_variance_ratio_]\n",
    "print\n",
    "print \"Variance explained by all 10 factors:\"\n",
    "print round(sum(pca.explained_variance_ratio_), 3)\n",
    "print\n",
    "print \"Num features:\"\n",
    "print len(numericData_std[0])\n",
    "print\n",
    "print \"Explained variance in 2-d: {0}\".format(pca.explained_variance_ratio_[0] + pca.explained_variance_ratio_[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (4) -- Results\n",
    "\n",
    "Now time to see if it was all worth it -- and apply the PCA projection onto our data set. We want to be able to make a pretty scatterplot grouping the data by different types (color, card type, rarity, etc.) so we will make a helper graphing function using the `plotly` library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 263,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import plotly.plotly as py\n",
    "py.sign_in('nhuber', 'bmopo8hk40')\n",
    "from plotly.graph_objs import *\n",
    "import plotly.tools as tls\n",
    "\n",
    "def chooseColor(group):\n",
    "    \n",
    "    if group == u'White': return '#B2B2B2'\n",
    "    if group == u'Artifact': return '#996633'\n",
    "    if group == u'Red' : return '#E50000'\n",
    "    if group == u'Blue': return '#0000FF'\n",
    "    if group == u'Green' : return '#006400'\n",
    "    if group == u'Black' : return '#000000'\n",
    "\n",
    "    if group == 'Instant': return '#E81A8C'\n",
    "    if group == 'Sorcery': return '#F2AB11'\n",
    "    if group == 'Creature' : return '#102DE8'\n",
    "    if group == 'Enchantment': return '#1BBF28'\n",
    "    if group == 'Land' : return '#000000'\n",
    "    if group == 'Artifact' : return '#82580E'\n",
    "    \n",
    "    if group == 'common' : return '#000000'\n",
    "    if group == 'uncommon': return '#9a9999'\n",
    "    if group == 'rare': return '#eae002'\n",
    "    if group == 'basic land': return '#ba7127'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 270,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# graphs data on pca axes grouped by type thetype\n",
    "    \n",
    "def graphByType(thetype, thetitle, centers=False, fix=1, typefilter='',\n",
    "               height=625, width=725, markerfontsize=9, titlefontsize=26):\n",
    "\n",
    "    # fix reflects data through y = 0 to be backwards\n",
    "    # compatible with previous annotated visualizations\n",
    "\n",
    "    # create graph data from pca results\n",
    "    \n",
    "    traces = []\n",
    "\n",
    "    if not typefilter:\n",
    "        typefilter = set(data[thetype])\n",
    "    \n",
    "    for group in typefilter:\n",
    "        \n",
    "        matches = []\n",
    "        for i in xrange(len(data[thetype])):\n",
    "            if data[thetype].irow(i) == group:\n",
    "                matches.append(i)\n",
    "\n",
    "        graphColor = chooseColor(group)\n",
    "\n",
    "        trace = Scatter(\n",
    "            x=Y_pca[matches,0],\n",
    "            y=fix * Y_pca[matches,1],\n",
    "            mode='text',\n",
    "            name=group,\n",
    "            marker=Marker(\n",
    "                size=8,\n",
    "                color=graphColor,\n",
    "                opacity=0.5),\n",
    "            text = data['cardName'].irow(matches),\n",
    "            textfont = Font(\n",
    "                family='Georgia',\n",
    "                size=markerfontsize,\n",
    "                color=graphColor\n",
    "                )\n",
    "            )\n",
    "        \n",
    "        traces.append(trace)\n",
    "\n",
    "        if centers:\n",
    "\n",
    "            traceCentroid = Scatter(\n",
    "                x = np.mean(Y_pca[matches,0]),\n",
    "                y = np.mean(fix * Y_pca[matches,1]),\n",
    "                mode = 'marker',\n",
    "                name = str(thetype) + \" center\",\n",
    "                marker = Marker(\n",
    "                    size = 26,\n",
    "                    color=graphColor),\n",
    "                opacity = 0.75\n",
    "            )\n",
    "\n",
    "            traces.append(traceCentroid)\n",
    "        \n",
    "    # Set up the scatter plot layout\n",
    "\n",
    "    dataToGraph = Data(traces)\n",
    "\n",
    "    # auto-focus on where most of the data is clustered\n",
    "    xRange = max(abs(np.percentile(np.array([x[0] for x in Y_pca]), 2.5)),\n",
    "                abs(np.percentile(np.array([x[0] for x in Y_pca]), 97.5)))\n",
    "    yRange = max(abs(np.percentile(np.array([x[1] for x in Y_pca]), 2.5)),\n",
    "                abs(np.percentile(np.array([x[1] for x in Y_pca]), 97.5)))\n",
    "\n",
    "    layout = Layout(title=thetitle,\n",
    "                titlefont=Font(family='Georgia', size=titlefontsize),\n",
    "                showlegend = True,\n",
    "                autosize = False,\n",
    "                height = height,\n",
    "                width = width,\n",
    "                xaxis=XAxis(\n",
    "                    range=[-xRange, +xRange],\n",
    "                    title='PC1', showline=False),\n",
    "                yaxis=YAxis(\n",
    "                    range=[-yRange, +yRange],\n",
    "                    title='PC2', showline=False))\n",
    "    \n",
    "    fig = Figure(data=dataToGraph, layout=layout)\n",
    "    return fig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (4a) -- Grouping by color"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first visualize all of our cards on the two PCA axes, grouped by color."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 271,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\"seamless=\"seamless\" src=\"https://plot.ly/~nhuber/345.embed\" height=\"525\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 271,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fig = graphByType('color', \"PCA on MtG by card color\")\n",
    "py.iplot(fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A few notes on this graph:\n",
    "- It's interactive: you can zoom into an area on the graph by dragging to create a rectangle\n",
    "- Also note that you can click the labels on the top right to turn on and off showing cards of different colors\n",
    "- It will probably have more meaning if you know about Magic and what each of these cards do; so, I'll offer my analysis below but if you do play and have alternate interpretations about how these cards are grouped, please do lmk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## <u>Result 1</u>: A tale of two psuedo-axes\n",
    "\n",
    "The primary result of this analysis is that a magic card can be mainly broken down into two components: How much does it behave like a spell vs. a creature? and How much does it affect the board or non-board resources? Visually, we are left with two \"psuedo-axes\":\n",
    "- On the left downward diagonal -- a <u>creature axis</u> which represents how \"creature-y\" a card is: big creatures are very creature-y, mid-sized utility creatures are somewhat creature-y, and enchantments/spells are not creature-y at all. \n",
    "- On the right upward diagonal, we have the <u>mana/hand axis</u> -- which is a spectrum on how much a card relates primarily to the **board** (i.e. permanents in play) or whether it affects **non-board resources** such as the player's cards and mana pools."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "<img src=\"./pca2.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Going through salient examples\n",
    "\n",
    "We'll now go through the highlighted examples in the above graph, from left to right, to understand and evaluate how the model performs; my evaluation out of 5 for each card plotting is in parenthesis in the title:\n",
    "\n",
    "### Juzam Djinn and Juggernaut (5/5)\n",
    "<img src=\"./cards/jugg.jpeg\" style=\"height:200px; display: inline-block; margin: 0 auto\"> <img src=\"./cards/juz.jpeg\" style=\"height:200px; display: inline; margin: 0\">\n",
    "- Classic examples of \"fatties,\" large creatures that dominate the board through sheer size\n",
    "- *Very creature-y* and *very board-related*\n",
    "\n",
    "### Goblin King, Old Man of the Sea and White Knight (5/5)\n",
    "<img src=\"./cards/gob.jpeg\" style=\"height:200px; display: inline-block\"> <img src=\"./cards/om.jpeg\" style=\"height:200px; display: inline-block; margin: 0\"> <img src=\"./cards/wknight.jpeg\" style=\"height:200px; display: inline-block; margin: 0\">\n",
    "- Classic examples of \"utility creatures\": medium size (2/2, 2/3, and 2/2 respectively) but have impact on the board through their abilities\n",
    "- *Mostly creature-y* and *very board-related*\n",
    "\n",
    "### Berserk, Raise dead (5/5)\n",
    "<img src=\"./cards/berserk.jpeg\" style=\"height:200px; display: inline-block\"> <img src=\"./cards/raise.jpeg\" style=\"height:200px; display: inline-block; margin: 0\">\n",
    "- Combat trick that offers a one-time pump for a creature in combat and enchantment that revives a creature from the graveyard\n",
    "- Appropriately *somewhat creature-y* and *very board-related*\n",
    "\n",
    "### Birds of Paradise (5/5), Manabarbs (4/5)\n",
    "<img src=\"./cards/bop.jpeg\" style=\"height:200px; display: inline-block\"> <img src=\"./cards/manabarbs.jpeg\" style=\"height:200px; display: inline-block; margin: 0\">\n",
    "- Birds of paradise -- a tiny, one-drop creature that provides mana ramping ability: the model correctly realizes it's a utility creature (i.e. medium creature-y) and is heavily related to a non-board resource, namely: mana.\n",
    "- Manabarbs is similarly a permanent, mana-based effect but it also impacts player's life totals so is appropriately in the middle of this axis\n",
    "- *Medium creature-y*, *mostly mana/hand*\n",
    "\n",
    "### Balance (4/5)\n",
    "<img src=\"./cards/balance.jpeg\" style=\"height:200px; display: inline-block\">\n",
    "- A high-impact spell that equalizes both players creature counts, cards in hand, and lands in play.\n",
    "- *Very spell-y* and related to *mostly non-board* resources (though it does equalize creatures as well)\n",
    "\n",
    "### Red elemental blast (4/5)\n",
    "<img src=\"./cards/reb.jpeg\" style=\"height:200px; display: inline-block\">\n",
    "- This is a tough one for the model: this card has two modes: destroy a blue permanent or counter a blue spell. Clearly, these are very different cards. But the model correctly predicts that it's very *spell-y* and mostly *board-related* \n",
    "\n",
    "### Wheel of Fortune, Ancestral Recall (5/5)\n",
    "<img src=\"./cards/wheel.jpeg\" style=\"height:200px; display: inline-block\"> <img src=\"./cards/arecall.jpeg\" style=\"height:200px; display: inline-block; margin: 0\">\n",
    "- Perfect categorization for Wheel: this is a completely unique spell in the game where each player discards their hand and draws 7 new cards. Not a creature at all, not related to the board at all; therefore, is correctly labelled *spell-y* and *mana/hand-y*\n",
    "- Great categorization again for Ancestral: an instant-speed draw spell (note that it's right next to Braingeyser as well): very *spell-y* very *mana/hand-y*\n",
    "\n",
    "### Howling Mine (3/5)\n",
    "<img src=\"./cards/howl.jpeg\" style=\"height:200px; display: inline-block\">\n",
    "- Primarily card-related (each player draws 2 instead of 1) and a permanent effect so somewhat *creature-y*\n",
    "- Strange card because it has a <u>permanent effect on card resources</u> which is rare (i.e. vs. draw spells, discard spells)\n",
    "\n",
    "### Demonic Tutor, Black Lotus (5/5)\n",
    "<img src=\"./cards/dt.jpeg\" style=\"height:200px; display: inline-block\"> <img src=\"./cards/lotus.jpeg\" style=\"height:200px; display: inline-block; margin: 0\">\n",
    "- DT: The canonical tutor effect; very *spell-y* and *hand-y*\n",
    "- Black Lotus: the canonical magic card; pure power in a one-time rush of mana (very *spell-y* and very *mana-y*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary\n",
    "\n",
    "The model does a very good job categorizing cards across these two psuedo-axes. Unsurprisingly, the hardest cards to categorize are those that <u>cross many axes</u> -- Balance in it's all-encompassing scope or Red Elemental Blast in its multiple modes -- or <u>have non-traditional effects</u> like Howling Mine being a card-drawing artifact."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <u>Result 2</u>: Exploring the color identities\n",
    "\n",
    "The PCA also tells a cool story of how different the colors are defined. Here each card is a small point, with the large point representing the \"average\" of all of the color's cards:\n",
    "\n",
    "<img src=\"./colors/art.png\" style=\"width:32%; display:inline-block\"><img src=\"./colors/blue.png\" style=\"width:32%; display:inline-block; margin: 0\"><img src=\"./colors/white.png\" style=\"width:32%; display:inline-block; margin: 0\">\n",
    "\n",
    "<img src=\"./colors/green.png\" style=\"width:32%; display:inline-block\"><img src=\"./colors/red.png\" style=\"width:32%; display:inline-block\"><img src=\"./colors/bl.png\" style=\"width:32%; display:inline-block\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Artifacts** are somewhat creature-y, somewhat spell-y (as they can have different effects depending on the card) but generally are more related to mana/hand resources than board effects (except for the Artifact creatures like Juggernaut that are correctly by other fatties like Juzam)\n",
    "- **Blue** and **White** seem to be, like all of the non-artifact colors, a mix of creatures and spells, but they skew towards the spell side; unlike **Green**, **Red**, and **Black** (the first two are very creature-y, the last a mix)\n",
    "- This is consistent with the intuitive color identity you have when you play the game: Blue and White are tricky, controlling decks; Red and Green are fatty/monster decks and Black is somewhat in between, arguably the most flexible color in the game in terms of large creatures, mana effects and also removal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <u>Result 3</u>: Exploring the type identities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 272,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\"seamless=\"seamless\" src=\"https://plot.ly/~nhuber/346.embed\" height=\"525\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 272,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fig = graphByType('Primary Type', \"PCA on MtG by card type\")\n",
    "py.iplot(fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model correctly associates creatures on the creature axis, instants and sorceries on the spell axis (with ones related to mana like Dark Ritual or Contract from Below higher on the mana/hand axis), enchantments in the \"spell-like\" but permanent effects region and lands in the mana-related zone. Artifacts because of their unique nature are incredibly hard to categorize and honestly the model probably doesn't do a great job for most of them, and sort of arbitrarily groups them in their own group as a whole, saying they're also mostly mana/hand-y, but there's likely a lot more that could be teased out here to get the artifacts to be distributed more uniformly/appropriately across these axes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <u>Result 4</u>: Exploring rarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 273,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\"seamless=\"seamless\" src=\"https://plot.ly/~nhuber/347.embed\" height=\"525\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 273,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fig = graphByType('rarity', \"PCA on MtG by card rarity\")\n",
    "py.iplot(fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "Rares are allowed to have all kinds of effects (i.e. spread across both axes), uncommons as well but less so (i.e. less smattered), commons limited to medium-sized creatures and non-mana/hand-related spells. This is consistent with the game designer's views on what \"feels like a common\" or what power levels different types of cards are able to have."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Conclusions and future work\n",
    "\n",
    "There's tons more work to do here that I'd love to if I have time:\n",
    "- Play with what features to use (which are the most impactful -- almost certainly type and color, but what else? could we automatically create them from card text boxes without domain knowledge? of those which are the most meaningful?)\n",
    "- What separates good cards from bad cards? how could we automatically detect/predict card quality?\n",
    "- What determines popular/interesting cards? (note that Gatherer also has community ratings for every card)\n",
    "- Does this analysis hold for the entire corpus of magic cards? how about for different sets? does it get better or worse over time?\n",
    "\n",
    "Until then, <br>\n",
    "@nhuber | nicholas.e.huber@gmail.com"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*custom css*\n",
    "<style>\n",
    "\n",
    "html {\n",
    "    font-family: \"Georgia\"\n",
    "}\n",
    "\n",
    "</style>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 268,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# MAIN RESULTS\n",
    "\n",
    "# two psuedo-axes: creatury-y, board-related\n",
    "# color identities\n",
    "# rarity limitations\n",
    "\n",
    "# type dominated as a feature; color, text, etc. only mattered so much\n",
    "# automated text feature extraction reduced interpretability"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}