PyData 2016 - Chicago - Day 3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Table of Contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* [1. Keynote: Developing Communities to Develop Themselves](#dev_communities)\n",
    "* [2. Implementing distributed grid search for deep learning using scikit-learn and joblib](#opensource_ml)\n",
    "* [3. Risk Management System using Python](#python_risk)\n",
    "* [4. Pyxley for flask + react powered dashboards](#pyxley)\n",
    "* [5. Keynote: Builtin Superheroes](#superheroes)\n",
    "* [6. Productionalizing](#production)\n",
    "* [7. Exploratory Data Analysis](#exploratory_analysis)\n",
    "* [8. Genotype-phenotype associations and machine learning](#genotype)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='dev_communities'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Developing Communities to Develop Themselves"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Matthew Turk](https://twitter.com/powersoffour)\n",
    "\n",
    "[yt-project](http://yt-project.org/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ingestion | Representation | Analysis | Visualization\n",
    "\n",
    "**Phases:**\n",
    "- Mostly developers, few non-dev users\n",
    "- Increasing non-dev users, many drive-by devs\n",
    "- Now there's some non-user devs\n",
    "\n",
    "~~**Sustaining Software**~~\n",
    "- Actually, \"sustaining inquiry\"\n",
    "- \"Conducting inquiry via software\n",
    "\n",
    "**Do we _actually_ need to sustain specific software**?\n",
    "- We want to sustain users, devs, their career, and their future as a whole?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**What is sustainability?**\n",
    "- Keep up with bug reports?\n",
    "- Add new features?\n",
    "- Compatible with new hardware?\n",
    "- People can learn to use it?\n",
    "- Grows new features?\n",
    "- Produces same results?\n",
    "- Transitions between people?\n",
    "\n",
    "\n",
    "$S(N) = \\frac{1}{(1-P)+(P/N)}$\n",
    "\n",
    "Quantifying bottlenecks. A representation of the bottlenecking in a community.\n",
    "\n",
    "**Challenges of software maintainance:**\n",
    "- bitrot\n",
    "- burnout\n",
    "- boringness\n",
    "\n",
    "~~**Empowering:**~~\n",
    "\n",
    ">Empowerment sets up a power dynamic\n",
    "\n",
    "**Community**\n",
    "\n",
    "- How do we increase diversity?\n",
    "- How can we foster careers?\n",
    "- How can we lower barriers?\n",
    "     * Acquire self-determination\n",
    "     * Leadership\n",
    "     * Usage of software\n",
    "     * Foster self-determination\n",
    "\n",
    "[Improv Science](http://improvscience.org/)\n",
    "\n",
    "**Values**\n",
    "- Two axes: technically challenging vs. socially challenging\n",
    "- Detrimental/humanitarian vs. Fuctional/Problematic\n",
    "- Product vs. project\n",
    "    * Project has a bidirectional flow\n",
    "    * Product is dictated to you \"the thing\"\n",
    "- Early-stage researchers are the ones who are most damaged by a product that suddenly changes direction.\n",
    "\n",
    "**Models**\n",
    "- Funded\n",
    "- Productized\n",
    "- Volunteer\n",
    "\n",
    "[Ashe Dryden's blog post](https://www.ashedryden.com/blog/the-ethics-of-unpaid-labor-and-the-oss-community)\n",
    "\n",
    "Barriers to entry (technical & social)\n",
    "\n",
    "Engagement (methods & modes)\n",
    "\n",
    "Investment \n",
    "\n",
    "> Credit is not a zero-sum game.\n",
    "\n",
    "> Code is not everything\n",
    "\n",
    "[Request for Commit Podcast](https://changelog.com/rfc-5/)\n",
    "\n",
    "**Scale**\n",
    "- Metcalfe's Law of scale\n",
    "\n",
    "**How does this look technically?**\n",
    "- Provide clear mechanisms for feedback, credit, and contribution\n",
    "- Don't bottleneck on one person who can describe all the contributions\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='opensource_ml'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Combining open source tech to implement a ML workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Mike Heilman](https://twitter.com/heilman13)\n",
    "\n",
    "[Slides](https://mheilman.github.io/pydata_chicago_2016/#/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Custom Machine Learning\n",
    "- Focus: supervised, discriminative deep learing with gradient-based optimization\n",
    "    * Alternatives (tree ensembles, probabilistic graphical models (PyMC3, STAN)\n",
    "- Goal:\n",
    "    * find a function $f ( y | X, W)$ that:\n",
    "        * parameters: weights W\n",
    "        * takes input: features X\n",
    "        * produces output: y\n",
    "- Procedures for finding f\n",
    "    * define an objective function $g (X, y, w)$\n",
    "        * find a setting for W that optimizes g\n",
    "        * often g is the difference between predicted y and actual y\n",
    "    * Define a function to compute derivative of *g* with respect to W\n",
    "        * Often duplicates engineering effort from implementing *f* and *g*\n",
    "        * For DL models, g, and derivative can be complicated\n",
    "    * Iterate\n",
    "\n",
    "- Graph-based numerical computation\n",
    "    * step 1 (model + objective implementation) is easy\n",
    "    * step 2 (differentiation)\n",
    "    * step 3 (iterative learning) is fast\n",
    "    \n",
    "- [TensorFlow](https://www.tensorflow.org/) + alternatives\n",
    "    * theano \n",
    "    * keras (wraps TF and theano): mostly for neural nets and DL\n",
    "    * mxnet\n",
    "    * torch\n",
    "    * caffe\n",
    "\n",
    "**Multilayer Preceptrons**\n",
    "- feed-forward neural network\n",
    "    * no recursion, no convolutions\n",
    "    * input layer, 1+ hidden layers, output layer\n",
    "- model interactions betwee features\n",
    "    * can approximate any function\n",
    "- old idea (Rosenblatt, 1961), but can be used with other methods\n",
    "    * ReLUs, dropout, Adam learn algorithm\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## Standard Interface for ML\n",
    " - Key feature: `sklearn.utils.estimator_checks.check_estimator` to check API conformity of a custom estimator\n",
    " - Facilitates: \n",
    "     * pipelines of transformations\n",
    "     * gridsearch over hyperparameters\n",
    " \n",
    " - Custom models need\n",
    "     * fit\n",
    "     * predict\n",
    "     * proba\n",
    "     * __init__ should attach args (fit does a lot of this)\n",
    "     * everything should be serializable with pickle\n",
    "         * joblib expects this\n",
    " \n",
    " - Civis MLP implementation\n",
    "     * GPU support\n",
    "     * sparse/dense matrix support\n",
    "     * [github link](https://github.com/civisanalytics/muffnn)\n",
    " \n",
    "- Pickle\n",
    "    * TF doesn't support pickling very well\n",
    "    * TF has a Saver class\n",
    "    * one can override `__getstate__`\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple Distributed Computing\n",
    "- ML models have lots of hyperparametrs\n",
    "    * deep learning: numbers and sizes of hidden layers\n",
    "    * probabilistic graphical models: parameters for pirors\n",
    "    * tree ensembles: depth, learning rate, samples per split\n",
    "- grid search\n",
    "    * try lots of settings\n",
    "    * often combined with k-fold CV\n",
    "    * crude but parallel\n",
    "- [JobLib](https://pypi.python.org/pypi/joblib)\n",
    "    * originally focused on single machine parallelizing\n",
    "\n",
    "- scikit-learn uses joblib to parallelize hyperparameter evaluation in GridSearchCV\n",
    "- if joblib uses a custom backend (like dask), sklearn will use that\n",
    "- Distributed grid search > distributed model fitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id ='python_risk' </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Contemporary Risk Management"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Piero Ferrante](http://twitter.com/the_real_pdf)\n",
    "\n",
    "[C2F0](https://c2fo.com/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## WFC (Water for commerce)\n",
    "- Investment fund and short-term lending platform for small-to-medium businesses\n",
    "- Over 5 years of daily invoices and adjustments\n",
    "- 40 day duration $\\rightarrow$ 6.25% yield\n",
    "\n",
    "### Why bother SMB lending?\n",
    "- Champion of the supplier\n",
    "- All risk is not created equal\n",
    "- Possible to do it without\n",
    "    * rage gauging borrowers\n",
    "    * misleading investors\n",
    "- Great data/tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Risk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concentration Risk: measuring diversity\n",
    "- More diversity of accounts receivable is better\n",
    "- Less concentration with 'junk' buyers\n",
    "\n",
    "## Default risk- forecasting accounts receivable\n",
    "- Want to forecast in R (use RPy2)\n",
    "- Model types\n",
    "    * ARMA/ARIMA/SARIMA\n",
    "    * Exponential smoothing (Holt-Winters)\n",
    "    * Bayesian Structural Time Series\n",
    "    * Regression (OLS, polynomial) -lm function\n",
    "    * Currently evaluating pyflux\n",
    "\n",
    "- Best model seeks to minimize the mean absolute percentage error\n",
    "- plots using matplotlib and seaborn\n",
    "- use statsmodels: seasonal decomposition\n",
    "    * Visually decouple\n",
    "\n",
    "## Default risk- predicting AR discontinuation\n",
    "\"AR Discontinuation\": supplier AR dropping to zero with all C2FO buyers\n",
    "\n",
    "**Challenges**:\n",
    "- Data Leakage\n",
    "    * do not observe that which would not have been observable at the time of prediction\n",
    "    * establish prediction cutoff\n",
    "    * remove all history after the cutoff date\n",
    "    * explore different winners\n",
    "- Engineering features\n",
    "- Training\n",
    "    * use scikit-learn for feature engineering\n",
    "        * encoding categoricals\n",
    "        * creating polynomials\n",
    "        * scaling features\n",
    "        * dimensionality reduction/feature selection\n",
    "    * use xgboost to train gradient-boosted trees. use in conjunction with hyperopt.\n",
    "        * currently evaluating spearmint\n",
    "- Primarily concerned with model recall & not overfitting\n",
    "\n",
    "## Default risk- predicting bankruptcy\n",
    "- Different than AR discontinuation\n",
    "- Prediction labels are different\n",
    "- Daily feeds from national bankruptcy database\n",
    "- Matching process\n",
    "- Perform data truncation and feature engineering\n",
    "- Enrich with macroeconomic data from the right pointi n time\n",
    "- Address class imbalances\n",
    "- Train models\n",
    "    * Clean data\n",
    "    * Match on 'unique values'\n",
    "        * TAX IDs & Phone numbers\n",
    "    * Use string matching on company names\n",
    "        * Levenshtei distance, jaro-winkler distance, jaccard distance\n",
    "    * geographical distance between known addresses\n",
    "        * Haversine distance\n",
    "    * tinker with weighting strategy\n",
    "- Tips\n",
    "    * use Cython or Numba\n",
    "\n",
    "## Fraud risk - Screening calls\n",
    "- Use NLP to transcribe and mine calls\n",
    "- spacy makes tokenization/lemmatization fast\n",
    "- identify conversations with red flags\n",
    "\n",
    "## Fraud risk - Analyze invoice congruency\n",
    "- Getting buyer-supplier score, then supplier-level score\n",
    "\n",
    "## Who to lend to?\n",
    "- WFC scores and train a classfier\n",
    "- loan duration\n",
    "- higher decile - creater % of n day forecast cumulative sum\n",
    "- Rates are calculated\n",
    "    * by observing suppliers' rates in C2FO markets\n",
    "    * adjusting for additioal risk when applicable\n",
    "    \n",
    "## Who to continue lending to?\n",
    "- Triggers\n",
    "    * Level shifts in AR patterns\n",
    "    * C2FO bid changes\n",
    "    * WFC score chagnes\n",
    "    * Adjustments\n",
    "    * Buyer reserves\n",
    "    \n",
    "## Miscellaneous tools\n",
    "- anaconda\n",
    "- luigi\n",
    "- dask\n",
    "- [spyre](https://github.com/adamhajari/spyre)\n",
    "\n",
    "## So what?\n",
    "- Objectivity often gives way to innovation\n",
    "- Tradeoffs must be evaluated in light of constraints\n",
    "    * would i might learning and/or maintaining the code\n",
    "    * do you need a beautiful front end?\n",
    "    * how fast is fast enough?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='pyxley'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pyxley"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Nicholas Kridler](https://github.com/nmkridler)\n",
    "\n",
    "[Twitter](http://twitter.com/nmkridler)\n",
    "\n",
    "[Merch Algorithms](http://multithreaded.stitchfix.com/algorithms/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspired by Shiny\n",
    "- ui.R and server.R\n",
    "- Abstraction\n",
    "    * components handle flask stuff (specify JSON output format for charts)\n",
    "- Decided on a JS pattern, wrote python wrappers, and didn't write documentation or tests\n",
    "\n",
    "## Lessons\n",
    "1. Do not plan for \"Zero interest\"\n",
    "2. Read some guides first: python open source basics\n",
    "    * PyPi has a test server\n",
    "    * Conda environments\n",
    "    * virtualenv\n",
    "    * [PyPi quick and dirty](https://hynek.me/articles/sharing-your-labor-of-love-pypi-quick-and-dirty/)\n",
    "    * [Open sourcing a python package the right way](https://jeffknupp.com/blog/2013/08/16/open-sourcing-a-python-project-the-right-way/)\n",
    "3. Make straightforward examples (and documentation)\n",
    "    * Make MVP examples instead of as much functionality at once\n",
    "4. Make sure it's something you love\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='superheroes'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Builtin superheroes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[David Beazley](https://twitter.com/dabeaz)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Secret weapon\n",
    "- builtin types\n",
    "    * tuple, list, set, dict\n",
    "    * collections module\n",
    "    * various builtin operations\n",
    "    \n",
    "## Everywhere\n",
    "- the built-in types are always available\n",
    "- built-in types are fast-- for coding\n",
    "\n",
    "## Fun\n",
    "- built-in types are fun to use\n",
    "- Cleverness is rewarded\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "row = ('Dave', 'Beazley', '4312 N Clark ST')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Beazley'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "row[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'4312 N Clark ST'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "row[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import namedtuple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Person = namedtuple('Person', ['first', 'last', 'address'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "row = Person('Dave', 'Beazley', 'address')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Dave'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "row.first"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List (enforcing order)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Dave', 'Thomas', 'Paula', 'Dave']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names = ['Dave', 'Thomas', 'Paula', 'Dave']\n",
    "names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set (uniqueness, membership)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Dave', 'Paula', 'Thomas'}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names = set(['Dave', 'Thomas', 'Paula', 'Dave'])\n",
    "names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dictionary: mapping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "94.23"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prices = {\n",
    "    'ACME': 94.23,\n",
    "    'YOW':45.2\n",
    "}\n",
    "prices['ACME']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'x': 1, 'y': 2, 'z': 2})"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from collections import Counter\n",
    "c = Counter('xyzzy')\n",
    "c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'a': 10, 'b': 13, 'x': 1, 'y': 2, 'z': 2})"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c['a'] +=10\n",
    "c['b'] +=13\n",
    "c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#one to many relationships, grouping, multidicts\n",
    "from collections import defaultdict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(list, {'blah': [13], 'spam': [42, 10]})"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "d = defaultdict(list)\n",
    "d['spam'].append(42)\n",
    "d['blah'].append(13)\n",
    "d['spam'].append(10)\n",
    "d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic Powers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## loops\n",
    "## iteration\n",
    "## reductions (sum, min, max, any, all)\n",
    "## variants (enumerate, zip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Superpowers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 4, 9, 16, 25, 36]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## List comprehesnsions\n",
    "## set comprehensions\n",
    "## Dict comprehensions\n",
    "nums = [1,2,3,4,5,6]\n",
    "squares = []\n",
    "for x in nums:\n",
    "    squares.append(x*x)\n",
    "squares"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 4, 9, 16, 25, 36]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "squares = [x*x for x in nums]\n",
    "squares"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iterpowers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<generator object <genexpr> at 0x107630990>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# generator expressions + reductions\n",
    "squares = (x*x for x in nums)\n",
    "squares"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "4\n",
      "9\n",
      "16\n",
      "25\n",
      "36\n"
     ]
    }
   ],
   "source": [
    "for n in squares:\n",
    "    print(n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some Data Fun"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Food inspections data here](https://data.cityofchicago.org/Health-Human-Services/Food-Inspections/4ijn-s7e5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "food = list(csv.DictReader(open('Food_Inspections.csv')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(food)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'AKA Name': 'JIMMY BEANS, A LOGAN SQUARE ROASTER',\n",
       " 'Address': '2553 W FULLERTON AVE ',\n",
       " 'City': 'CHICAGO',\n",
       " 'DBA Name': 'JIMMY BEANS, A LOGAN SQUARE ROASTER',\n",
       " 'Facility Type': 'Restaurant',\n",
       " 'Inspection Date': '08/18/2016',\n",
       " 'Inspection ID': '1950659',\n",
       " 'Inspection Type': 'License',\n",
       " 'Latitude': '41.92475050821695',\n",
       " 'License #': '2483159',\n",
       " 'Location': '(41.92475050821695, -87.69188517704096)',\n",
       " 'Longitude': '-87.69188517704096',\n",
       " 'Results': 'Pass',\n",
       " 'Risk': 'Risk 2 (Medium)',\n",
       " 'State': 'IL',\n",
       " 'Violations': '32. FOOD AND NON-FOOD CONTACT SURFACES PROPERLY DESIGNED, CONSTRUCTED AND MAINTAINED - Comments: NOTED NO SPLASH GUARD AT THE HAND WASH SINK OF THE FRONT PREP AREA BY THE PREP TABLE. INSTRUCTED TO PROVIDE A SPLASH GUARD.',\n",
       " 'Zip': '60647'}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "food[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Business Not Located',\n",
       " 'Fail',\n",
       " 'No Entry',\n",
       " 'Not Ready',\n",
       " 'Out of Business',\n",
       " 'Pass',\n",
       " 'Pass w/ Conditions'}"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#all possible outcomes because sets are unique\n",
    "{ row['Results'] for row in food}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fail = [row for row in food if row['Results']=='Fail']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25328"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(fail)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'AKA Name': 'YUM DUM',\n",
       " 'Address': '2300 S THROOP ST ',\n",
       " 'City': 'CHICAGO',\n",
       " 'DBA Name': 'YUM DUM TRUCK',\n",
       " 'Facility Type': 'Mobile Food Preparer',\n",
       " 'Inspection Date': '08/17/2016',\n",
       " 'Inspection ID': '1950614',\n",
       " 'Inspection Type': 'License',\n",
       " 'Latitude': '41.85045102427',\n",
       " 'License #': '2483952',\n",
       " 'Location': '(41.85045102427, -87.65879785567869)',\n",
       " 'Longitude': '-87.65879785567869',\n",
       " 'Results': 'Fail',\n",
       " 'Risk': 'Risk 1 (High)',\n",
       " 'State': 'IL',\n",
       " 'Violations': '34. FLOORS: CONSTRUCTED PER CODE, CLEANED, GOOD REPAIR, COVING INSTALLED, DUST-LESS CLEANING METHODS USED - Comments: OBSERVED FLOORS UNDER PREP AND HOT HOLDING TABLES WITH EXCESSIVE GREASE AND FOOD DEBRIS. INSTRUCTED TO CLEAN AND MAINTAIN. | 35. WALLS, CEILINGS, ATTACHED EQUIPMENT CONSTRUCTED PER CODE: GOOD REPAIR, SURFACES CLEAN AND DUST-LESS CLEANING METHODS - Comments: OBSERVED EXCESSIVE ACCUMULATED DUST ON THE VENT FANS LOCATED ON THE CEILING. INSTRUCTED TO CLEAN AND MAINTAIN. | 38. VENTILATION: ROOMS AND EQUIPMENT VENTED AS REQUIRED: PLUMBING: INSTALLED AND MAINTAINED - Comments: OBSERVED A LEAK UNDER THE EXPOSED HAND SINK LOCATED IN THE FOOD PREP AREA. INSTRUCTED TO REPAIR AND MAINTAIN. | 30. FOOD IN ORIGINAL CONTAINER, PROPERLY LABELED: CUSTOMER ADVISORY POSTED AS NEEDED - Comments: OBSERVED NEW LICENSE NUMBER MUST BE PRINTED ON BOTH SIDES OF THE TRUCK. REMOVE THE OLD LICENSE NUMBER.',\n",
       " 'Zip': '60608'}"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fail[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "worst = Counter(row['DBA Name'] for row in fail)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('SUBWAY', 205),\n",
       " ('DUNKIN DONUTS', 132),\n",
       " (\"MCDONALD'S\", 89),\n",
       " ('7-ELEVEN', 44),\n",
       " ('MCDONALDS', 39)]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "worst.most_common(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('SUBWAY', 205),\n",
       " ('DUNKIN DONUTS', 132),\n",
       " (\"MCDONALD'S\", 89),\n",
       " ('7-ELEVEN', 44),\n",
       " ('MCDONALDS', 39),\n",
       " ('CHIPOTLE MEXICAN GRILL', 34),\n",
       " ('POTBELLY SANDWICH WORKS LLC', 34),\n",
       " (\"HAROLD'S CHICKEN SHACK\", 32),\n",
       " ('CITGO', 30),\n",
       " (\"PAPA JOHN'S PIZZA\", 27),\n",
       " (\"McDONALD'S\", 27),\n",
       " ('Subway', 23),\n",
       " ('MARATHON', 22),\n",
       " ('LAS ISLAS MARIAS', 22),\n",
       " (\"DOMINO'S PIZZA\", 22)]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "worst.most_common(15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# taking a dictionary row and making new things\n",
    "fail = [{**row, 'DBA Name': row['DBA Name'].replace(\"'\", '').upper()}\n",
    "       for row in fail]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "worst = Counter(row['DBA Name'] for row in fail)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('SUBWAY', 228),\n",
       " ('MCDONALDS', 168),\n",
       " ('DUNKIN DONUTS', 144),\n",
       " ('CHIPOTLE MEXICAN GRILL', 48),\n",
       " ('7-ELEVEN', 47)]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "worst.most_common(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('SUBWAY', 228),\n",
       " ('MCDONALDS', 168),\n",
       " ('DUNKIN DONUTS', 144),\n",
       " ('CHIPOTLE MEXICAN GRILL', 48),\n",
       " ('7-ELEVEN', 47),\n",
       " ('POTBELLY SANDWICH WORKS LLC', 34),\n",
       " ('HAROLDS CHICKEN SHACK', 33),\n",
       " ('CITGO', 30),\n",
       " ('PAPA JOHNS PIZZA', 30),\n",
       " ('JIMMY JOHNS', 30),\n",
       " ('DOMINOS PIZZA', 23),\n",
       " ('MC DONALDS', 22),\n",
       " ('AU BON PAIN', 22),\n",
       " ('SUBWAY SANDWICHES', 22),\n",
       " ('MARATHON', 22),\n",
       " ('LAS ISLAS MARIAS', 22),\n",
       " ('KFC', 22),\n",
       " ('FOREVER YOGURT', 22),\n",
       " ('DUNKIN DONUTS/BASKIN ROBBINS', 22),\n",
       " ('SHARKS FISH & CHICKEN', 22)]"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "worst.most_common(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bad = Counter(row['Address'] for row in fail)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('11601 W TOUHY AVE ', 180),\n",
       " ('324 N LEAVITT ST ', 59),\n",
       " ('500 W MADISON ST ', 58),\n",
       " ('2300 S THROOP ST ', 33),\n",
       " ('2637 S THROOP ST ', 33)]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bad.most_common(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "by_year = defaultdict(Counter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for row in fail:\n",
    "    by_year[row['Inspection Date'][-4:]][row['Address']]+=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('11601 W TOUHY AVE ', 39),\n",
       " ('500 W MADISON ST ', 12),\n",
       " ('324 N LEAVITT ST ', 9),\n",
       " ('307 S KEDZIE AVE ', 9),\n",
       " ('12 S MICHIGAN AVE ', 8)]"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "by_year['2015'].most_common(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('11601 W TOUHY AVE ', 32),\n",
       " ('500 W MADISON ST ', 17),\n",
       " ('324 N LEAVITT ST ', 15),\n",
       " ('113-125 N GREEN ST ', 12),\n",
       " ('131 N CLINTON ST ', 10)]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "by_year['2014'].most_common(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('11601 W TOUHY AVE ', 37),\n",
       " ('700 E GRAND AVE ', 10),\n",
       " ('2300 S THROOP ST ', 10),\n",
       " ('301 E NORTH WATER ST ', 9),\n",
       " ('12760 S HALSTED ST ', 8)]"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "by_year['2013'].most_common(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('11601 W TOUHY AVE ', 180),\n",
       " ('324 N LEAVITT ST ', 59),\n",
       " ('500 W MADISON ST ', 58),\n",
       " ('2300 S THROOP ST ', 33),\n",
       " ('2637 S THROOP ST ', 33)]"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bad.most_common(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'11601 W TOUHY AVE '"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "_[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ohare = [row for row in fail if row['Address'].startswith('11601 W TOUHY')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "181"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(ohare)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'11601 W TOUHY AVE ', '11601 W TOUHY AVE T2 F12'}"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{row['Address'] for row in ohare}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'AMERICAN AIRLINES',\n",
       " 'AMERICAS DOG',\n",
       " 'ANDIAMOS OHARE, LLC',\n",
       " 'ARAMARK AT UNITED AIRLINES',\n",
       " 'ARGO TEA',\n",
       " 'ARGO TEA CAFE-OHARE T2',\n",
       " 'AUNTIE ANNES',\n",
       " 'AUNTIE ANNES PRETZELS',\n",
       " 'B JS  MARKET',\n",
       " 'BRITISH AIRWAYS',\n",
       " 'BURRITO BEACH',\n",
       " 'CAFFE  MERCATO',\n",
       " 'CHICAGO BLACKHAWKS STANLEYS T2 BAR',\n",
       " 'CHICAGO NEWS & GIFTS',\n",
       " 'CHILIS T - 3',\n",
       " 'CHILIS T-I',\n",
       " 'CHILIS- G CONCOURSE',\n",
       " 'CNN',\n",
       " 'EFIES CANTEEN INC',\n",
       " 'ELIS CHEESECAKE',\n",
       " 'FARMERS FRIDGE',\n",
       " 'FRESH ON THE FLY',\n",
       " 'FRONTERA TORTAS  BY RICK BAYLESS GATE K4 T3',\n",
       " 'FRONTERA TORTAS BY RICK  BAYLESS',\n",
       " 'GARRETT POPCORN SHOPS',\n",
       " 'GATEGOURMET',\n",
       " 'GOLD COAST DOGS',\n",
       " 'GREEN MARKET',\n",
       " 'HILTON OHARE',\n",
       " 'HOST INTERNATIONAL B05',\n",
       " 'HOST INTERNATIONAL INC',\n",
       " 'HOST INTERNATIONAL INC, CHILIS T-2',\n",
       " 'HOST INTERNATIONAL INC-GOOSE ISLAND T3',\n",
       " 'HOST INTERNATIONAL INC-PRAIRIE TAP',\n",
       " 'HOST INTERNATIONAL INC.',\n",
       " 'HOT DOG EXPRESS',\n",
       " 'HUDSON',\n",
       " 'HUDSON NEWS',\n",
       " 'HUDSON NEWS OHARE JOINT VENTURE',\n",
       " 'ICE BAR',\n",
       " 'INTELLIGENTSIA',\n",
       " 'JAMBA JUICE',\n",
       " 'KOREAN AIR LOUNGE',\n",
       " 'LA TAPENADES GATE H14',\n",
       " 'LOU MITCHELLS EXPRESS INC',\n",
       " 'MACARONI GRILL',\n",
       " 'MCDONALDS',\n",
       " 'MCDONALDS RESTAURANT',\n",
       " 'NATURAL BREAK',\n",
       " 'NUTS ON CLARK',\n",
       " 'OHARE BAR',\n",
       " 'OHARE HILTON HOTEL',\n",
       " 'PARADES A CHICAGO BAR',\n",
       " 'PUBLICAN TAVERN K1',\n",
       " 'REGGIOS PIZZA EXPRESS',\n",
       " 'ROCKY MOUNTAIN CHOCOLATE FACTORY',\n",
       " 'RUSH STREET',\n",
       " 'SALAD WORKS',\n",
       " 'SARAHS CANDIES',\n",
       " 'SKYBRIDGE RESTAURANT & BAR',\n",
       " 'STARBUCKS',\n",
       " 'STARBUCKS HK APEX',\n",
       " 'STARBUCKS L03',\n",
       " 'SUBWAY SANDWICH',\n",
       " 'THE GODDESS & GROCER',\n",
       " 'THE GREAT AMERICAN BAGEL',\n",
       " 'TOCCO',\n",
       " 'TORTAS FRONTERA',\n",
       " 'TRAVEL TRADERS #3081 @ HILTON OHARE',\n",
       " 'TUSCANY CAFE',\n",
       " 'UNITED CLUB',\n",
       " 'UNITED CLUB ,T-1  CONCOURSE C',\n",
       " 'UNITED CLUB, TERMINAL 2 CONCOURSE F',\n",
       " 'UNITED CLUB,TERMINAL 1 CONCOURSE B SOUTH',\n",
       " 'UNITED FIRST INTERNATIONAL LOUNGE T1,CONCOURSE C',\n",
       " 'WOLFGANG EXPRESS',\n",
       " 'WOLFGANG PUCK, T-3',\n",
       " 'ZOOTS'}"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "{row['DBA Name'] for row in ohare}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'AKA Name': \"REGGIO'S PIZZA EXPRESS (T3 G3)\",\n",
       " 'Address': '11601 W TOUHY AVE ',\n",
       " 'City': 'CHICAGO',\n",
       " 'DBA Name': 'REGGIOS PIZZA EXPRESS',\n",
       " 'Facility Type': 'Restaurant',\n",
       " 'Inspection Date': '08/16/2016',\n",
       " 'Inspection ID': '1950494',\n",
       " 'Inspection Type': 'License',\n",
       " 'Latitude': '42.008536400868735',\n",
       " 'License #': '2428080',\n",
       " 'Location': '(42.008536400868735, -87.91442843927047)',\n",
       " 'Longitude': '-87.91442843927047',\n",
       " 'Results': 'Fail',\n",
       " 'Risk': 'Risk 1 (High)',\n",
       " 'State': 'IL',\n",
       " 'Violations': '',\n",
       " 'Zip': '60666'}"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ohare[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# find the worst location in o'hare to eat\n",
    "c = Counter(row['AKA Name'] for row in ohare)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('MACARONI GRILL (T3-K2)', 6),\n",
       " ('Gategourmet (BLDG 741)', 5),\n",
       " ('ADMIRALS CLUB/AMERICAN AIRLINES (T3/H&K)', 5),\n",
       " ('United Employee Cafeteria (T1 C LL)', 4),\n",
       " (\"CHILI'S TOO (T2  F4)\", 4),\n",
       " ('HUDSON NEWS', 4),\n",
       " ('ARGO TEA  (T3 ROTUNDA)', 4),\n",
       " (\"CHILI'S  TOO (T3-H2)\", 4),\n",
       " ('UNITED CLUB (T1 B6)', 4),\n",
       " ('WOLFGANG PUCK (T3 K1)', 4)]"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'AKA Name': \"REGGIO'S PIZZA EXPRESS (T3 G3)\",\n",
       " 'Address': '11601 W TOUHY AVE ',\n",
       " 'City': 'CHICAGO',\n",
       " 'DBA Name': 'REGGIOS PIZZA EXPRESS',\n",
       " 'Facility Type': 'Restaurant',\n",
       " 'Inspection Date': '08/16/2016',\n",
       " 'Inspection ID': '1950494',\n",
       " 'Inspection Type': 'License',\n",
       " 'Latitude': '42.008536400868735',\n",
       " 'License #': '2428080',\n",
       " 'Location': '(42.008536400868735, -87.91442843927047)',\n",
       " 'Longitude': '-87.91442843927047',\n",
       " 'Results': 'Fail',\n",
       " 'Risk': 'Risk 1 (High)',\n",
       " 'State': 'IL',\n",
       " 'Violations': '',\n",
       " 'Zip': '60666'}"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ohare[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "inspections = defaultdict(list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for row in ohare:\n",
    "    inspections[row['License #']].append(row)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'AKA Name': \"REGGIO'S PIZZA EXPRESS (T3 G3)\",\n",
       "  'Address': '11601 W TOUHY AVE ',\n",
       "  'City': 'CHICAGO',\n",
       "  'DBA Name': 'REGGIOS PIZZA EXPRESS',\n",
       "  'Facility Type': 'Restaurant',\n",
       "  'Inspection Date': '08/16/2016',\n",
       "  'Inspection ID': '1950494',\n",
       "  'Inspection Type': 'License',\n",
       "  'Latitude': '42.008536400868735',\n",
       "  'License #': '2428080',\n",
       "  'Location': '(42.008536400868735, -87.91442843927047)',\n",
       "  'Longitude': '-87.91442843927047',\n",
       "  'Results': 'Fail',\n",
       "  'Risk': 'Risk 1 (High)',\n",
       "  'State': 'IL',\n",
       "  'Violations': '',\n",
       "  'Zip': '60666'}]"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inspections['2428080']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['34154', '34201', '34229', '2363771', '56366', '1885160', '2289520', '2124574', '2016732', '29570', '2428079', '2277363', '1142116', '34183', '1909532', '1333098', '2447055', '1909539', '2124567', '2289525', '2289531', '85188', '2232034', '34203', '2261733', '2192969', '1621425', '2141979', '1884293', '34211', '34205', '2289511', '2021757', '1333242', '1141505', '34173', '2192968', '1927556', '34142', '2192963', '1069382', '2114331', '23894', '2289495', '2451545', '34224', '1947515', '34192', '2289515', '2204037', '2103989', '34199', '1120626', '2109577', '1879167', '2289527', '2428080', '2016727', '2363760', '2017724', '1879166', '2363762', '34190', '34169', '2289524', '34234', '1333092', '2289084', '34212', '2261728', '2069938', '1069379', '64032', '34167', '1879164', '2125489', '0', '2232035', '1974743', '2009092', '2277391', '1333235', '1884292', '15531', '1898075', '1224624', '1916161', '34146', '2009095', '1381615', '1916219', '1888807', '2184012', '2125246', '1878675', '56367', '37170', '2463991', '34139', '2013208', '1926528', '1942304', '2016729', '51206', '1718776', '1675026', '2284294', '34220', '2284027', '1042895', '2299087'])"
      ]
     },
     "execution_count": 82,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inspections.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['04/07/2016', '09/04/2014', '09/20/2011', '01/26/2010']"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#finding failing inspections date\n",
    "[row['Inspection Date']for row in inspections['34192']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'AKA Name': \"REGGIO'S PIZZA EXPRESS (T3 G3)\",\n",
       " 'Address': '11601 W TOUHY AVE ',\n",
       " 'City': 'CHICAGO',\n",
       " 'DBA Name': 'REGGIOS PIZZA EXPRESS',\n",
       " 'Facility Type': 'Restaurant',\n",
       " 'Inspection Date': '08/16/2016',\n",
       " 'Inspection ID': '1950490',\n",
       " 'Inspection Type': 'License',\n",
       " 'Latitude': '42.008536400868735',\n",
       " 'License #': '2428079',\n",
       " 'Location': '(42.008536400868735, -87.91442843927047)',\n",
       " 'Longitude': '-87.91442843927047',\n",
       " 'Results': 'Fail',\n",
       " 'Risk': 'Risk 1 (High)',\n",
       " 'State': 'IL',\n",
       " 'Violations': '40. REFRIGERATION AND METAL STEM THERMOMETERS PROVIDED AND CONSPICUOUS - Comments: INSTRUCTED TO PROVIDE THERMOMETER VISIBLE AND ACCURATE INSIDE PIZZA HOT HOLDING UNIT. | 33. FOOD AND NON-FOOD CONTACT EQUIPMENT UTENSILS CLEAN, FREE OF ABRASIVE DETERGENTS - Comments: INSTRUCTED TO DETAIL CLEAN AND MAINTAIN INTERIOR SURFACES OF 2 DOOR PREP COOLER (BY EXPOSED HAND SINK). | 16. FOOD PROTECTED DURING STORAGE, PREPARATION, DISPLAY, SERVICE AND TRANSPORTATION - Comments: FOUND INTERIOR SURFACES OF ICE MACHINE IN REAR NOT CLEAN WITH PINK AND BLACK MOLD LIKE BUILD-UP. INSTRUCTED TO WASH, RINSE AND SANITIZE THE AFFECTED AREAS. \\nSERIOUS VIOLATION 7-38-005 (A)',\n",
       " 'Zip': '60666'}"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#what is the most common way that a place at o'hare fails the inspection\n",
    "#numeric codes and comments\n",
    "ohare[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['40. REFRIGERATION AND METAL STEM THERMOMETERS PROVIDED AND CONSPICUOUS - Comments: INSTRUCTED TO PROVIDE THERMOMETER VISIBLE AND ACCURATE INSIDE PIZZA HOT HOLDING UNIT. ',\n",
       " ' 33. FOOD AND NON-FOOD CONTACT EQUIPMENT UTENSILS CLEAN, FREE OF ABRASIVE DETERGENTS - Comments: INSTRUCTED TO DETAIL CLEAN AND MAINTAIN INTERIOR SURFACES OF 2 DOOR PREP COOLER (BY EXPOSED HAND SINK). ',\n",
       " ' 16. FOOD PROTECTED DURING STORAGE, PREPARATION, DISPLAY, SERVICE AND TRANSPORTATION - Comments: FOUND INTERIOR SURFACES OF ICE MACHINE IN REAR NOT CLEAN WITH PINK AND BLACK MOLD LIKE BUILD-UP. INSTRUCTED TO WASH, RINSE AND SANITIZE THE AFFECTED AREAS. \\nSERIOUS VIOLATION 7-38-005 (A)']"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ohare[1]['Violations'].split('|')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "violations = _"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['40. REFRIGERATION AND METAL STEM THERMOMETERS PROVIDED AND CONSPICUOUS - Comments: INSTRUCTED TO PROVIDE THERMOMETER VISIBLE AND ACCURATE INSIDE PIZZA HOT HOLDING UNIT. ',\n",
       " ' 33. FOOD AND NON-FOOD CONTACT EQUIPMENT UTENSILS CLEAN, FREE OF ABRASIVE DETERGENTS - Comments: INSTRUCTED TO DETAIL CLEAN AND MAINTAIN INTERIOR SURFACES OF 2 DOOR PREP COOLER (BY EXPOSED HAND SINK). ',\n",
       " ' 16. FOOD PROTECTED DURING STORAGE, PREPARATION, DISPLAY, SERVICE AND TRANSPORTATION - Comments: FOUND INTERIOR SURFACES OF ICE MACHINE IN REAR NOT CLEAN WITH PINK AND BLACK MOLD LIKE BUILD-UP. INSTRUCTED TO WASH, RINSE AND SANITIZE THE AFFECTED AREAS. \\nSERIOUS VIOLATION 7-38-005 (A)']"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "violations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['40. REFRIGERATION AND METAL STEM THERMOMETERS PROVIDED AND CONSPICUOUS',\n",
       " '33. FOOD AND NON-FOOD CONTACT EQUIPMENT UTENSILS CLEAN, FREE OF ABRASIVE DETERGENTS',\n",
       " '16. FOOD PROTECTED DURING STORAGE, PREPARATION, DISPLAY, SERVICE AND TRANSPORTATION']"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[v[:v.find('- Comments:')].strip() for v in violations] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "all_violations = [row['Violations'].split('|') for row in ohare]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "c = Counter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for violations in all_violations:\n",
    "    for v in violations:\n",
    "        c[v[:v.find('- Comments:')].strip()]+=1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('33. FOOD AND NON-FOOD CONTACT EQUIPMENT UTENSILS CLEAN, FREE OF ABRASIVE DETERGENTS',\n",
       "  107),\n",
       " ('34. FLOORS: CONSTRUCTED PER CODE, CLEANED, GOOD REPAIR, COVING INSTALLED, DUST-LESS CLEANING METHODS USED',\n",
       "  104),\n",
       " ('35. WALLS, CEILINGS, ATTACHED EQUIPMENT CONSTRUCTED PER CODE: GOOD REPAIR, SURFACES CLEAN AND DUST-LESS CLEANING METHODS',\n",
       "  89),\n",
       " ('18. NO EVIDENCE OF RODENT OR INSECT OUTER OPENINGS PROTECTED/RODENT PROOFED, A WRITTEN LOG SHALL BE MAINTAINED AVAILABLE TO THE INSPECTORS',\n",
       "  80),\n",
       " ('32. FOOD AND NON-FOOD CONTACT SURFACES PROPERLY DESIGNED, CONSTRUCTED AND MAINTAINED',\n",
       "  69)]"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c.most_common(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When to use pandas?\n",
    "\n",
    "- CSVs\n",
    "- Production/corner cases\n",
    "\n",
    "Built-in stuff is useful for very unstructured/messy data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<a id ='production'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Productionalizing a Data Science Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "[Tudor Radoaca](https://www.linkedin.com/in/tudor-radoaca-b7a65349): Software Engineer\n",
    "\n",
    "[Nicole Carlson](http://parsingscience.com/): Data Scientist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Scoring specialists/Developing Model\n",
    "- Quantifies reliability of a specialist based on retention and completion metrics\n",
    "- Checking score distribution for workers\n",
    "\n",
    "## Handoff\n",
    "- Jupyter notebook with queries and weights and flexibility to change them\n",
    "\n",
    "## Implementation\n",
    "- Specialists can view the score\n",
    "- Persistent Score History\n",
    "- Scalable implementation\n",
    "- Flexible architecture\n",
    "- Accesibility to Data Scientist\n",
    "\n",
    "### Celery\n",
    "- asynchronous task queue using multi-processing\n",
    "- grouped tasks returning results to a comon parent task\n",
    "\n",
    "### Final solution: Bolero\n",
    "- Implementing Score Features\n",
    "- Software engineers changed the model\n",
    "\n",
    "## Takeaways\n",
    "1. Talk in person if things get heated\n",
    "2. Document everything\n",
    "3. Ask question\n",
    "4. Involve stakeholders every step of the way\n",
    "5. We can build something amazing together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "sean@shiftgig.com for data careers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<a id='exploratory_analysis'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Exploratory Data Analysis to Discover Patterns in Image and Document Collections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Mehrdad Yazdani](https://twitter.com/crude2refined)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Two problems\n",
    "- How do we know we've engineered our data to be ML friendly? How do we know we engineered the right way?\n",
    "- We have lots of data: now what?\n",
    "\n",
    ">What to do with large amounts of unlabeled data?\n",
    "\n",
    "[Lev Manovich](http://manovich.net/)\n",
    "\n",
    "[Pyimageplot](https://github.com/myazdani/pyImagePlot)\n",
    "\n",
    "> The simple graph has brought more information to the data analyst's mind than any other device\n",
    "~ John Tukey\n",
    "\n",
    "## EDA: two main approaches\n",
    "- Statistical modeling (summary statistics)\n",
    "    * Scalable\n",
    "    * Cons: models are based on assumptions that may be wrong\n",
    "- Visualization\n",
    "    * not scalable\n",
    "    * no assumptions are made\n",
    "\n",
    "- EDA on a handful of variables is straightforward\n",
    "    * just plot every pair!\n",
    "    \n",
    "- Otherwise\n",
    "    * PCA\n",
    "    * Multidimensional scaling\n",
    "    * t-SNE\n",
    "    \n",
    "## Feature engineering\n",
    "- Manually designing the input x\n",
    "\n",
    "## Pipeline\n",
    "\n",
    "$Data \\rightarrow Extract \\> Features \\rightarrow Visualize$\n",
    "\n",
    "- Montage\n",
    "- Image histogram\n",
    "\n",
    "- Pretrained neural networks can be used as feature extractors\n",
    "- get features from a pre-trained neural net in [Caffe](http://caffe.berkeleyvision.org/)\n",
    "- SkiCaffe : scikit_learn wrapper for extracting features\n",
    "\n",
    "## Visualizing collections of documents\n",
    "- generate term-frequency for each document\n",
    "- visualize with word clouds\n",
    "- generate tf-idf\n",
    "    * term frequency-inverse document frequency"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='genotype'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Genotype-Phenotype Associations and Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Mat Kallada](https://twitter.com/mathewkallada)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ML Problems in Medicine\n",
    "### Drug Design\n",
    "- Prediction problem: predict if a particular molecule will stick onto a part of the bacteria to a point where it doesn't move\n",
    "- Extract features from bacterial protein + molecule, train classifier, and look for binary outcome\n",
    "### Vaccine discovery\n",
    "- Predict which parts of the infectious disease are useful for developing a vaccine\n",
    "### Disease prediction\n",
    "- given someone's dna, predict if this person has a disease\n",
    "\n",
    "\n",
    "## Diseases\n",
    "### Background\n",
    "- DNA: a raw feature vector representation for life\n",
    "- genome: subset of dna that creates proteins\n",
    "\n",
    "### Classification Formulation\n",
    "- Standard supervised ML task\n",
    "- Training pairs, feature engineering, train model\n",
    "#### Problems\n",
    "- Representing sequential data\n",
    "- Missing genetic data\n",
    "- High dimensional space\n",
    "- Typically imbalanced/skewed class distribution\n",
    "\n",
    "### Genome feature vector representations\n",
    "- by definition, something that is fixed length\n",
    "- SNP Vectors\n",
    "    * Single nucleotide polymoriphisms\n",
    "    * Random insertions/deletions or substitutions\n",
    "    * Aline with reference genome (use alignment algorithm)\n",
    "    * record snps: record the differences\n",
    "    * problem: genomes have 3.3 billion amino acids\n",
    "- Gene presence/Absence Vectors\n",
    "    * apply a clustering algorithm on the genes\n",
    "    * cluster membership determines new features\n",
    "    * Has or has not gene \n",
    "    * does not account for mutations\n",
    "    * DBSCAN: similarity metric = edit distance (usually hamming distance)\n",
    "- Character n-grams\n",
    "    * K-mers\n",
    "    * as K -> 1, sparser feature space\n",
    "    * as K-> m, binary feature space\n",
    "    * sliding window representation\n",
    "    * alignment-based approaches are computationally expensive\n",
    "    \n",
    "### Gene Feature Selection\n",
    "- Human genome has 3.3 billion nucleic acids, 20K genes\n",
    "- How to perform feature selection?\n",
    "    * computational limitations\n",
    "    * constraints on hypothesis space\n",
    "    * interpret important features\n",
    "- Two generic FS approaches\n",
    "    * univariate feature selection\n",
    "    * multivariate selection\n",
    "- Univariate feature selection\n",
    "    * compute a metric between each feature and label: accuracy, correlation\n",
    "- Multivariate feature selection\n",
    "    * construct all models with feature subsets of size K\n",
    "    * evaluate accuract for all models\n",
    "    * keep all features with a certain accuracy\n",
    "### Gene feature imputation\n",
    "- Missing data problem\n",
    "- Missing at random\n",
    "- missing by design\n",
    "- missing at completely random\n",
    "- Mean/deletion strategies\n",
    "- Model-based imputation (python libary fancyimpute)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='ml_cleaning'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning for Cleaning Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Cathy Deng](https://twitter.com/cthydng)\n",
    "\n",
    "[Forest Gregg](https://github.com/fgregg)\n",
    "\n",
    "[Slides](https://docs.google.com/presentation/d/1k42esoWoc_WezfPfQ5vxbHTsuFOvAshEusD-GFCElTQ/edit#slide=id.p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sources of Ambiguity\n",
    "- Data entered by humans\n",
    "- Data involving free text\n",
    "- Data without unique identifiers\n",
    "## Machine Learning\n",
    "- Work at scale\n",
    "- leave an audit trail\n",
    "- because regex is often painful\n",
    "## Libraries\n",
    "- [US Addresses](https://github.com/datamade/usaddress)\n",
    "- a python library for inferring structure in addresses\n",
    "- use conditional random fields\n",
    "    * learns feature of individual components\n",
    "    * learns relative order of components\n",
    "- Put together a lot of training data (real, parsed addresses)\n",
    "- feed through model which pays attention to features and order\n",
    "- [Probable People](https://github.com/datamade/probablepeople)\n",
    "    * Campaign finance data\n",
    "- [Parserator](https://github.com/datamade/parserator)\n",
    "## More data cleaning problems\n",
    "### De-duplicating data\n",
    "* What is similarity within a column?\n",
    "* What is similarity across columns?\n",
    "* How can those decisions be made quickly with lots of data?\n",
    "* [Dedupe package](https://github.com/datamade/dedupe)\n",
    "    * record similarity\n",
    "    * *learns* from the dataset\n",
    "    * Smart comparisons\n",
    "        * only compare records that share the first 5 characters\n",
    "- Can handle ~ 1 million records in a couple hours\n",
    "    * 90% precision & 90% recall on test datasets\n",
    "    * [dedupe.io](http://dedupe.io) as service\n",
    "        * data review & validation steps with active learning\n",
    "        * distributed tasks\n",
    "        * build-in record linkage\n",
    "        * API access \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Resources from talks I didn't attend\n",
    "[Data Cleaning Tool](https://github.com/elliewix/data-profile-tool)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}