classes/13-expected_value_cost_benefit_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Business Application: Expected Value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Extend our metrics into business application\n",
    "* Identify and understand a cost benefit matrix\n",
    "* Using a confusion matrix with a cost benefit matrix to solve for expected value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Class Notes "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How do we use business goals as evaluation metrics?\n",
    "\n",
    "So far we've discussed one crosswise metric that helps us evaluate false positive rate vs true positive rate (AUC). However, to Everyday Business Guy, AUC means nothing: he is more familiar with accuracy and would rather just have some model that best predicts accuracy. To improve the understanding of the model and connect it to Everyday Business Guy, we want to associate the confusion matrix with an expected value, using a cost benefit analysis matrix.\n",
    "\n",
    "### What is expected value?\n",
    "\n",
    "Expected value associates probabilities for the confusion matrix with values. Expected value says \"this is the expected outcome of the model, should we choose to use it for the business.\"\n",
    "\n",
    "Mathmatically, expected value's equation looks like this:\n",
    "\n",
    "$EV = \\sum p(o_x) * v(o_x)$\n",
    "\n",
    "where:\n",
    "\n",
    "$EV$: Expected Value  \n",
    "$p(o_x)$: [P]robability of [o]bservation [x] occurring  \n",
    "$v(o_x)$: The [v]alue of [o]bservation [x] occuring\n",
    "\n",
    "\n",
    "### How do we find these values?\n",
    "\n",
    "The technique we'll use is a _cost benefit matrix_. This is very similar to our confusion matrix:\n",
    "\n",
    "$\\begin{bmatrix}TP & FP\\\\FN & TN\\end{bmatrix}$\n",
    "\n",
    "Which to find probabilities turns into this:\n",
    "\n",
    "$\\begin{bmatrix}p(TP) & p(FP)\\\\p(FN) & p(TN)\\end{bmatrix}$\n",
    "\n",
    "and our cost benefit matrix will be somewhat similar:\n",
    "\n",
    "$\\begin{bmatrix}b(TP) & c(FP)\\\\c(FN) & b(TN)\\end{bmatrix}$\n",
    "\n",
    "where:\n",
    "\n",
    "$b$ represents benefit (the benefits of accurately predicting positives and negatives), while...   \n",
    "$c$ represents cost (the costs of misclassifying positives and negatives).\n",
    "\n",
    "To simplify we'll associate benefits to be positive value and costs to be negative value:\n",
    "\n",
    "$\\begin{bmatrix}v(TP) & -v(FP)\\\\-v(FN) & v(TN)\\end{bmatrix}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example: Ads for a startup\n",
    "\n",
    "You run a startup service and are interested in acquiring customers. The estimated CLV (customer lifetime value) of a client is about \\$75, while the estimated acquisition cost runs at about \\$30. To maximize profit, you're interested in a model that accurately targets the correct clients for your business. You come to these conclusions on this ad:\n",
    "\n",
    "Benefit of a True Positive: \\$45. Our CLV is \\$75, and to acquire them with this marketing technique is \\$30.<br />\n",
    "Benefit of a True Negative: \\$0. We would have never acquired this customer (0), and we never spent money to acquire them (0).<br />\n",
    "Cost of a False Positive: \\$30. We spent \\$30 on a client that would not have used the service.<br />\n",
    "Cost of a False Negative: \\$0. We did not spend money on this client, so technically our cost is 0, though we would have gained if our model correctly targetted this client.<br />\n",
    "\n",
    "This is represented in our cost benefit matrix:\n",
    "\n",
    "$\\begin{bmatrix}45 & -30\\\\0 & 0\\end{bmatrix}$\n",
    "\n",
    "We end up with a confusion matrix that predicts response to the targeted ad this way:\n",
    "\n",
    "$\\begin{bmatrix}45423 & 13041\\\\98724 & 12324\\end{bmatrix}$\n",
    "\n",
    "and solve for probabilities ($h_x, a$ in this case means hypothesis given all hypothesised values):\n",
    "\n",
    "$p(h_{TP}, a) = 45423 / 169512 = 0.26796333$\n",
    "\n",
    "$p(h_{TN}, a) = 12324 / 169512 = 0.0727028175$\n",
    "\n",
    "$p(h_{FP}, a) = 13041 / 169512 = 0.07693260654$\n",
    "\n",
    "$p(h_{FN}, a) = 98724 / 169512 = 0.5824012459$\n",
    "\n",
    "We'll multiply these probabilities against the cost benefit analysis:\n",
    "\n",
    "$0.26796333 * 45 = 12.05834985$\n",
    "\n",
    "$0.0727028175 * 0 = 0$\n",
    "\n",
    "$0.07693260654 * -30 = -2.307978196$\n",
    "\n",
    "$0.5824012459 * 0 = 0$\n",
    "\n",
    "and sum the results to get the resulting **expected value**:\n",
    "\n",
    "$12.05834985 + 0 + (-2.307978196) + 0 = 9.75$\n",
    "\n",
    "which means on average, if we target the clients and they respond as predicted, we're expected almost a 10 dollar return on clients.\n",
    "\n",
    "Let's have a function that accepts a confusion matrix as well as a the cost benefit matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9.7503716551\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "def find_expected_value(confusion, cost_benefit):\n",
    "    # if you use a probability matrix instead, this next line will return the same matrix back\n",
    "    probabilities = confusion.astype('float') / confusion.sum()\n",
    "    return (probabilities * cost_benefit.astype('float')).sum()\n",
    "\n",
    "cb = np.array([[45.0, -30.0], [0, 0]])\n",
    "\n",
    "conf = np.array([[45423, 13041], [98724, 12324]])\n",
    "print find_expected_value(conf, cb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, the marketing team has drawn up a new ad targeting a different market. This ad is more expensive, but the target market is more specific, which means it's easier to predict if a client will respond to the ad or not. We end up building a model and predicting the confusion matrix to look something like this:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "new_conf = np.array([[62153, 7501], [4735, 32041]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and a cost benefit matrix:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "new_cb = np.array([[25.0, -50.0], [0, 0]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does our end result come out? Does this more expensive ad end up being worth the return?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11.0755895894\n"
     ]
    }
   ],
   "source": [
    "print find_expected_value(new_conf, new_cb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating a new decision line\n",
    "\n",
    "We can use cost benefit analysis to help us draw new decision lines as well. For example, we may want to target a client with an ad only when we a greater likelihood in order to make a profit (> 0). We can use the following math to determine that new boundary:\n",
    "\n",
    "$p(x)~b(TP) - (1 - p(x))~c(FP) > 0$\n",
    "\n",
    "which then simplifies to this:\n",
    "\n",
    "$p(x)~b(TP) > (1 - p(x))~c(FP)$\n",
    "\n",
    "in our first ad, we had a $TP$ benefit of 45 dollars and a $FP$ cost of 30 dollars:\n",
    "\n",
    "$p(x) * 45 > (1 - p(x)) * 30$\n",
    "\n",
    "which then simplifies to (solving $p(x)$):\n",
    "\n",
    "$p(x) > \\dfrac{30}{75} = .4$\n",
    "\n",
    "so we should _only_ target customers with an expected probaility of .40.\n",
    "\n",
    "How does our second ad fair?\n",
    "\n",
    "$p(x) > \\dfrac{25}{75} = .33$\n",
    "\n",
    "This second ad we can target a larger audience in order to maximize likely profit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A side note on sklearn confusion matrices:\n",
    "\n",
    "SKlearn's confusion matrices follow the syntax as specified in [wikipedia](http://en.wikipedia.org/wiki/Confusion_matrix), which is predicted in columns and actual in rows. This is the opposite of what we expect given previous diagrams. An easy solution to address this would be to flip the confusion matrix function, and assign label order:\n",
    "\n",
    "```python\n",
    "metrics.confusion_matrix(y_predicted, y_actual, labels=[1, 0])\n",
    "```\n",
    "\n",
    "And this will create your matrix as expected:\n",
    "\n",
    "$\\begin{bmatrix}TP & FP\\\\FN & TN\\end{bmatrix}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### applications to car auction / lemons\n",
    "\n",
    "In the lemons data set there was a column called \"VehBCost,\" which represents the acquisition price of the vehicle. Imagine you run a car acquisition business that attempts to purchase used vehicles and flip them for 10% more given marginal repairs (let's assume 0). We'll use the benchmark results for probabilities in the confusion matrix, and then assume the following cost benefits for each result:\n",
    "\n",
    "A true positive (TP) was a lemon, so knowing ahead that it would be a bad purchase means our Benefit is 0\n",
    "\n",
    "A true negative (TN) was not a lemon, so purchasing would be on average 10% of the VehBCost\n",
    "\n",
    "A false positive (FP) was not actually a lemon, but our model suggested we shouldn't purchase it. Cost is 0.\n",
    "\n",
    "A false negative (FN) was a lemon, but our model suggested to purchase it. Our cost is the entire VehBCost.\n",
    "\n",
    "our confusion matrix from the benchmark looks like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# computed AUC comes out to be 0.64\n",
    "lemons_confusion = np.array([[292, 66], [5997, 44733]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The average VehBCost is \\$6728.22. We can assume some average expected value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lemons_cb = np.array([[0, 0], [-6278.22, 672.82]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do we end up getting as an expected value?\n",
    "\n",
    "What should our new decision line be?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Your Turn\n",
    "\n",
    "Continue working on personal projects or the lemons project with the remainder of class today."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Resources\n",
    "\n",
    "* [Wikipedia](http://en.wikipedia.org/wiki/Expected_value) is always a good start, but\n",
    "* [Khan Academy](https://www.khanacademy.org/math/probability/random-variables-topic/expected-value/v/term-life-insurance-and-death-probability) has a great set of videos that deep dive into Expected Value.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}