docs/source/objectives/custom_objectives.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Custom Objective Functions\n",
    "\n",
    "Often times, the objective function is very specific to the use-case or business problem. To get the right objective to optimize requires thinking through the decisions or actions that will be taken using the model and assigning a cost/benefit to doing that correctly or incorrectly based on known outcomes in the training data.\n",
    "\n",
    "Once you have determined the objective for your business, you can provide that to EvalML to optimize by defining a custom objective function.\n",
    "\n",
    "## How to Create a Objective Function\n",
    "\n",
    "To create a custom objective function, we must define 2 functions:\n",
    "\n",
    "* The **\"objective function\"**: this function takes the predictions, true labels, and any other information about the future and returns a score of how well the model performed.\n",
    "\n",
    "* The **\"decision function\"**: this function takes prediction probabilities that were output from the model and a threshold and returns a prediction. \n",
    "\n",
    "To evaluate a particular model, EvalML automatically finds the best threshold to pass to the decision function to generate predictions and then scores the resulting predictions using the objective function. The score from the objective function determines which set of pipeline hyperparameters EvalML will try next.\n",
    "\n",
    "To give a concrete example, let's look at how the fraud detection objective function is built."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from evalml.objectives.binary_classification_objective import BinaryClassificationObjective\n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "class FraudCost(BinaryClassificationObjective):\n",
    "    \"\"\"Score the percentage of money lost of the total transaction amount process due to fraud\"\"\"\n",
    "    name = \"Fraud Cost\"\n",
    "    greater_is_better = False\n",
    "    score_needs_proba = False\n",
    "\n",
    "    def __init__(self, retry_percentage=.5, interchange_fee=.02,\n",
    "                 fraud_payout_percentage=1.0, amount_col='amount'):\n",
    "        \"\"\"Create instance of FraudCost\n",
    "\n",
    "        Arguments:\n",
    "            retry_percentage (float): What percentage of customers that will retry a transaction if it\n",
    "                is declined. Between 0 and 1. Defaults to .5\n",
    "\n",
    "            interchange_fee (float): How much of each successful transaction you can collect.\n",
    "                Between 0 and 1. Defaults to .02\n",
    "\n",
    "            fraud_payout_percentage (float): Percentage of fraud you will not be able to collect.\n",
    "                Between 0 and 1. Defaults to 1.0\n",
    "\n",
    "            amount_col (str): Name of column in data that contains the amount. Defaults to \"amount\"\n",
    "        \"\"\"\n",
    "        self.retry_percentage = retry_percentage\n",
    "        self.interchange_fee = interchange_fee\n",
    "        self.fraud_payout_percentage = fraud_payout_percentage\n",
    "        self.amount_col = amount_col\n",
    "\n",
    "    def decision_function(self, ypred_proba, threshold=0.0, X=None):\n",
    "        \"\"\"Determine if a transaction is fraud given predicted probabilities, threshold, and dataframe with transaction amount\n",
    "\n",
    "            Arguments:\n",
    "                ypred_proba (pd.Series): Predicted probablities\n",
    "                X (pd.DataFrame): Dataframe containing transaction amount\n",
    "                threshold (float): Dollar threshold to determine if transaction is fraud\n",
    "\n",
    "            Returns:\n",
    "                pd.Series: Series of predicted fraud labels using X and threshold\n",
    "        \"\"\"\n",
    "        if not isinstance(X, pd.DataFrame):\n",
    "            X = pd.DataFrame(X)\n",
    "\n",
    "        if not isinstance(ypred_proba, pd.Series):\n",
    "            ypred_proba = pd.Series(ypred_proba)\n",
    "\n",
    "        transformed_probs = (ypred_proba.values * X[self.amount_col])\n",
    "        return transformed_probs > threshold\n",
    "\n",
    "    def objective_function(self, y_true, y_predicted, X):\n",
    "        \"\"\"Calculate amount lost to fraud per transaction given predictions, true values, and dataframe with transaction amount\n",
    "\n",
    "            Arguments:\n",
    "                y_predicted (pd.Series): predicted fraud labels\n",
    "                y_true (pd.Series): true fraud labels\n",
    "                X (pd.DataFrame): dataframe with transaction amounts\n",
    "\n",
    "            Returns:\n",
    "                float: amount lost to fraud per transaction\n",
    "        \"\"\"\n",
    "        if not isinstance(X, pd.DataFrame):\n",
    "            X = pd.DataFrame(X)\n",
    "\n",
    "        if not isinstance(y_predicted, pd.Series):\n",
    "            y_predicted = pd.Series(y_predicted)\n",
    "\n",
    "        if not isinstance(y_true, pd.Series):\n",
    "            y_true = pd.Series(y_true)\n",
    "\n",
    "        # extract transaction using the amount columns in users data\n",
    "        try:\n",
    "            transaction_amount = X[self.amount_col]\n",
    "        except KeyError:\n",
    "            raise ValueError(\"`{}` is not a valid column in X.\".format(self.amount_col))\n",
    "\n",
    "        # amount paid if transaction is fraud\n",
    "        fraud_cost = transaction_amount * self.fraud_payout_percentage\n",
    "\n",
    "        # money made from interchange fees on transaction\n",
    "        interchange_cost = transaction_amount * (1 - self.retry_percentage) * self.interchange_fee\n",
    "\n",
    "        # calculate cost of missing fraudulent transactions\n",
    "        false_negatives = (y_true & ~y_predicted) * fraud_cost\n",
    "\n",
    "        # calculate money lost from fees\n",
    "        false_positives = (~y_true & y_predicted) * interchange_cost\n",
    "\n",
    "        loss = false_negatives.sum() + false_positives.sum()\n",
    "\n",
    "        loss_per_total_processed = loss / transaction_amount.sum()\n",
    "\n",
    "        return loss_per_total_processed\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}