notebooks/02_numerical_pipeline_scaling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "81dd3fc7",
   "metadata": {},
   "source": [
    "# Preprocessing for numerical features\n",
    "\n",
    "In this notebook, we will still use only numerical features.\n",
    "\n",
    "We will introduce these new aspects:\n",
    "\n",
    "* an example of preprocessing, namely **scaling numerical variables**;\n",
    "* using a scikit-learn **pipeline** to chain preprocessing and model\n",
    "  training;\n",
    "* assessing the statistical performance of our model via **cross-validation**\n",
    "  instead of a single train-test split.\n",
    "\n",
    "## Data preparation\n",
    "\n",
    "First, let's load the full adult census dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c2565827",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:23:59.738777Z",
     "start_time": "2021-05-21T10:23:59.498798Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "adult_census = pd.read_csv(\"../datasets/adult-census.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "89b11a48",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:24:00.867645Z",
     "start_time": "2021-05-21T10:24:00.647307Z"
    }
   },
   "outputs": [],
   "source": [
    "# to display nice model diagram\n",
    "from sklearn import set_config\n",
    "set_config(display='diagram')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f9a7970",
   "metadata": {},
   "source": [
    "We will now drop the target from the data we will use to train our\n",
    "predictive model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0de7e635",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:24:03.509009Z",
     "start_time": "2021-05-21T10:24:03.502552Z"
    }
   },
   "outputs": [],
   "source": [
    "target_name = \"class\"\n",
    "target = adult_census[target_name]\n",
    "data = adult_census.drop(columns=target_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9750d41a",
   "metadata": {},
   "source": [
    "Then, we select only the numerical columns, as seen in the previous\n",
    "notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "337c29b7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:24:05.099272Z",
     "start_time": "2021-05-21T10:24:05.085115Z"
    }
   },
   "outputs": [],
   "source": [
    "numerical_columns = [\n",
    "    \"age\", \"capital-gain\", \"capital-loss\", \"hours-per-week\"]\n",
    "\n",
    "data_numeric = data[numerical_columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e05cb1d4",
   "metadata": {},
   "source": [
    "Finally, we can divide our dataset into a train and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "2fb103ee",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:24:06.378269Z",
     "start_time": "2021-05-21T10:24:06.332109Z"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data_numeric, target, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e2f4ff5",
   "metadata": {},
   "source": [
    "## Model fitting with preprocessing\n",
    "\n",
    "A range of preprocessing algorithms in scikit-learn allow us to transform\n",
    "the input data before training a model. In our case, we will standardize the\n",
    "data and then train a new logistic regression model on that new version of\n",
    "the dataset.\n",
    "\n",
    "Let's start by printing some statistics about the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f4d6cfa6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:24:41.696049Z",
     "start_time": "2021-05-21T10:24:41.662017Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>capital-gain</th>\n",
       "      <th>capital-loss</th>\n",
       "      <th>hours-per-week</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>36631.000000</td>\n",
       "      <td>36631.000000</td>\n",
       "      <td>36631.000000</td>\n",
       "      <td>36631.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>38.642352</td>\n",
       "      <td>1087.077721</td>\n",
       "      <td>89.665311</td>\n",
       "      <td>40.431247</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>13.725748</td>\n",
       "      <td>7522.692939</td>\n",
       "      <td>407.110175</td>\n",
       "      <td>12.423952</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>17.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>28.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>40.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>37.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>40.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>48.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>45.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>90.000000</td>\n",
       "      <td>99999.000000</td>\n",
       "      <td>4356.000000</td>\n",
       "      <td>99.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                age  capital-gain  capital-loss  hours-per-week\n",
       "count  36631.000000  36631.000000  36631.000000    36631.000000\n",
       "mean      38.642352   1087.077721     89.665311       40.431247\n",
       "std       13.725748   7522.692939    407.110175       12.423952\n",
       "min       17.000000      0.000000      0.000000        1.000000\n",
       "25%       28.000000      0.000000      0.000000       40.000000\n",
       "50%       37.000000      0.000000      0.000000       40.000000\n",
       "75%       48.000000      0.000000      0.000000       45.000000\n",
       "max       90.000000  99999.000000   4356.000000       99.000000"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4f17e9c",
   "metadata": {},
   "source": [
    "We see that the dataset's features span across different ranges. Some\n",
    "algorithms make some assumptions regarding the feature distributions and\n",
    "usually normalizing features will be helpful to address these assumptions.\n",
    "\n",
    "<div class=\"admonition tip alert alert-warning\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Tip</p>\n",
    "<p>Here are some reasons for scaling features:</p>\n",
    "<ul class=\"last simple\">\n",
    "<li>Models that rely on the distance between a pair of samples, for instance\n",
    "k-nearest neighbors, should be trained on normalized features to make each\n",
    "feature contribute approximately equally to the distance computations.</li>\n",
    "<li>Many models such as logistic regression use a numerical solver (based on\n",
    "gradient descent) to find their optimal parameters. This solver converges\n",
    "faster when the features are scaled.</li>\n",
    "</ul>\n",
    "</div>\n",
    "\n",
    "Whether or not a machine learning model requires scaling the features depends\n",
    "on the model family. Linear models such as logistic regression generally\n",
    "benefit from scaling the features while other models such as decision trees\n",
    "do not need such preprocessing (but will not suffer from it).\n",
    "\n",
    "We show how to apply such normalization using a scikit-learn transformer\n",
    "called `StandardScaler`. This transformer shifts and scales each feature\n",
    "individually so that they all have a 0-mean and a unit standard deviation.\n",
    "\n",
    "We will investigate different steps used in scikit-learn to achieve such a\n",
    "transformation of the data.\n",
    "\n",
    "First, one needs to call the method `fit` in order to learn the scaling from\n",
    "the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2e23d8f3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:25:58.460583Z",
     "start_time": "2021-05-21T10:25:58.454310Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-69b329d2-d7e5-4720-9293-1542e57a999c {color: black;background-color: white;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c pre{padding: 0;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-toggleable {background-color: white;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.2em 0.3em;box-sizing: border-box;text-align: center;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-estimator {font-family: monospace;background-color: #f0f8ff;margin: 0.25em 0.25em;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-estimator:hover {background-color: #d4ebff;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-item {z-index: 1;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-parallel-item:only-child::after {width: 0;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0.2em;box-sizing: border-box;padding-bottom: 0.1em;background-color: white;position: relative;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-69b329d2-d7e5-4720-9293-1542e57a999c div.sk-container {display: inline-block;position: relative;}</style><div id=\"sk-69b329d2-d7e5-4720-9293-1542e57a999c\" class\"sk-top-container\"><div class=\"sk-container\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"a0154cc3-9ae2-485c-9b73-67153d602ab7\" type=\"checkbox\" checked><label class=\"sk-toggleable__label\" for=\"a0154cc3-9ae2-485c-9b73-67153d602ab7\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div>"
      ],
      "text/plain": [
       "StandardScaler()"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "scaler = StandardScaler()\n",
    "scaler.fit(data_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "269e470d",
   "metadata": {},
   "source": [
    "The `fit` method for transformers is similar to the `fit` method for\n",
    "predictors. The main difference is that the former has a single argument (the\n",
    "data matrix), whereas the latter has two arguments (the data matrix and the\n",
    "target).\n",
    "\n",
    "![Transformer fit diagram](../figures/api_diagram-transformer.fit.svg)\n",
    "\n",
    "In this case, the algorithm needs to compute the mean and standard deviation\n",
    "for each feature and store them into some NumPy arrays. Here, these\n",
    "statistics are the model states.\n",
    "\n",
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">The fact that the model states of this scaler are arrays of means and\n",
    "standard deviations is specific to the <tt class=\"docutils literal\">StandardScaler</tt>. Other\n",
    "scikit-learn transformers will compute different statistics and store them\n",
    "as model states, in the same fashion.</p>\n",
    "</div>\n",
    "\n",
    "We can inspect the computed means and standard deviations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8cd8c9cf",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:26:37.945394Z",
     "start_time": "2021-05-21T10:26:37.941927Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  38.64235211, 1087.07772106,   89.6653108 ,   40.43124676])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler.mean_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "35ba6356",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:26:44.999672Z",
     "start_time": "2021-05-21T10:26:44.996949Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  13.72556083, 7522.59025606,  407.10461772,   12.42378265])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scaler.scale_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b1a233a",
   "metadata": {},
   "source": [
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">scikit-learn convention: if an attribute is learned from the data, its name\n",
    "ends with an underscore (i.e. <tt class=\"docutils literal\">_</tt>), as in <tt class=\"docutils literal\">mean_</tt> and <tt class=\"docutils literal\">scale_</tt> for the\n",
    "<tt class=\"docutils literal\">StandardScaler</tt>.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08f984b4",
   "metadata": {},
   "source": [
    "Scaling the data is applied to each feature individually (i.e. each column in\n",
    "the data matrix). For each feature, we subtract its mean and divide by its\n",
    "standard deviation.\n",
    "\n",
    "Once we have called the `fit` method, we can perform data transformation by\n",
    "calling the method `transform`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "3e25c6e3",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:28:41.697287Z",
     "start_time": "2021-05-21T10:28:41.693358Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.17177061, -0.14450843,  5.71188483, -2.28845333],\n",
       "       [ 0.02605707, -0.14450843, -0.22025127, -0.27618374],\n",
       "       [-0.33822677, -0.14450843, -0.22025127,  0.77019645],\n",
       "       ...,\n",
       "       [-0.77536738, -0.14450843, -0.22025127, -0.03471139],\n",
       "       [ 0.53605445, -0.14450843, -0.22025127, -0.03471139],\n",
       "       [ 1.48319243, -0.14450843, -0.22025127, -2.69090725]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train_scaled = scaler.transform(data_train)\n",
    "data_train_scaled"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cec6edc",
   "metadata": {},
   "source": [
    "Let's illustrate the internal mechanism of the `transform` method and put it\n",
    "to perspective with what we already saw with predictors.\n",
    "\n",
    "![Transformer transform diagram](../figures/api_diagram-transformer.transform.svg)\n",
    "\n",
    "The `transform` method for transformers is similar to the `predict` method\n",
    "for predictors. It uses a predefined function, called a **transformation\n",
    "function**, and uses the model states and the input data. However, instead of\n",
    "outputting predictions, the job of the `transform` method is to output a\n",
    "transformed version of the input data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4148e9d",
   "metadata": {},
   "source": [
    "Finally, the method `fit_transform` is a shorthand method to call\n",
    "successively `fit` and then `transform`.\n",
    "\n",
    "![Transformer fit_transform diagram](../figures/api_diagram-transformer.fit_transform.svg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "82eb0eb4",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:29:52.076299Z",
     "start_time": "2021-05-21T10:29:52.069286Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.17177061, -0.14450843,  5.71188483, -2.28845333],\n",
       "       [ 0.02605707, -0.14450843, -0.22025127, -0.27618374],\n",
       "       [-0.33822677, -0.14450843, -0.22025127,  0.77019645],\n",
       "       ...,\n",
       "       [-0.77536738, -0.14450843, -0.22025127, -0.03471139],\n",
       "       [ 0.53605445, -0.14450843, -0.22025127, -0.03471139],\n",
       "       [ 1.48319243, -0.14450843, -0.22025127, -2.69090725]])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train_scaled = scaler.fit_transform(data_train)\n",
    "data_train_scaled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "cbd1b750",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:29:57.902253Z",
     "start_time": "2021-05-21T10:29:57.885511Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>capital-gain</th>\n",
       "      <th>capital-loss</th>\n",
       "      <th>hours-per-week</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>3.663100e+04</td>\n",
       "      <td>3.663100e+04</td>\n",
       "      <td>3.663100e+04</td>\n",
       "      <td>3.663100e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>-2.273364e-16</td>\n",
       "      <td>3.530310e-17</td>\n",
       "      <td>3.840667e-17</td>\n",
       "      <td>1.844684e-16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.000014e+00</td>\n",
       "      <td>1.000014e+00</td>\n",
       "      <td>1.000014e+00</td>\n",
       "      <td>1.000014e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-1.576792e+00</td>\n",
       "      <td>-1.445084e-01</td>\n",
       "      <td>-2.202513e-01</td>\n",
       "      <td>-3.173852e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>-7.753674e-01</td>\n",
       "      <td>-1.445084e-01</td>\n",
       "      <td>-2.202513e-01</td>\n",
       "      <td>-3.471139e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>-1.196565e-01</td>\n",
       "      <td>-1.445084e-01</td>\n",
       "      <td>-2.202513e-01</td>\n",
       "      <td>-3.471139e-02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6.817680e-01</td>\n",
       "      <td>-1.445084e-01</td>\n",
       "      <td>-2.202513e-01</td>\n",
       "      <td>3.677425e-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>3.741752e+00</td>\n",
       "      <td>1.314865e+01</td>\n",
       "      <td>1.047970e+01</td>\n",
       "      <td>4.714245e+00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                age  capital-gain  capital-loss  hours-per-week\n",
       "count  3.663100e+04  3.663100e+04  3.663100e+04    3.663100e+04\n",
       "mean  -2.273364e-16  3.530310e-17  3.840667e-17    1.844684e-16\n",
       "std    1.000014e+00  1.000014e+00  1.000014e+00    1.000014e+00\n",
       "min   -1.576792e+00 -1.445084e-01 -2.202513e-01   -3.173852e+00\n",
       "25%   -7.753674e-01 -1.445084e-01 -2.202513e-01   -3.471139e-02\n",
       "50%   -1.196565e-01 -1.445084e-01 -2.202513e-01   -3.471139e-02\n",
       "75%    6.817680e-01 -1.445084e-01 -2.202513e-01    3.677425e-01\n",
       "max    3.741752e+00  1.314865e+01  1.047970e+01    4.714245e+00"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_train_scaled = pd.DataFrame(data_train_scaled,\n",
    "                                 columns=data_train.columns)\n",
    "data_train_scaled.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6a356bb",
   "metadata": {},
   "source": [
    "We can easily combine these sequential operations with a scikit-learn\n",
    "`Pipeline`, which chains together operations and is used as any other\n",
    "classifier or regressor. The helper function `make_pipeline` will create a\n",
    "`Pipeline`: it takes as arguments the successive transformations to perform,\n",
    "followed by the classifier or regressor model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "6eee30aa",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:30:11.214570Z",
     "start_time": "2021-05-21T10:30:11.197277Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 {color: black;background-color: white;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 pre{padding: 0;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-toggleable {background-color: white;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.2em 0.3em;box-sizing: border-box;text-align: center;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;margin: 0.25em 0.25em;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-estimator:hover {background-color: #d4ebff;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 2em;bottom: 0;left: 50%;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-item {z-index: 1;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-parallel-item {display: flex;flex-direction: column;position: relative;background-color: white;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-parallel-item:only-child::after {width: 0;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0.2em;box-sizing: border-box;padding-bottom: 0.1em;background-color: white;position: relative;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-label label {font-family: monospace;font-weight: bold;background-color: white;display: inline-block;line-height: 1.2em;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-label-container {position: relative;z-index: 2;text-align: center;}#sk-883f5907-99c6-4b23-9f93-be65f6fcda05 div.sk-container {display: inline-block;position: relative;}</style><div id=\"sk-883f5907-99c6-4b23-9f93-be65f6fcda05\" class\"sk-top-container\"><div class=\"sk-container\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"15617cf2-e24c-4a2c-a82e-febd1039391c\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"15617cf2-e24c-4a2c-a82e-febd1039391c\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[('standardscaler', StandardScaler()),\n",
       "                ('logisticregression', LogisticRegression())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"3067d81a-f081-44b7-8399-61a1ad5a15b5\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"3067d81a-f081-44b7-8399-61a1ad5a15b5\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"85c0edea-9497-4840-a8b5-35ec994f184a\" type=\"checkbox\" ><label class=\"sk-toggleable__label\" for=\"85c0edea-9497-4840-a8b5-35ec994f184a\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression()</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('standardscaler', StandardScaler()),\n",
       "                ('logisticregression', LogisticRegression())])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import time\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "model = make_pipeline(StandardScaler(), LogisticRegression())\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "123d0e10",
   "metadata": {},
   "source": [
    "The `make_pipeline` function did not require us to give a name to each step.\n",
    "Indeed, it was automatically assigned based on the name of the classes\n",
    "provided; a `StandardScaler` will be a step named `\"standardscaler\"` in the\n",
    "resulting pipeline. We can check the name of each steps of our model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6808b772",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:30:37.336504Z",
     "start_time": "2021-05-21T10:30:37.333676Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'standardscaler': StandardScaler(),\n",
       " 'logisticregression': LogisticRegression()}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.named_steps"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5de71c68",
   "metadata": {},
   "source": [
    "This predictive pipeline exposes the same methods as the final predictor:\n",
    "`fit` and `predict` (and additionally `predict_proba`, `decision_function`,\n",
    "or `score`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "f6510403",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:30:50.971587Z",
     "start_time": "2021-05-21T10:30:50.900208Z"
    }
   },
   "outputs": [],
   "source": [
    "start = time.time()\n",
    "model.fit(data_train, target_train)\n",
    "elapsed_time = time.time() - start"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f627dbf3",
   "metadata": {},
   "source": [
    "We can represent the internal mechanism of a pipeline when calling `fit`\n",
    "by the following diagram:\n",
    "\n",
    "![pipeline fit diagram](../figures/api_diagram-pipeline.fit.svg)\n",
    "\n",
    "When calling `model.fit`, the method `fit_transform` from each underlying\n",
    "transformer (here a single transformer) in the pipeline will be called to:\n",
    "\n",
    "- learn their internal model states\n",
    "- transform the training data. Finally, the preprocessed data are provided to\n",
    "  train the predictor.\n",
    "\n",
    "To predict the targets given a test set, one uses the `predict` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "08a83c06",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:31:43.979180Z",
     "start_time": "2021-05-21T10:31:43.971463Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([' <=50K', ' <=50K', ' >50K', ' <=50K', ' <=50K'], dtype=object)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predicted_target = model.predict(data_test)\n",
    "predicted_target[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "668254af",
   "metadata": {},
   "source": [
    "Let's show the underlying mechanism:\n",
    "\n",
    "![pipeline predict diagram](../figures/api_diagram-pipeline.predict.svg)\n",
    "\n",
    "The method `transform` of each transformer (here a single transformer) is\n",
    "called to preprocess the data. Note that there is no need to call the `fit`\n",
    "method for these transformers because we are using the internal model states\n",
    "computed when calling `model.fit`. The preprocessed data is then provided to\n",
    "the predictor that will output the predicted target by calling its method\n",
    "`predict`.\n",
    "\n",
    "As a shorthand, we can check the score of the full predictive pipeline\n",
    "calling the method `model.score`. Thus, let's check the computational and\n",
    "statistical performance of such a predictive pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "2514017b",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:32:26.854694Z",
     "start_time": "2021-05-21T10:32:26.821556Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The accuracy using a Pipeline is 0.807 with a fitting time of 0.067 seconds in 12 iterations\n"
     ]
    }
   ],
   "source": [
    "model_name = model.__class__.__name__\n",
    "score = model.score(data_test, target_test)\n",
    "print(f\"The accuracy using a {model_name} is {score:.3f} \"\n",
    "      f\"with a fitting time of {elapsed_time:.3f} seconds \"\n",
    "      f\"in {model[-1].n_iter_[0]} iterations\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "545fd684",
   "metadata": {},
   "source": [
    "We could compare this predictive model with the predictive model used in\n",
    "the previous notebook which did not scale features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "7f9e26cc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:32:40.178110Z",
     "start_time": "2021-05-21T10:32:40.005757Z"
    }
   },
   "outputs": [],
   "source": [
    "model = LogisticRegression()\n",
    "start = time.time()\n",
    "model.fit(data_train, target_train)\n",
    "elapsed_time = time.time() - start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6323f710",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:32:41.390236Z",
     "start_time": "2021-05-21T10:32:41.357344Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The accuracy using a LogisticRegression is 0.807 with a fitting time of 0.171 seconds in 59 iterations\n"
     ]
    }
   ],
   "source": [
    "model_name = model.__class__.__name__\n",
    "score = model.score(data_test, target_test)\n",
    "print(f\"The accuracy using a {model_name} is {score:.3f} \"\n",
    "      f\"with a fitting time of {elapsed_time:.3f} seconds \"\n",
    "      f\"in {model.n_iter_[0]} iterations\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75aebd9a",
   "metadata": {},
   "source": [
    "We see that scaling the data before training the logistic regression was\n",
    "beneficial in terms of computational performance. Indeed, the number of\n",
    "iterations decreased as well as the training time. The statistical\n",
    "performance did not change since both models converged.\n",
    "\n",
    "<div class=\"admonition warning alert alert-danger\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Warning</p>\n",
    "<p class=\"last\">Working with non-scaled data will potentially force the algorithm to iterate\n",
    "more as we showed in the example above. There is also the catastrophic\n",
    "scenario where the number of required iterations are more than the maximum\n",
    "number of iterations allowed by the predictor (controlled by the <tt class=\"docutils literal\">max_iter</tt>)\n",
    "parameter. Therefore, before increasing <tt class=\"docutils literal\">max_iter</tt>, make sure that the data\n",
    "are well scaled.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4213c869",
   "metadata": {},
   "source": [
    "## Model evaluation using cross-validation\n",
    "\n",
    "In the previous example, we split the original data into a training set and a\n",
    "testing set. This strategy has several issues: in a setting where the amount\n",
    "of data is small, the subset used to train or test will be small. Besides, a\n",
    "single split does not give information regarding the confidence of the\n",
    "results obtained.\n",
    "\n",
    "Instead, we can use cross-validation. Cross-validation consists of repeating\n",
    "the procedure such that the training and testing sets are different each\n",
    "time. Statistical performance metrics are collected for each repetition and\n",
    "then aggregated. As a result we can get an estimate of the variability of the\n",
    "model's statistical performance.\n",
    "\n",
    "Note that there exists several cross-validation strategies, each of them\n",
    "defines how to repeat the `fit`/`score` procedure. In this section, we will\n",
    "use the K-fold strategy: the entire dataset is split into `K` partitions. The\n",
    "`fit`/`score` procedure is repeated `K` times where at each iteration `K - 1`\n",
    "partitions are used to fit the model and `1` partition is used to score. The\n",
    "figure below illustrates this K-fold strategy.\n",
    "\n",
    "![Cross-validation diagram](../figures/cross_validation_diagram.png)\n",
    "\n",
    "<div class=\"admonition note alert alert-info\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Note</p>\n",
    "<p class=\"last\">This figure shows the particular case of K-fold cross-validation strategy.\n",
    "As mentioned earlier, there are a variety of different cross-validation\n",
    "strategies. Some of these aspects will be covered in more details in future\n",
    "notebooks.</p>\n",
    "</div>\n",
    "\n",
    "For each cross-validation split, the procedure trains a model on all the red\n",
    "samples and evaluate the score of the model on the blue samples.\n",
    "Cross-validation is therefore computationally intensive because it requires\n",
    "training several models instead of one.\n",
    "\n",
    "In scikit-learn, the function `cross_validate` allows to do cross-validation\n",
    "and you need to pass it the model, the data, and the target. Since there\n",
    "exists several cross-validation strategies, `cross_validate` takes a\n",
    "parameter `cv` which defines the splitting strategy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "c1ec55c5",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:34:26.801439Z",
     "start_time": "2021-05-21T10:34:26.093319Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.71 s, sys: 5.33 s, total: 7.04 s\n",
      "Wall time: 683 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'fit_time': array([0.06771517, 0.10739303, 0.11240768, 0.10973525, 0.10993814]),\n",
       " 'score_time': array([0.02388787, 0.02575541, 0.02660251, 0.02641964, 0.02654028]),\n",
       " 'test_score': array([0.79557785, 0.80049135, 0.79965192, 0.79873055, 0.80436118])}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "from sklearn.model_selection import cross_validate\n",
    "\n",
    "model = make_pipeline(StandardScaler(), LogisticRegression())\n",
    "cv_result = cross_validate(model, data_numeric, target, cv=5)\n",
    "cv_result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd568809",
   "metadata": {},
   "source": [
    "The output of `cross_validate` is a Python dictionary, which by default\n",
    "contains three entries: (i) the time to train the model on the training data\n",
    "for each fold, (ii) the time to predict with the model on the testing data\n",
    "for each fold, and (iii) the default score on the testing data for each fold.\n",
    "\n",
    "Setting `cv=5` created 5 distinct splits to get 5 variations for the training\n",
    "and testing sets. Each training set is used to fit one model which is then\n",
    "scored on the matching test set. This strategy is called K-fold\n",
    "cross-validation where `K` corresponds to the number of splits.\n",
    "\n",
    "Note that by default the `cross_validate` function discards the 5 models that\n",
    "were trained on the different overlapping subset of the dataset. The goal of\n",
    "cross-validation is not to train a model, but rather to estimate\n",
    "approximately the generalization performance of a model that would have been\n",
    "trained to the full training set, along with an estimate of the variability\n",
    "(uncertainty on the generalization accuracy).\n",
    "\n",
    "You can pass additional parameters to `cross_validate` to get more\n",
    "information, for instance training scores. These features will be covered in\n",
    "a future notebook.\n",
    "\n",
    "Let's extract the test scores from the `cv_result` dictionary and compute\n",
    "the mean accuracy and the variation of the accuracy across folds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "6502bafa",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-21T10:35:45.927414Z",
     "start_time": "2021-05-21T10:35:45.925302Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The mean cross-validation accuracy is: 0.800 +/- 0.003\n"
     ]
    }
   ],
   "source": [
    "scores = cv_result[\"test_score\"]\n",
    "print(\"The mean cross-validation accuracy is: \"\n",
    "      f\"{scores.mean():.3f} +/- {scores.std():.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11e2014e",
   "metadata": {},
   "source": [
    "Note that by computing the standard-deviation of the cross-validation scores,\n",
    "we can estimate the uncertainty of our model statistical performance. This is\n",
    "the main advantage of cross-validation and can be crucial in practice, for\n",
    "example when comparing different models to figure out whether one is better\n",
    "than the other or whether the statistical performance differences are within\n",
    "the uncertainty.\n",
    "\n",
    "In this particular case, only the first 2 decimals seem to be trustworthy. If\n",
    "you go up in this notebook, you can check that the performance we get\n",
    "with cross-validation is compatible with the one from a single train-test\n",
    "split."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06d75da1",
   "metadata": {},
   "source": [
    "In this notebook we have:\n",
    "\n",
    "* seen the importance of **scaling numerical variables**;\n",
    "* used a **pipeline** to chain scaling and logistic regression training;\n",
    "* assessed the statistical performance of our model via **cross-validation**."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}