sklearn_ML_pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This notebook requires scikit-learn version 0.23.1 or later for some features.\n",
    "# If you are using Google Colab, uncomment the line below, run it, and restart runtime.\n",
    "\n",
    "# !pip install --upgrade scikit-learn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uqGDNgy9oEDT"
   },
   "source": [
    "## Machine Learning Pipelines and Automated Hyperparameter Tuning\n",
    "\n",
    "### [Fred Feng](http://umich.edu/~fredfeng/) (fredfeng@umich.edu)\n",
    "\n",
    "### University of Michigan-Dearborn\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prerequisites\n",
    "\n",
    "- #### Some familiarity with Python and its common libraries \n",
    "    - #### numpy, pandas\n",
    "    - #### [Introduction to Data Analysis in Python](https://youtu.be/7IsFmtvBOyc) workshop\n",
    "\n",
    "\n",
    "- #### Basic understanding of machine learning models (logistic regression)\n",
    "\n",
    "\n",
    "### Goal of this talk\n",
    "\n",
    "- #### Give you a taste of using machine learning pipeline and hyperparameter tuning with scikit-learn.\n",
    "\n",
    "\n",
    "### [Scikit-learn](https://scikit-learn.org/)\n",
    "\n",
    "- #### A machine learning library in Python\n",
    "- #### Open source and free\n",
    "- #### Implemented a large number of common machine learning models\n",
    "- #### Clean, uniform, and streamlined API\n",
    "- #### Widely used across [industries](https://scikit-learn.org/stable/testimonials/testimonials.html) and academia\n",
    "\n",
    "\n",
    "### What scikit-learn is *not* for\n",
    "\n",
    "- #### In-depth statistical analysis, hypothesis testing\n",
    "\n",
    "  - #### [StatsModels](https://www.statsmodels.org/)\n",
    "\n",
    "- #### Deep learning, reinforcement learning\n",
    "\n",
    "  - #### Karas (TensorFlow), PyTorch\n",
    "\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "aI8sGUCwmnQv"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1. Read in and explore the data. \n",
    "\n",
    "### Bank customer data for a marketing campaign [(Data source)](https://archive.ics.uci.edu/ml/datasets/Bank+Marketing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "c2xszAJsnmdy"
   },
   "outputs": [],
   "source": [
    "url = 'http://umich.edu/~fredfeng/workshops/bank.csv'\n",
    "\n",
    "df = pd.read_csv(url)\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ce8RCArknlB3"
   },
   "outputs": [],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Our goal is to develop a model that predicts whether a customer will subscribe the service or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['subscribed'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.groupby('subscribed').mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2. A logistic regression model with a few numerical features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1. Specifying what features to include by constructing a <font color=\"red\">feature matrix</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df[['age', 'balance', 'duration']]\n",
    "\n",
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2. Specify the target (i.e., the output of the model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df['subscribed']\n",
    "\n",
    "y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  A golden rule: <font color=\"red\">Models should never be tested on the same data they were trained on.</font>\n",
    "\n",
    "### Step 3. Split the data to a <font color=\"green\">train set</font> and a <font color=\"DarkViolet\">test set</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, \n",
    "                                                    test_size=.2, \n",
    "                                                    random_state=99, \n",
    "                                                    stratify=y\n",
    "                                                   )\n",
    "\n",
    "X_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4. Instantiate the [logistic regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "clf = LogisticRegression()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5. Train the model by appying the `fit()` method based on the <font color=\"green\">train set</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Coefficients:', clf.coef_, '\\nIntercept:', clf.intercept_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6. Make predictions from the trained model for the <font color=\"DarkViolet\">test set</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 7. Evaluate the model by comparing the predictions with the target in the <font color=\"DarkViolet\">test set</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.column_stack((y_test, y_pred))[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "cm = metrics.confusion_matrix(y_test, y_pred)\n",
    "\n",
    "cm\n",
    "\n",
    "# pd.DataFrame(data=cm, \n",
    "#              columns=['predict: 0', 'predict: 1'], \n",
    "#              index=['true: 0', 'true: 1'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics.accuracy_score(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(metrics.classification_report(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ROC curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "y_pred_proba = clf.predict_proba(X_test)[:, 1]\n",
    "\n",
    "fpr, tpr, _ = roc_curve(y_test, y_pred_proba)\n",
    "\n",
    "plt.figure(figsize=(6, 5))\n",
    "\n",
    "plt.plot(fpr, tpr)\n",
    "plt.plot([0, 1], [0, 1], 'k--')\n",
    "\n",
    "fontsize = 18\n",
    "plt.xlabel('False positive rate', fontsize=fontsize)\n",
    "plt.ylabel('True positive rate', fontsize=fontsize)\n",
    "plt.title('ROC curve', fontsize=fontsize)\n",
    "plt.xticks(fontsize=fontsize)\n",
    "plt.yticks(fontsize=fontsize)\n",
    "plt.xlim([0, 1])\n",
    "plt.ylim([0, 1])\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-validation\n",
    "\n",
    "![k-fold cross validataion](https://miro.medium.com/max/3115/1*me-aJdjnt3ivwAurYkB7PA.png)\n",
    "\n",
    "[Image source](https://medium.com/@sebastiannorena/some-model-tuning-methods-bfef3e6544f0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold\n",
    "\n",
    "cv = KFold(n_splits=5)\n",
    "\n",
    "# cv = KFold(n_splits=5, shuffle=True)\n",
    "\n",
    "# cv = KFold(n_splits=5, shuffle=True, random_state=99)\n",
    "\n",
    "for k in cv.split(X[:15]):\n",
    "    print(k)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for small or unbalanced data it's better to use stratified cross-validation\n",
    "\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=88)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "cross_val_score(clf, X, y, cv=cv, scoring='roc_auc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lastly, we can use the model to make predictions for out-of-sample data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new = pd.DataFrame(data=[[43, 3285, 1721], \n",
    "                           [58, 5920, 255]],\n",
    "                     columns=X.columns, \n",
    "                     index=['Tom', 'Jerry'])\n",
    "\n",
    "X_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf.predict(X_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3. Feature preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df[['marital', 'day', 'duration', 'campaign', 'previous']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X['marital'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, \n",
    "                                                    test_size=.2, \n",
    "                                                    random_state=99, \n",
    "                                                    stratify=y)\n",
    "\n",
    "X_test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing: [Imputation of missing values](https://scikit-learn.org/stable/modules/impute.html)\n",
    "\n",
    "- ### Almost all real world data sets contain missing values.\n",
    "\n",
    "- ### If we discard the row if it contains any missing values, we may end up losing a lot of data that may be valuable.\n",
    "\n",
    "- ### A better strategy is to impute the missing values.\n",
    "\n",
    "\n",
    "### Missing <font color=\"red\">numerical</font> values can be imputed with the mean (by default) or median of the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.isnull().sum() "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X[['day', 'campaign']].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X[['day', 'campaign']].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "SimpleImputer().fit_transform(X[['day', 'campaign']])[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Missing <font color=\"red\">categorical</font> values can be imputed with the <font color=\"green\">most frequent</font> of the column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X['marital'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X['marital'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "marital_imputed = SimpleImputer(strategy='most_frequent').fit_transform(X[['marital']])\n",
    "\n",
    "marital_imputed[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing: Encode categorical features using [One Hot Encoding](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "enc = OneHotEncoder(sparse=False)\n",
    "\n",
    "enc.fit_transform(marital_imputed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "enc.categories_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocessing: Scale the features\n",
    "\n",
    "### [StandardScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html): scale a feature to zero mean and unit variance\n",
    "\n",
    "### [MinMaxScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html): scale a feature to a given range (from 0 to 1 by default)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X['duration'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler, MinMaxScaler\n",
    "\n",
    "StandardScaler().fit(X[['duration']]).transform(X[['duration']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "StandardScaler().fit_transform(X[['duration']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MinMaxScaler().fit_transform(X[['duration']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [FunctionTransformer](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.FunctionTransformer.html): Constructs a transformer from a custom function.\n",
    "\n",
    "### Here we make a logarithm transformation based on the  numpy [`log1p`](https://numpy.org/doc/stable/reference/generated/numpy.log1p.html) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import FunctionTransformer\n",
    "\n",
    "log_transformer = FunctionTransformer(np.log1p)\n",
    "\n",
    "log_transformer.fit_transform(df[['duration']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 4. Machine learning pipelines\n",
    "\n",
    "<img src=\"https://cdn.pixabay.com/photo/2014/10/30/23/04/pressure-water-line-509871_1280.jpg\" alt=\"pipeline\" style=\"width: 600px;\"/>\n",
    "\n",
    "<br>\n",
    "\n",
    "<img src=\"https://github.com/frfeng/sklearn-tutorial/raw/main/pipeline.png\" alt=\"pipeline\" style=\"width: 600px;\"/>\n",
    "\n",
    "### Why pipelines?\n",
    "\n",
    "- ### It simplifies and automates the machine learning workflow.\n",
    "\n",
    "- ### Separation of concerns: it separates the workflow into modular and reusable parts.\n",
    "\n",
    "- ### It makes it harder to make mistakes.\n",
    "\n",
    "    - ### Ensure the same preprocessings being used for the train, test, and out-of-sample data\n",
    "\n",
    "### \"*If you are not using a pipeline, you are probably doing it wrong.*\"\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's do the following preprocessings\n",
    "\n",
    "- ### \"marital\" column: imputation -> one-hot encoding\n",
    "- ### \"duration\" column: log tranformation \n",
    "- ### \"day\" & \"campaign\" column: imputation -> standardization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A [column transformer](https://scikit-learn.org/stable/modules/generated/sklearn.compose.make_column_transformer.html#sklearn.compose.make_column_transformer) allows different columns to be transformed separately in parallel. \n",
    "\n",
    "<img src=\"https://cdn.pixabay.com/photo/2020/05/08/16/37/pipes-5146458_1280.jpg\" alt=\"pipeline\" style=\"width: 600px;\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.compose import make_column_transformer\n",
    "\n",
    "imp_ohe = make_pipeline(SimpleImputer(strategy='most_frequent'),\n",
    "                        OneHotEncoder(sparse=False)\n",
    "                       )\n",
    "\n",
    "imp_std = make_pipeline(SimpleImputer(), \n",
    "                        StandardScaler()\n",
    "                       )\n",
    "\n",
    "preprocessor = make_column_transformer((imp_ohe, ['marital']),\n",
    "                                       (FunctionTransformer(np.log1p), ['duration']),\n",
    "                                       (imp_std, ['day', 'campaign']), \n",
    "                                        remainder='passthrough')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A [column selector](https://scikit-learn.org/stable/modules/generated/sklearn.compose.make_column_selector.html) allows selecting columns by data type or name pattern. \n",
    "\n",
    "### It can be useful when the data contains many features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import make_column_selector\n",
    "\n",
    "preprocessor = make_column_transformer((imp_ohe, make_column_selector(dtype_include=object)),\n",
    "                                       (FunctionTransformer(np.log1p), ['duration']),\n",
    "                                       (imp_std, ['day', 'campaign']), \n",
    "                                        remainder='passthrough')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.set_printoptions(edgeitems=10, suppress=True)\n",
    "preprocessor.fit_transform(X_train).round(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = make_pipeline(preprocessor, clf)\n",
    "\n",
    "pipe.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize a pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import set_config\n",
    "set_config(display='diagram')\n",
    "\n",
    "pipe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use the pipeline to make predictions for out-of-sample data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_new = pd.DataFrame(data=[['single', 20, 16, 5, 4], \n",
    "                           ['married', 7, 352, 2, 0]],\n",
    "                     columns=X.columns, \n",
    "                     index=['Tom', 'Jerry'])\n",
    "\n",
    "X_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe.predict(X_new)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cross-validation with a pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cross_val_score(pipe, X, y, cv=5, scoring='roc_auc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now let's try another classification model [k-nearest neighbors](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "clf = KNeighborsClassifier(n_neighbors=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = make_pipeline(preprocessor, clf) # we reuse the same preprocessor from earlier\n",
    "\n",
    "pipe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cross_val_score(pipe, X, y, cv=5, scoring='roc_auc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameter tuning using [grid search cross-validation](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_grid = {\n",
    "    'columntransformer__pipeline-2__standardscaler': [StandardScaler(), MinMaxScaler(), 'passthrough'],\n",
    "    'columntransformer__pipeline-2__simpleimputer__strategy': ['mean', 'median'],\n",
    "    'kneighborsclassifier__n_neighbors': [3, 5, 7], \n",
    "    'kneighborsclassifier__metric': ['euclidean', 'manhattan'], \n",
    "    'kneighborsclassifier__weights': ['uniform', 'distance']\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gs = GridSearchCV(pipe, param_grid=param_grid, cv=5, scoring='roc_auc', n_jobs=-1)\n",
    "\n",
    "gs.fit(X_train, y_train)\n",
    "\n",
    "gs.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In the above example, we set up \n",
    "\n",
    "- ### 3 options for the standardization\n",
    "- ### 2 options for the imputation\n",
    "- ### 3 options for the number of neighbors in knn\n",
    "- ### 2 options for the distance metric in knn\n",
    "- ### 2 options for the weights in knn\n",
    "\n",
    "### That is a total of $3\\times2\\times3\\times2\\times2=72$ model configurations.\n",
    "\n",
    "### With a 5-fold cross-validation for each, we did a total of $72\\times5=360$ model fittings to the data.\n",
    "\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bonus: a summary of scikit-learn's uniform APIs \n",
    "\n",
    "- ### scikit-learn's main API is implemented around so-called estimators.\n",
    "\n",
    "- ### An estimator is any object that learns from data (e.g., a regression or classification model, or a transformer such as a scaler).\n",
    "\n",
    "<br>\n",
    "\n",
    "- ### Fit estimator to data: `estimator.fit(X, [y])`\n",
    "- ### Transform data using fitted estimator: `estimator.transform(X)`\n",
    "    - ### e.g., preprocessing, dimentionality reduction\n",
    "- ### Predict using fitted estimator: `estimator.predict(X)`\n",
    "    - ### e.g., regression, classification, clustering\n",
    "\n",
    "### Resources from scikit-learn official website\n",
    "\n",
    "- ### [Examples](https://scikit-learn.org/stable/auto_examples/index.html)\n",
    "\n",
    "- ### [Tutorials](https://scikit-learn.org/stable/tutorial/index.html)\n",
    "\n",
    "- ### [User guide](https://scikit-learn.org/stable/user_guide.html)\n",
    "\n",
    "- ### [scikit-learn MOOC course](https://inria.github.io/scikit-learn-mooc/)\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "midas_python_worshop_2020.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}