notebooks/03_categorical_pipeline_column_transformer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0ba63af5",
   "metadata": {},
   "source": [
    "# Using numerical and categorical variables together\n",
    "\n",
    "In the previous notebooks, we showed the required preprocessing to apply\n",
    "when dealing with numerical and categorical variables. However, we decoupled\n",
    "the process to treat each type individually. In this notebook, we will show\n",
    "how to combine these preprocessing steps.\n",
    "\n",
    "We will first load the entire adult census dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d53eff51",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "adult_census = pd.read_csv(\"../datasets/adult-census.csv\")\n",
    "# drop the duplicated column `\"education-num\"` as stated in the first notebook\n",
    "adult_census = adult_census.drop(columns=\"education-num\")\n",
    "\n",
    "target_name = \"class\"\n",
    "target = adult_census[target_name]\n",
    "\n",
    "data = adult_census.drop(columns=[target_name])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0b0f0b1",
   "metadata": {},
   "source": [
    "## Selection based on data types\n",
    "\n",
    "We will separate categorical and numerical variables using their data\n",
    "types to identify them, as we saw previously that `object` corresponds\n",
    "to categorical columns (strings). We make use of `make_column_selector`\n",
    "helper to select the corresponding columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5781b87d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import make_column_selector as selector\n",
    "\n",
    "numerical_columns_selector = selector(dtype_exclude=object)\n",
    "categorical_columns_selector = selector(dtype_include=object)\n",
    "\n",
    "numerical_columns = numerical_columns_selector(data)\n",
    "categorical_columns = categorical_columns_selector(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f70665d3",
   "metadata": {},
   "source": [
    "## Dispatch columns to a specific processor\n",
    "\n",
    "In the previous sections, we saw that we need to treat data differently\n",
    "depending on their nature (i.e. numerical or categorical).\n",
    "\n",
    "Scikit-learn provides a `ColumnTransformer` class which will send specific\n",
    "columns to a specific transformer, making it easy to fit a single predictive\n",
    "model on a dataset that combines both kinds of variables together\n",
    "(heterogeneously typed tabular data).\n",
    "\n",
    "We first define the columns depending on their data type:\n",
    "\n",
    "* **one-hot encoding** will be applied to categorical columns. Besides, we\n",
    "  use `handle_unknown=\"ignore\"` to solve the potential issues due to rare\n",
    "  categories.\n",
    "* **numerical scaling** numerical features which will be standardized.\n",
    "\n",
    "Now, we create our `ColumnTransfomer` by specifying three values:\n",
    "the preprocessor name, the transformer, and the columns.\n",
    "First, let's create the preprocessors for the numerical and categorical\n",
    "parts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9d12e88",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OneHotEncoder, StandardScaler\n",
    "\n",
    "categorical_preprocessor = OneHotEncoder(handle_unknown=\"ignore\")\n",
    "numerical_preprocessor = StandardScaler()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d15a1a4",
   "metadata": {},
   "source": [
    "Now, we create the transformer and associate each of these preprocessors\n",
    "with their respective columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e5e8018",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "\n",
    "preprocessor = ColumnTransformer([\n",
    "    ('one-hot-encoder', categorical_preprocessor, categorical_columns),\n",
    "    ('standard-scaler', numerical_preprocessor, numerical_columns)])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f446fa9",
   "metadata": {},
   "source": [
    "We can take a minute to represent graphically the structure of a\n",
    "`ColumnTransformer`:\n",
    "\n",
    "![columntransformer diagram](../figures/api_diagram-columntransformer.svg)\n",
    "\n",
    "A `ColumnTransformer` does the following:\n",
    "\n",
    "* It **splits the columns** of the original dataset based on the column names\n",
    "  or indices provided. We will obtain as many subsets as the number of\n",
    "  transformers passed into the `ColumnTransformer`.\n",
    "* It **transforms each subsets**. A specific transformer is applied to\n",
    "  each subset: it will internally call `fit_transform` or `transform`. The\n",
    "  output of this step is a set of transformed datasets.\n",
    "* It then **concatenate the transformed datasets** into a single dataset.\n",
    "\n",
    "The important thing is that `ColumnTransformer` is like any other\n",
    "scikit-learn transformer. In particular it can be combined with a classifier\n",
    "in a `Pipeline`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0e4afcc",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.pipeline import make_pipeline\n",
    "\n",
    "model = make_pipeline(preprocessor, LogisticRegression(max_iter=500))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bfa00406",
   "metadata": {},
   "source": [
    "We can display an interactive diagram with the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6e4f19e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import set_config\n",
    "set_config(display='diagram')\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57e79e1c",
   "metadata": {},
   "source": [
    "The final model is more complex than the previous models but still follows\n",
    "the same API (the same set of methods that can be called by the user):\n",
    "\n",
    "- the `fit` method is called to preprocess the data and then train the\n",
    "  classifier of the preprocessed data;\n",
    "- the `predict` method makes predictions on new data;\n",
    "- the `score` method is used to predict on the test data and compare the\n",
    "  predictions to the expected test labels to compute the accuracy.\n",
    "\n",
    "Let's start by splitting our data into train and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "601146e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data, target, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2e3cbc1",
   "metadata": {},
   "source": [
    "\n",
    "<div class=\"admonition caution alert alert-warning\">\n",
    "<p class=\"first admonition-title\" style=\"font-weight: bold;\">Caution!</p>\n",
    "<p class=\"last\">Be aware that we use <tt class=\"docutils literal\">train_test_split</tt> here for didactic purposes, to show\n",
    "the scikit-learn API.</p>\n",
    "</div>\n",
    "\n",
    "Now, we can train the model on the train set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "705e1040",
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = model.fit(data_train, target_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3cb1fa8",
   "metadata": {},
   "source": [
    "Then, we can send the raw dataset straight to the pipeline. Indeed, we do not\n",
    "need to make any manual preprocessing (calling the `transform` or\n",
    "`fit_transform` methods) as it will be handled when calling the `predict`\n",
    "method. As an example, we predict on the five first samples from the test\n",
    "set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c90b2e17",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa2ce7c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.predict(data_test)[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee5e5d13",
   "metadata": {},
   "outputs": [],
   "source": [
    "target_test[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8765ddd2",
   "metadata": {},
   "source": [
    "To get directly the accuracy score, we need to call the `score` method. Let's\n",
    "compute the accuracy score on the entire test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11639350",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.score(data_test, target_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2806b268",
   "metadata": {},
   "source": [
    "## Evaluation of the model with cross-validation\n",
    "\n",
    "As previously stated, a predictive model should be evaluated by\n",
    "cross-validation. Our model is usable with the cross-validation tools of\n",
    "scikit-learn as any other predictors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69cff10f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_validate\n",
    "\n",
    "cv_results = cross_validate(model, data, target, cv=5)\n",
    "cv_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d6edf8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = cv_results[\"test_score\"]\n",
    "print(\"The mean cross-validation accuracy is: \"\n",
    "      f\"{scores.mean():.3f} +/- {scores.std():.3f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb9f8b4c",
   "metadata": {},
   "source": [
    "The compound model has a higher predictive accuracy than the two models that\n",
    "used numerical and categorical variables in isolation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ff3dd9b",
   "metadata": {},
   "source": [
    "## Fitting a more powerful model\n",
    "\n",
    "**Linear models** are nice because they are usually cheap to train,\n",
    "**small** to deploy, **fast** to predict and give a **good baseline**.\n",
    "\n",
    "However, it is often useful to check whether more complex models such as an\n",
    "ensemble of decision trees can lead to higher predictive performance. In this\n",
    "section we will use such a model called **gradient-boosting trees** and\n",
    "evaluate its statistical performance. More precisely, the scikit-learn model\n",
    "we will use is called `HistGradientBoostingClassifier`. Note that boosting\n",
    "models will be covered in more details in a future module.\n",
    "\n",
    "For tree-based models, the handling of numerical and categorical variables is\n",
    "simpler than for linear models:\n",
    "* we do **not need to scale the numerical features**\n",
    "* using an **ordinal encoding for the categorical variables** is fine even if\n",
    "  the encoding results in an arbitrary ordering\n",
    "\n",
    "Therefore, for `HistGradientBoostingClassifier`, the preprocessing pipeline\n",
    "is slightly simpler than the one we saw earlier for the `LogisticRegression`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "586447f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.experimental import enable_hist_gradient_boosting\n",
    "from sklearn.ensemble import HistGradientBoostingClassifier\n",
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "\n",
    "categorical_preprocessor = OrdinalEncoder(handle_unknown=\"use_encoded_value\",\n",
    "                                          unknown_value=-1)\n",
    "\n",
    "preprocessor = ColumnTransformer([\n",
    "    ('categorical', categorical_preprocessor, categorical_columns)],\n",
    "    remainder=\"passthrough\")\n",
    "\n",
    "model = make_pipeline(preprocessor, HistGradientBoostingClassifier())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff87ec88",
   "metadata": {},
   "source": [
    "Now that we created our model, we can check its statistical performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2af3008",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "_ = model.fit(data_train, target_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8aee948b",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.score(data_test, target_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d01c6739",
   "metadata": {},
   "source": [
    "We can observe that we get significantly higher accuracies with the Gradient\n",
    "Boosting model. This is often what we observe whenever the dataset has a\n",
    "large number of samples and limited number of informative features (e.g. less\n",
    "than 1000) with a mix of numerical and categorical variables.\n",
    "\n",
    "This explains why Gradient Boosted Machines are very popular among\n",
    "datascience practitioners who work with tabular data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "273dcfb2",
   "metadata": {},
   "source": [
    "In this notebook we:\n",
    "\n",
    "* used a `ColumnTransformer` to apply different preprocessing for\n",
    "  categorical and numerical variables;\n",
    "* used a pipeline to chain the `ColumnTransformer` preprocessing and\n",
    "  logistic regresssion fitting;\n",
    "* seen that **gradient boosting methods** can outperform **linear\n",
    "  models**."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}