notebooks/parameter_tuning_sol_02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# \ud83d\udcc3 Solution for Exercise M3.01\n",
    "\n",
    "The goal is to write an exhaustive search to find the best parameters\n",
    "combination maximizing the model generalization performance.\n",
    "\n",
    "Here we use a small subset of the Adult Census dataset to make the code\n",
    "faster to execute. Once your code works on the small subset, try to\n",
    "change `train_size` to a larger value (e.g. 0.8 for 80% instead of\n",
    "20%)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "adult_census = pd.read_csv(\"../datasets/adult-census.csv\")\n",
    "\n",
    "target_name = \"class\"\n",
    "target = adult_census[target_name]\n",
    "data = adult_census.drop(columns=[target_name, \"education-num\"])\n",
    "\n",
    "data_train, data_test, target_train, target_test = train_test_split(\n",
    "    data, target, train_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.compose import make_column_selector as selector\n",
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "\n",
    "categorical_preprocessor = OrdinalEncoder(handle_unknown=\"use_encoded_value\",\n",
    "                                          unknown_value=-1)\n",
    "preprocessor = ColumnTransformer(\n",
    "    [('cat_preprocessor', categorical_preprocessor,\n",
    "      selector(dtype_include=object))],\n",
    "    remainder='passthrough', sparse_threshold=0)\n",
    "\n",
    "from sklearn.ensemble import HistGradientBoostingClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "model = Pipeline([\n",
    "    (\"preprocessor\", preprocessor),\n",
    "    (\"classifier\", HistGradientBoostingClassifier(random_state=42))\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Use the previously defined model (called `model`) and using two nested `for`\n",
    "loops, make a search of the best combinations of the `learning_rate` and\n",
    "`max_leaf_nodes` parameters. In this regard, you will need to train and test\n",
    "the model by setting the parameters. The evaluation of the model should be\n",
    "performed using `cross_val_score` on the training set. We will use the\n",
    "following parameters search:\n",
    "- `learning_rate` for the values 0.01, 0.1, 1 and 10. This parameter controls\n",
    "  the ability of a new tree to correct the error of the previous sequence of\n",
    "  trees\n",
    "- `max_leaf_nodes` for the values 3, 10, 30. This parameter controls the\n",
    "  depth of each tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# solution\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "learning_rate = [0.01, 0.1, 1, 10]\n",
    "max_leaf_nodes = [3, 10, 30]\n",
    "\n",
    "best_score = 0\n",
    "best_params = {}\n",
    "for lr in learning_rate:\n",
    "    for mln in max_leaf_nodes:\n",
    "        print(f\"Evaluating model with learning rate {lr:.3f}\"\n",
    "              f\" and max leaf nodes {mln}... \", end=\"\")\n",
    "        model.set_params(\n",
    "            classifier__learning_rate=lr,\n",
    "            classifier__max_leaf_nodes=mln\n",
    "        )\n",
    "        scores = cross_val_score(model, data_train, target_train, cv=2)\n",
    "        mean_score = scores.mean()\n",
    "        print(f\"score: {mean_score:.3f}\")\n",
    "        if mean_score > best_score:\n",
    "            best_score = mean_score\n",
    "            best_params = {'learning-rate': lr, 'max leaf nodes': mln}\n",
    "            print(f\"Found new best model with score {best_score:.3f}!\")\n",
    "\n",
    "print(f\"The best accuracy obtained is {best_score:.3f}\")\n",
    "print(f\"The best parameters found are:\\n {best_params}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Now use the test set to score the model using the best parameters\n",
    "that we found using cross-validation in the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# solution\n",
    "best_lr = best_params['learning-rate']\n",
    "best_mln = best_params['max leaf nodes']\n",
    "\n",
    "model.set_params(classifier__learning_rate=best_lr,\n",
    "                 classifier__max_leaf_nodes=best_mln)\n",
    "model.fit(data_train, target_train)\n",
    "test_score = model.score(data_test, target_test)\n",
    "\n",
    "print(f\"Test score after the parameter tuning: {test_score:.3f}\")"
   ]
  }
 ],
 "metadata": {
  "jupytext": {
   "main_language": "python"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}