part_2/2.1 Machine Learning Pipelines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Machine Learning Pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Models as a task graph (DAG)\n",
    "<img src=\"../part_1/img/rolson17-ml-pipeline.png\">\n",
    "<div style=\"text-align: right\">Source: R. Olson et. al. (2017) \"Data-driven Advice for Applying Machine Learning to Bioinformatics Problems.\"</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Why a DAG?\n",
    "\n",
    "### Encapsulation\n",
    "\n",
    "Feature engineering operators, post-processing (e.g. calibration) are part of your model.\n",
    "\n",
    "### Optimization\n",
    "\n",
    "Feature engineering operators have hyper-parameters too. Furthermore, we want to search over graphs.\n",
    "\n",
    "### Safety\n",
    "\n",
    "Ensure that no information from testing can \"leak\" into the model fitting phase.\n",
    "\n",
    "### Train-test-skew\n",
    "\n",
    "Contracts between operators are more subtle than just message schemas; assumptions about the data distribution are included as well. Operators must not change after the model was fit!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Pipelines in Scikit-learn\n",
    "\n",
    "### [Pipeline](https://scikit-learn.org/stable/modules/compose.html#pipeline)\n",
    "\n",
    "A linear sequence of `Estimator` objects that represent a fixed set of steps, often feature extraction, normalization and classification. It implements the [Composite design pattern (GoF)](https://en.wikipedia.org/wiki/Composite_pattern).\n",
    "\n",
    "All estimators in a pipeline, except the last one, must be transformers (i.e. must have a `transform` method).\n",
    "\n",
    "Pipelines allow nested parameters:\n",
    "```python\n",
    "\n",
    "est = Pipeline(steps=[('scaler', StandardScaler()), ('lr', LinearRegression())])\n",
    "est.set_params(scaler__with_std=False)\n",
    "```\n",
    "\n",
    "### Exercise 2.1.1\n",
    "\n",
    "Build a `Pipeline` version of the `PolynomialRegressor` and search for the optimal `degree` of the polynomial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "%run ../part_1/utils.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = generate_curve_data()\n",
    "fig = plot_data(X, y, fn=true_fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "# Excercise: change this estimator to polynomial regression\n",
    "est = LinearRegression()\n",
    "\n",
    "gs = GridSearchCV(estimator=est, param_grid={}, \n",
    "                  scoring='neg_mean_squared_error', cv=3)\n",
    "\n",
    "gs.fit(X[:, np.newaxis], y)\n",
    "\n",
    "fig = plot_data(X, y, fn=true_fn)\n",
    "plot_estimator(gs.best_estimator_, fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Beyond a chain of estimators\n",
    "\n",
    "\n",
    "### [FeatureUnion](https://scikit-learn.org/stable/modules/compose.html#featureunion-composite-feature-spaces)\n",
    "\n",
    "Combines several transformer objects into a new transformer that combines their output.\n",
    "Takes a list of transformer objects, each of which is fit to the data independently. \n",
    "The transformers are applied in parallel, and the feature matrices they output are concatenated side-by-side into a wider matrix (via `np.hstack`).\n",
    "\n",
    "```python\n",
    "feature_extractors = [('poly', PolynomialFeatures()), ('pca', PCA())]\n",
    "est = Pipeline(steps=[('fu_1', FeatureUnion(feature_extractors)), \n",
    "                      ('rfe', RFE(LinearRegression(), 5, step=1)), \n",
    "                      ('lr', LinearRegression())])\n",
    "```\n",
    "\n",
    "### Caching of transformers\n",
    "\n",
    "Pipelines can do [caching](https://scikit-learn.org/stable/modules/compose.html#caching-transformers-avoid-repeated-computation) via `memory` which helps to avoid repeated computation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Working with heterogenous data sources\n",
    "\n",
    "### [ColumnTransformer](https://scikit-learn.org/stable/modules/compose.html#columntransformer-for-heterogeneous-data)\n",
    "\n",
    "Many datasets contain features of different types, say text, numeric, categorical, and dates, where each type of feature requires separate preprocessing or feature extraction steps.\n",
    "\n",
    "This component is still experimental in 0.21.\n",
    "\n",
    "```python\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "column_trans = ColumnTransformer(\n",
    "    [('city_category', OneHotEncoder(dtype='int'),['city']),\n",
    "     ('title_bow', CountVectorizer(), 'title')],\n",
    "    remainder='drop')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Kaggle Time!\n",
    "\n",
    "### Dataset\n",
    "\n",
    "[Blue Book for Bulldozers (aka Fastiron) Kaggle competition](https://www.kaggle.com/c/bluebook-for-bulldozers/data). \n",
    "\n",
    "Predict the auction sale price for a piece of heavy equipment to create a \"blue book\" for bulldozers.\n",
    "\n",
    "Performance metric: mean absolute error (MAE)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "url = 'https://s3.amazonaws.com/datarobot_public_datasets/fastiron-train-sample_80.csv'\n",
    "df = pd.read_csv(url, parse_dates=['saledate'])\n",
    "df['saledate'] = df['saledate'].astype(np.int64)\n",
    "target_feature = 'SalePrice'\n",
    "categorical_features = ['UsageBand', 'fiBaseModel', 'fiModelDesc', \n",
    "                        'fiModelSeries', 'ProductSize',\n",
    "                        'state', 'ProductGroup', 'ProductGroupDesc','Enclosure',\n",
    "                        'Enclosure_Type', 'Tire_Size',\n",
    "                        ]\n",
    "numeric_features = ['MachineID', 'ModelID', 'datasource', 'YearMade', 'saledate']\n",
    "text_features = ['fiProductClassDesc']\n",
    "\n",
    "y = df.pop(target_feature)\n",
    "X = df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "y.plot.hist(50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "y_log1p = np.log1p(y)\n",
    "y_log1p.plot.hist(50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Example Pipeline\n",
    "\n",
    "<img src=\"img/pipeline-example.png\">\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import utils\n",
    "from utils import ColumnGroupSelector\n",
    "from sklearn.metrics import mean_absolute_error\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0, shuffle=True)\n",
    "\n",
    "# y should be an ndarray\n",
    "y_train = y_train.values\n",
    "y_test = y_test.values\n",
    "\n",
    "\n",
    "numeric_pipeline = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='mean')),\n",
    "    ('scaler', StandardScaler())])\n",
    "\n",
    "## Categorical pipeline\n",
    "# categorical_pipeline = Pipeline(steps=[...])\n",
    "\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        ('num', numeric_pipeline, numeric_features),\n",
    "        ## ('cat', categorical_pipeline, categorical_features),\n",
    "    ])\n",
    "\n",
    "# Append estimator to preprocessing pipeline.\n",
    "# Now we have a full prediction pipeline.\n",
    "est = Pipeline(steps=[('preprocessor', preprocessor),\n",
    "                      ('lr', LinearRegression())])\n",
    "\n",
    "est.fit(X_train, y_train)\n",
    "print(\"model score: %.3f\" % mean_absolute_error(y_test, est.predict(X_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Exercise 2.1.2\n",
    "\n",
    "Implement the following pipeline:\n",
    "\n",
    "<img src=\"img/exercise_2_1_2.png\">\n",
    "\n",
    "Tune hyper-parameters:\n",
    "  * `RandomForestRegressor(criterion='mse')` with `['mse', 'mae']`\n",
    "  * Set `RandomForestRegressor(n_estimators=10)` to avoid long runtime\n",
    "  * Set `SelectFromModel`s model to `ElasticNet(alpha=0.1)` and select `max_features=100` and `threshold=None`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import OrdinalEncoder\n",
    "from sklearn.feature_selection import SelectFromModel\n",
    "from sklearn.linear_model import ElasticNet\n",
    "\n",
    "\n",
    "numeric_pipeline = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='constant', fill_value=-99999)),\n",
    "    ])\n",
    "\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        ('num', numeric_pipeline, numeric_features),\n",
    "        ('cat', categorical_pipeline, categorical_features),\n",
    "    ])\n",
    "\n",
    "# Append estimator to preprocessing pipeline.\n",
    "# Now we have a full prediction pipeline.\n",
    "est = Pipeline(steps=[('preprocessor', preprocessor),\n",
    "                      ('rf', RandomForestRegressor(n_estimators=10, n_jobs=-1))])\n",
    "\n",
    "gs = GridSearchCV(estimator=est, param_grid={'rf__criterion': ['mse', 'mae']}, \n",
    "                  scoring='neg_mean_absolute_error', cv=3, refit=True, verbose=10)\n",
    "\n",
    "gs.fit(X_train, y_train)\n",
    "\n",
    "print(\"best params: {}\".format(gs.best_params_))\n",
    "print(\"model score: %.3f\" % mean_absolute_error(y_test, gs.best_estimator_.predict(X_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Exercise 2.1.3\n",
    "\n",
    "Add a sub-pipeline to process text:\n",
    "\n",
    "<img src=\"img/exercise_2_1_3.png\">\n",
    "\n",
    "Notes:\n",
    "  * `RandomForestRegressor` is not doing great with wide & sparse Bag-of-Words representations; you can use a sub-model to predict the `SalePrice` just from BoW features and feed this predictions into the `RandomForestRegressor`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "numeric_pipeline = Pipeline(steps=[\n",
    "    ('imputer', SimpleImputer(strategy='constant', fill_value=-99999)),\n",
    "    ])\n",
    "\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[\n",
    "        ('num', numeric_pipeline, numeric_features),\n",
    "        #('cat', categorical_pipeline, categorical_features),\n",
    "        #('text', text_pipeline, text_features[0])\n",
    "    ])\n",
    "\n",
    "# Append estimator to preprocessing pipeline.\n",
    "# Now we have a full prediction pipeline.\n",
    "est = Pipeline(steps=[('preprocessor', preprocessor),\n",
    "                      ('rf', RandomForestRegressor(n_estimators=10, criterion='mae', n_jobs=-1))])\n",
    "\n",
    "est.fit(X_train, y_train)\n",
    "\n",
    "print(\"model score: %.3f\" % mean_absolute_error(y_test, est.predict(X_test)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}