notebooks/20-10-12-getting-started-sklearn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting started with scikit-learn\n",
    "\n",
    "\n",
    "## Loading packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install scikit-learn==0.23.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## !!! Restart kernel to use above version of scikit-learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# general\n",
    "import math\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# plotting\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "plt.style.use('seaborn-whitegrid')\n",
    "%matplotlib inline\n",
    "\n",
    "# scipy\n",
    "from scipy import stats\n",
    "\n",
    "# scikit-learn\n",
    "from sklearn import datasets\n",
    "from sklearn import linear_model\n",
    "from sklearn.metrics import mean_squared_error, r2_score\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fitting a model without scikit-learn\n",
    "\n",
    "An example with [generated regression data](https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_regression.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create some data and create a scatter plot\n",
    "X, y, coef = datasets.make_regression(n_samples=100, n_features=1, noise=50, random_state=0, coef=True)\n",
    "\n",
    "plt.plot(X, y, '.', color='grey');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fit a line through the points:\n",
    "\n",
    "$y = a + b*x$\n",
    "\n",
    "where $a$ the intercept (the value of $y$ where $x=0$) and $b$ is the gradient of the line. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is actually a straightforward problem that can be solved directly with linear algebra. From this [example](https://machinelearningmastery.com/solve-linear-regression-using-linear-algebra/):\n",
    "\n",
    "$X^T . X . b = X^T . y$\n",
    "\n",
    "$b = (X^T . X)^-1 . X^T . y$\n",
    "\n",
    "Note that it is assumed here that $a=0$\n",
    "\n",
    "*[This course](https://www.coursera.org/learn/machine-learning) by Andrew Ng goes into much more detail.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "b = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yhat = X.dot(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(X, y, '.', color='grey');\n",
    "plt.plot(X, yhat, color='red')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to solve this is to use the mean difference between the line and all points using the [least squares method](https://en.wikipedia.org/wiki/Least_squares), where you want to minimize (find the smallest possible value) for the root means square error:\n",
    "\n",
    "$RMSE = \\sqrt{ \\frac{1}{N}\\sum_{i=1}^{N} (\\hat{y}_{i} - y_{i})^2}$\n",
    "\n",
    "where $y_{i}$ is the observed value and $\\hat{y}_{i}$ the simulated value for each point $i$ and $N$ the number of data points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "RMSE = np.sqrt(sum(np.power(yhat-y,2))/len(y))\n",
    "RMSE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fitting a model with [SciPy](https://docs.scipy.org/doc/scipy/reference/)\n",
    "\n",
    "This package contains various [optimization algorithms](https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html), including the [least squares method](https://docs.scipy.org/doc/scipy/reference/tutorial/optimize.html#least-squares-minimization-least-squares) that can be used to fit the above line:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "slope, intercept, r_value, p_value, std_err = stats.linregress(X[:,0], y)\n",
    "print(\"slope: %f    intercept: %f\" % (slope, intercept))\n",
    "print(\"r_squared: %f    std_err: %f\" % (r_value**2, std_err))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`r_squared` is the coefficient of determination\n",
    "\n",
    "`std_err` is the [standard error](https://en.wikipedia.org/wiki/Standard_error)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(X, y, '.', color='grey');\n",
    "plt.plot(X, yhat, color='red')\n",
    "plt.plot(X, intercept + slope*X, 'blue')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fitting a model with Scikit-learn\n",
    "\n",
    "When the data becomes more complex, e.g. many more variables, non-linear, including binary or multiple classes fitting a 'line' through the data is less straightforward. \n",
    "\n",
    "The principle stays the same: find the best 'line' through your data points. This 'line' is the machine learning model.  \n",
    "\n",
    "With scikit-learn:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linreg = linear_model.LinearRegression()\n",
    "\n",
    "linreg.fit(X,y)\n",
    "\n",
    "ypred= linreg.predict(X)\n",
    "\n",
    "print(linreg)\n",
    "print('Intercept: \\n',linreg.intercept_)\n",
    "print('Coefficients: \\n', linreg.coef_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(X, y, '.', color='grey');\n",
    "plt.plot(X, yhat, color='red')\n",
    "plt.plot(X, intercept + slope*X, 'blue')\n",
    "plt.plot(X, ypred, color='green')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above a line was fitted in 3 different ways, with a few metrics calculated. This is only one part of the process when building and testing a model. A complete workflow consists of these steps:\n",
    "\n",
    "* Data exploration\n",
    "* Data preprocessing\n",
    "* Splitting data for training and testing\n",
    "* Preparing a model\n",
    "* Assembling all of the steps using pipeline\n",
    "* Training the model\n",
    "* Running predictions on the model\n",
    "* Evaluating and visualizing model performance\n",
    "\n",
    "Ideally you would go through each of these steps one by one, but in practice you will go back and forth between them when you start to understand the data better.\n",
    "\n",
    "# Full example\n",
    "\n",
    "## [Diabetes dataset](https://scikit-learn.org/stable/datasets/index.html#diabetes-dataset)\n",
    "\n",
    "An example with data provided with scikit-learn:\n",
    "\n",
    "* Ten baseline variables: age, sex, body mass index, average blood pressure, and six blood serum measurements were obtained for each of n = 442 diabetes patients, as well as the response of interest, a quantitative measure of disease progression one year after baseline\n",
    "* Each of these 10 feature variables have been mean centered and scaled by the standard deviation times n_samples (i.e. the sum of squares of each column totals 1)\n",
    "* The target is a quantitative measure of disease progression one year after baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# as_frame=True loads the data as a pandas DataFrame (X) and Series (y)\n",
    "diabetes = datasets.load_diabetes(as_frame=True)\n",
    "\n",
    "X = diabetes.data\n",
    "y = diabetes.target\n",
    "\n",
    "feature_names = diabetes.feature_names\n",
    "print(feature_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.to_csv('diabetes.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.insert(0, \"target\", y)\n",
    "sns.pairplot(X[['target','age', 'sex', 'bmi', 'bp']], kind='reg', diag_kind='kde');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.pairplot(X[['target','s1', 's2', 's3', 's4', 's5', 's6']], kind='reg', diag_kind='kde');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X.corr().style.background_gradient(cmap='coolwarm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3 groups of data:\n",
    "\n",
    "* `age`, `bmi` and `bp` - continuous\n",
    "* `sex` - 2 classes (0 or 1)\n",
    "* `s1`,`s2`,`s3`,`s4`,`s5` and `s6` - seem strongly correlated, continuous\n",
    "* all are normalized, so this step can be skipped"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use one features\n",
    "X1 = X[['age']]\n",
    "\n",
    "X1_train, X1_test, y1_train, y1_test = train_test_split(X1, y, test_size=0.25, random_state=42)\n",
    "\n",
    "# Create linear regression object\n",
    "regr1 = linear_model.LinearRegression()\n",
    "\n",
    "# Train the model using the training data\n",
    "regr1.fit(X1_train, y1_train)\n",
    "\n",
    "# Make predictions using the test data\n",
    "y1_pred = regr1.predict(X1_test)\n",
    "\n",
    "print('Coefficients: \\n', regr1.coef_)\n",
    "print('RMSE: %.2f' % np.sqrt(mean_squared_error(y1_test, y1_pred)))\n",
    "print('Coefficient of determination (r2): %.2f' % r2_score(y1_test, y1_pred))\n",
    "\n",
    "plt.plot(X1_train, y1_train, '.', color='grey');\n",
    "plt.plot(X1_test, y1_pred, '.', color='red');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use three features\n",
    "X2 = X[['age', 'bmi', 'bp']]\n",
    "\n",
    "# Split the data into training/testing sets\n",
    "X2_train, X2_test, y2_train, y2_test = train_test_split(X2, y, test_size=0.25, random_state=42)\n",
    "\n",
    "# Create linear regression object\n",
    "regr2 = linear_model.LinearRegression()\n",
    "\n",
    "# Train the model using the training data\n",
    "regr2.fit(X2_train, y2_train)\n",
    "\n",
    "# Make predictions using the test data\n",
    "y2_pred = regr2.predict(X2_test)\n",
    "\n",
    "print('Coefficients: \\n', regr2.coef_)\n",
    "print('RMSE: %.2f' % np.sqrt(mean_squared_error(y2_test, y2_pred)))\n",
    "print('Coefficient of determination: %.2f' % r2_score(y2_test, y2_pred))\n",
    "\n",
    "plt.plot(X[['age']], y, '.', color='grey');\n",
    "plt.plot(X2_test[['age']], y2_pred, '.', color='red');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot(y2_test, y2_pred, '.', color='red');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the blood measurements\n",
    "X3 = X[['s1', 's2', 's3', 's4', 's5', 's6']]\n",
    "\n",
    "X3_train, X3_test, y3_train, y3_test = train_test_split(X3, y, test_size=0.25, random_state=42)\n",
    "\n",
    "# Create linear regression object\n",
    "regr3 = linear_model.LinearRegression()\n",
    "\n",
    "# Train the model using the training data\n",
    "regr3.fit(X3_train, y3_train)\n",
    "\n",
    "# Make predictions using the test data\n",
    "y3_pred = regr3.predict(X3_test)\n",
    "\n",
    "print('Coefficients: \\n', regr3.coef_)\n",
    "print('Mean squared error: %.2f' % mean_squared_error(y3_test, y3_pred))\n",
    "print('Coefficient of determination: %.2f' % r2_score(y3_test, y3_pred))\n",
    "\n",
    "plt.plot(y3_test, y3_pred, '.', color='grey');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use all data\n",
    "X4 = X[['age', 'bmi', 'sex', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']]\n",
    "\n",
    "X4_train, X4_test, y4_train, y4_test = train_test_split(X4, y, test_size=0.25, random_state=42)\n",
    "\n",
    "# Create linear regression object\n",
    "regr4 = linear_model.LinearRegression()\n",
    "\n",
    "# Train the model using the training data\n",
    "regr4.fit(X4_train, y4_train)\n",
    "\n",
    "# Make predictions using the test data\n",
    "y4_pred = regr4.predict(X4_test)\n",
    "\n",
    "print('Coefficients: \\n', regr4.coef_)\n",
    "print('RMSE: %.2f' % np.sqrt(mean_squared_error(y4_test, y4_pred)))\n",
    "print('Coefficient of determination (r2): %.2f' % r2_score(y4_test, y4_pred))\n",
    "\n",
    "plt.plot(y4_test, y4_pred, '.', color='grey');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot outputs\n",
    "plt.scatter(X4_test['bmi'], y4_test, color='grey')\n",
    "plt.scatter(X4_test['bmi'], y4_pred, color='red')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# what we have so far\n",
    "\n",
    "models = ['LinearRegression', 'LinearRegression', 'LinearRegression', 'LinearRegression']\n",
    "r2s = [r2_score(y1_test, y1_pred),r2_score(y2_test, y2_pred),r2_score(y3_test, y3_pred),r2_score(y4_test, y4_pred)]\n",
    "RMSEs = [np.sqrt(mean_squared_error(y1_test, y1_pred)),np.sqrt(mean_squared_error(y2_test, y2_pred)),\n",
    "        np.sqrt(mean_squared_error(y3_test, y3_pred)),np.sqrt(mean_squared_error(y4_test, y4_pred))]\n",
    "\n",
    "age = [regr1.coef_[0], regr2.coef_[0], float('NaN'), regr4.coef_[0]]\n",
    "bmi = [float('NaN'), regr2.coef_[1], float('NaN'), regr4.coef_[1]]\n",
    "bp = [float('NaN'), regr2.coef_[2], float('NaN'), regr4.coef_[3]]\n",
    "sex = [float('NaN'), float('NaN'), float('NaN'), regr4.coef_[2]]\n",
    "s1 = [float('NaN'), float('NaN'), regr3.coef_[0], regr4.coef_[4]]\n",
    "s2 = [float('NaN'), float('NaN'), regr3.coef_[1], regr4.coef_[5]]\n",
    "s3 = [float('NaN'), float('NaN'), regr3.coef_[2], regr4.coef_[6]]\n",
    "s4 = [float('NaN'), float('NaN'), regr3.coef_[3], regr4.coef_[7]]\n",
    "s5 = [float('NaN'), float('NaN'), regr3.coef_[4], regr4.coef_[8]]\n",
    "s6 = [float('NaN'), float('NaN'), regr3.coef_[5], regr4.coef_[9]]\n",
    "\n",
    "summary = pd.DataFrame(list(zip(models, r2s, RMSEs, age, bmi, bp, sex, s1, s2, s3, s4 ,s5, s6)), \n",
    "               columns =['model', 'r2', 'RMSE','age','bmi','bp','sex', 's1', 's2', 's3', 's4', 's5', 's6']) \n",
    "\n",
    "summary.transpose()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [Ridge regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html#sklearn.linear_model.Ridge) or $L_2$ regularization\n",
    "\n",
    "* Ridge regression addresses some of the problems of Ordinary Least Squares by imposing a penalty on the size of the coefficients. \n",
    "* The complexity parameter $\\alpha$ controls the amount of shrinkage: the larger the value of $\\alpha$, the greater the amount of shrinkage and thus the coefficients become more robust to collinearity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n",
    "X_train = X_train.drop('target', axis=1)\n",
    "X_test = X_test.drop('target', axis=1)\n",
    "\n",
    "regr5 = linear_model.Ridge(alpha=0.2)\n",
    "#regr5 = linear_model.Ridge(alpha=0.2,solver='lsqr')\n",
    "regr5.fit(X_train, y_train)\n",
    "y5_pred = regr5.predict(X_test)\n",
    "\n",
    "summary.loc[len(summary)] = ['Ridge (alpha=0.2)',\n",
    "                     r2_score(y_test, y5_pred),\n",
    "                     np.sqrt(mean_squared_error(y_test, y5_pred)),\n",
    "                     regr5.coef_[0],regr5.coef_[1],regr5.coef_[2],regr5.coef_[3],regr5.coef_[4],\n",
    "                     regr5.coef_[5],regr5.coef_[6],regr5.coef_[7],regr5.coef_[8],regr5.coef_[9]]\n",
    "\n",
    "summary.transpose()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [Lasso](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html#sklearn.linear_model.Lasso) or $L_1$ regularization\n",
    "\n",
    "* The Lasso is a linear model that estimates sparse coefficients. It is useful in some contexts due to its tendency to prefer solutions with fewer non-zero coefficients, effectively reducing the number of features upon which the given solution is dependent. For this reason Lasso and its variants are fundamental to the field of compressed sensing\n",
    "\n",
    "> Have a look at [this example](https://scikit-learn.org/stable/auto_examples/linear_model/plot_lasso_model_selection.html#sphx-glr-auto-examples-linear-model-plot-lasso-model-selection-py) for a method to "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "regr6 = linear_model.Lasso(alpha=0.1)\n",
    "regr6.fit(X_train, y_train)\n",
    "y6_pred = regr6.predict(X_test)\n",
    "\n",
    "summary.loc[len(summary)] = ['Lasso (alpha=0.1)',\n",
    "                     r2_score(y_test, y6_pred),\n",
    "                     np.sqrt(mean_squared_error(y_test, y6_pred)),\n",
    "                     regr6.coef_[0],regr6.coef_[1],regr6.coef_[2],regr6.coef_[3],regr6.coef_[4],\n",
    "                     regr6.coef_[5],regr6.coef_[6],regr6.coef_[7],regr6.coef_[8],regr6.coef_[9]]\n",
    "\n",
    "summary.transpose()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Feature selection\n",
    "\n",
    "The Lasso regression can be used to perform [feature selection](https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html#sklearn.feature_selection.SelectFromModel). \n",
    "\n",
    "[Another example](https://scikit-learn.org/stable/auto_examples/feature_selection/plot_select_from_model_diabetes.html#sphx-glr-auto-examples-feature-selection-plot-select-from-model-diabetes-py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_selection import SelectFromModel\n",
    "\n",
    "model = SelectFromModel(regr6, prefit=True)\n",
    "print(X_train.shape)\n",
    "X_train_new = model.transform(X_train)\n",
    "X_test_new = model.transform(X_test)\n",
    "X_train_new.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.get_support()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(feature_names)\n",
    "new_feature_names = ['sex', 'bmi', 'bp', 's1', 's3', 's5', 's6']\n",
    "new_feature_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# retrain model\n",
    "regr7 = linear_model.Lasso(alpha=0.1)\n",
    "regr7.fit(X_train_new, y_train)\n",
    "y7_pred = regr7.predict(X_test_new)\n",
    "\n",
    "summary.loc[len(summary)] = ['Lasso (alpha=0.1, 7 feat.)',\n",
    "                     r2_score(y_test, y7_pred),\n",
    "                     np.sqrt(mean_squared_error(y_test, y7_pred)),\n",
    "                     float('NaN'),regr7.coef_[0],regr7.coef_[1],regr7.coef_[2],regr7.coef_[3],\n",
    "                     float('NaN'),regr7.coef_[4],float('NaN'),regr7.coef_[5],regr7.coef_[6]]\n",
    "\n",
    "summary.transpose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "regr8 = linear_model.Ridge(alpha=0.2)\n",
    "regr8.fit(X_train_new, y_train)\n",
    "y8_pred = regr8.predict(X_test_new)\n",
    "\n",
    "summary.loc[len(summary)] = ['Ridge (alpha=0.2, 7 feat.)',\n",
    "                     r2_score(y_test, y8_pred),\n",
    "                     np.sqrt(mean_squared_error(y_test, y8_pred)),\n",
    "                     float('NaN'),regr8.coef_[0],regr8.coef_[1],regr8.coef_[2],regr8.coef_[3],\n",
    "                     float('NaN'),regr8.coef_[4],float('NaN'),regr8.coef_[5],regr8.coef_[6]]\n",
    "\n",
    "summary.transpose()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hyperparameter optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.logspace(-3, 3, 40)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "alphas = np.logspace(-3, -1, 30)\n",
    "\n",
    "for Model in [linear_model.Ridge, linear_model.Lasso]:\n",
    "    scores = [cross_val_score(Model(alpha), X_train, y_train, cv=3).mean()\n",
    "              for alpha in alphas]\n",
    "    plt.plot(alphas, scores, label=Model.__name__) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "for Model in [linear_model.Ridge, linear_model.Lasso]:\n",
    "    gscv = GridSearchCV(Model(), dict(alpha=alphas), cv=3).fit(X_train, y_train)\n",
    "    print('%s: %s' % (Model.__name__, gscv.best_params_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import RidgeCV, LassoCV\n",
    "for Model in [RidgeCV, LassoCV]:\n",
    "    model = Model(alphas=alphas, cv=3).fit(X_train, y_train)\n",
    "    print('%s: %s' % (Model.__name__, model.alpha_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for Model in [RidgeCV, LassoCV]:\n",
    "    scores = cross_val_score(Model(alphas=alphas, cv=3), X_train, y_train, cv=3)\n",
    "    print(Model.__name__, np.mean(scores))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [DecisionTreeRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html?highlight=decisiontreeregressor#sklearn.tree.DecisionTreeRegressor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.tree import DecisionTreeRegressor\n",
    "\n",
    "tree = DecisionTreeRegressor(max_depth=3,random_state=42)\n",
    "tree.fit(X_train, y_train)\n",
    "tree_pred = tree.predict(X_test)\n",
    "\n",
    "print(tree.feature_importances_)\n",
    "\n",
    "summary.loc[len(summary)] = ['DecisionTree',\n",
    "                     r2_score(y_test, tree_pred),\n",
    "                     np.sqrt(mean_squared_error(y_test, tree_pred)),\n",
    "                     float('NaN'),float('NaN'),float('NaN'),float('NaN'),float('NaN'),\n",
    "                     float('NaN'),float('NaN'),float('NaN'),float('NaN'),float('NaN')]\n",
    "\n",
    "summary.round(3)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [AdaBoostRegressor](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.AdaBoostRegressor.html#sklearn.ensemble.AdaBoostRegressor)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [Polynomial features](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html#)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "poly = PolynomialFeatures(2)\n",
    "#poly = PolynomialFeatures(interaction_only=True)\n",
    "\n",
    "X_train_poly = poly.fit_transform(X_train)\n",
    "X_test_poly = poly.fit_transform(X_test)\n",
    "print(X_train_poly.shape)\n",
    "\n",
    "regr6 = linear_model.Lasso(alpha=0.1)\n",
    "regr6.fit(X_train_poly, y_train)\n",
    "y6_pred = regr6.predict(X_test_poly)\n",
    "\n",
    "\n",
    "print(r2_score(y_test, y6_pred))\n",
    "print(np.sqrt(mean_squared_error(y_test, y6_pred)))\n",
    "\n",
    "regr6.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SHAP\n",
    "\n",
    "[Example](https://shap.readthedocs.io/en/latest/example_notebooks/kernel_explainer/Diabetes%20regression.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shap\n",
    "shap.initjs()\n",
    "\n",
    "X_train_summary = shap.kmeans(X_train, 10)\n",
    "\n",
    "def print_accuracy(f):\n",
    "    print(\"Root mean squared test error = {0}\".format(np.sqrt(np.mean((f(X_test) - y_test)**2))))\n",
    "    time.sleep(0.5) # to let the print get out before any progress bars\n",
    "\n",
    "lin_regr = linear_model.LinearRegression()\n",
    "lin_regr.fit(X_train, y_train)\n",
    "\n",
    "ex = shap.KernelExplainer(lin_regr.predict, X_train_summary)\n",
    "shap_values = ex.shap_values(X_test.iloc[0,:])\n",
    "shap.force_plot(ex.expected_value, shap_values, X_test.iloc[0,:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shap_values = ex.shap_values(X_test.iloc[2,:])\n",
    "shap.force_plot(ex.expected_value, shap_values, X_test.iloc[2,:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shap_values = ex.shap_values(X_test)\n",
    "shap.summary_plot(shap_values, X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shap.dependence_plot(\"s1\", shap_values, X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learn more\n",
    "\n",
    "* [Learn regression algorithms using Python and scikit-learn](https://developer.ibm.com/tutorials/learn-regression-algorithms-using-python-and-scikit-learn/)\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}