9-EX-Lasso-eNet-v1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creative Commons CC BY 4.0 Lynd Bacon & Associates, Ltd. Not warranted to be suitable for any particular purpose. (You're on your own!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lasso, and elasticNet\n",
    "\n",
    "Here are two more supervised learning algorithms that use shrinkage for regularization.\n",
    "\n",
    "Lasso is an acronym for \"least absolute shrinkage and selection operator.\"  It uses L1 regularization: the penalty applied to the cost (loss) function minimized during training is an L1 norm:  a constant (alpha here) times the sum of the absolute values of the regression weights.\n",
    "\n",
    "elasticNet is a regularization method that combines the regularization penalties used in ridge regression (the L2 norm) and by the Lasso(L1 norm).\n",
    "\n",
    "An important difference between the Lasso and Ridge regression is that the latter can shrink coefficients, but it can never make them equal to zero short of `alpha` going to $\\infty$.  Lasso _can_ shrink coefficients to zero, an advantage when there are lots of regressors, and interpretation of their importance is of interest.  Lasso can be used for variable _selection_.\n",
    "\n",
    "In the following we'll again use the inpatient satisfaction data.  We'll apply the Lasso.  We'll rescale the predictor variables, use cross validation, and search for a \"good\" shrinkage parameter value.  Then we'll experiment a little with elasticNet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting Some Packages and the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn import linear_model\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "from sklearn.metrics import mean_squared_error, r2_score # Basic metrics\n",
    "from sklearn.model_selection import GridSearchCV, train_test_split\n",
    "from sklearn.pipeline import Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get that pesky satisfaction data, dummy code the patient categories, and create the numpy arrays we need.  \n",
    "\n",
    "Assuming that that data file is in the DATA subdir:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['patSat', 'q2', 'q3', 'q4', 'q5', 'q6', 'q7', 'q8', 'q9', 'ptCat1',\n",
       "       'ptCat2'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ptSatDF=pd.read_csv('DATA/ML/DECART-patSat.csv')\n",
    "patSatDF2=ptSatDF.copy()\n",
    "patSatDF2[['ptCat1','ptCat2']]=pd.get_dummies(patSatDF2.ptCat,drop_first=True)\n",
    "patSatDF2=patSatDF2.drop(['caseID','ptCat'],axis=1)\n",
    "patSatDF2.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setting Up for CV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1358, 10)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(453, 10)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(1358,)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(453, 10)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X=patSatDF2.iloc[:,1:].to_numpy()\n",
    "y=patSatDF2.iloc[:,0].to_numpy()\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=99)\n",
    "X_train.shape\n",
    "X_test.shape\n",
    "y_train.shape\n",
    "X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating a Pipeline, a Grid to Search, and doing CV Lasso\n",
    "\n",
    "Since we're going to do MinMax scaling on our \"features\" (predictors), we need to be sure that we do it separately for our training data and our test data.  \n",
    "\n",
    "To make doing this more convenient we're going to set up a \"pipeline\" of methods that will sequentially apply rescaling to the X data within CV folds, and separately for the training and test X data.\n",
    "\n",
    "Note that in the specification of the grid `param_grid`, the `alpha` parameter array is named `lasso__alpha`, \"lasso(double underscore)alpha\". The "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=20, error_score='raise-deprecating',\n",
       "             estimator=Pipeline(memory=None,\n",
       "                                steps=[('scaler',\n",
       "                                        MinMaxScaler(copy=True,\n",
       "                                                     feature_range=(0, 1))),\n",
       "                                       ('lasso',\n",
       "                                        Lasso(alpha=1.0, copy_X=True,\n",
       "                                              fit_intercept=True, max_iter=1000,\n",
       "                                              normalize=False, positive=False,\n",
       "                                              precompute=False, random_state=99,\n",
       "                                              selection='cyclic', tol=0.0001,\n",
       "                                              warm_start=False))],\n",
       "                                verbose=False),\n",
       "             iid='warn', n_jobs=None,\n",
       "             param_grid={'lasso__alpha': [0.001, 0.01, 0.1, 1, 10, 100]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=None, verbose=0)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best cross-validation accuracy: 0.6747\n",
      "Test set score: 0.7283\n",
      "Best parameters: {'lasso__alpha': 0.01}\n"
     ]
    }
   ],
   "source": [
    "lassoReg=linear_model.Lasso(random_state=99, normalize=False)\n",
    "pipe = Pipeline([(\"scaler\", MinMaxScaler()), (\"lasso\", lassoReg)])\n",
    "#\n",
    "param_grid={'lasso__alpha': [0.001, 0.01, 0.1, 1, 10, 100]} # alpha values\n",
    "#\n",
    "grid = GridSearchCV(pipe, param_grid=param_grid, cv=20)\n",
    "grid.fit(X_train, y_train)\n",
    "print(\"Best cross-validation accuracy: {:.4f}\".format(grid.best_score_))\n",
    "print(\"Test set score: {:.4f}\".format(grid.score(X_test, y_test)))\n",
    "print(\"Best parameters: {}\".format(grid.best_params_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time</th>\n",
       "      <th>std_fit_time</th>\n",
       "      <th>mean_score_time</th>\n",
       "      <th>std_score_time</th>\n",
       "      <th>param_lasso__alpha</th>\n",
       "      <th>params</th>\n",
       "      <th>split0_test_score</th>\n",
       "      <th>split1_test_score</th>\n",
       "      <th>split2_test_score</th>\n",
       "      <th>split3_test_score</th>\n",
       "      <th>...</th>\n",
       "      <th>split13_test_score</th>\n",
       "      <th>split14_test_score</th>\n",
       "      <th>split15_test_score</th>\n",
       "      <th>split16_test_score</th>\n",
       "      <th>split17_test_score</th>\n",
       "      <th>split18_test_score</th>\n",
       "      <th>split19_test_score</th>\n",
       "      <th>mean_test_score</th>\n",
       "      <th>std_test_score</th>\n",
       "      <th>rank_test_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.003758</td>\n",
       "      <td>0.003261</td>\n",
       "      <td>0.000705</td>\n",
       "      <td>0.000052</td>\n",
       "      <td>0.001</td>\n",
       "      <td>{'lasso__alpha': 0.001}</td>\n",
       "      <td>0.737480</td>\n",
       "      <td>0.549968</td>\n",
       "      <td>0.745594</td>\n",
       "      <td>0.667290</td>\n",
       "      <td>...</td>\n",
       "      <td>0.687571</td>\n",
       "      <td>0.787872</td>\n",
       "      <td>0.721716</td>\n",
       "      <td>0.622673</td>\n",
       "      <td>0.702962</td>\n",
       "      <td>0.667604</td>\n",
       "      <td>0.690136</td>\n",
       "      <td>0.674172</td>\n",
       "      <td>0.080751</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.001537</td>\n",
       "      <td>0.000020</td>\n",
       "      <td>0.000692</td>\n",
       "      <td>0.000013</td>\n",
       "      <td>0.01</td>\n",
       "      <td>{'lasso__alpha': 0.01}</td>\n",
       "      <td>0.736189</td>\n",
       "      <td>0.552228</td>\n",
       "      <td>0.740032</td>\n",
       "      <td>0.663569</td>\n",
       "      <td>...</td>\n",
       "      <td>0.690830</td>\n",
       "      <td>0.790482</td>\n",
       "      <td>0.720442</td>\n",
       "      <td>0.624000</td>\n",
       "      <td>0.699477</td>\n",
       "      <td>0.664623</td>\n",
       "      <td>0.689167</td>\n",
       "      <td>0.674723</td>\n",
       "      <td>0.077819</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.001493</td>\n",
       "      <td>0.000023</td>\n",
       "      <td>0.000690</td>\n",
       "      <td>0.000010</td>\n",
       "      <td>0.1</td>\n",
       "      <td>{'lasso__alpha': 0.1}</td>\n",
       "      <td>0.683930</td>\n",
       "      <td>0.554172</td>\n",
       "      <td>0.654748</td>\n",
       "      <td>0.606068</td>\n",
       "      <td>...</td>\n",
       "      <td>0.670422</td>\n",
       "      <td>0.745071</td>\n",
       "      <td>0.651251</td>\n",
       "      <td>0.596415</td>\n",
       "      <td>0.635112</td>\n",
       "      <td>0.619380</td>\n",
       "      <td>0.652360</td>\n",
       "      <td>0.639161</td>\n",
       "      <td>0.054744</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.001357</td>\n",
       "      <td>0.000014</td>\n",
       "      <td>0.000691</td>\n",
       "      <td>0.000007</td>\n",
       "      <td>1</td>\n",
       "      <td>{'lasso__alpha': 1}</td>\n",
       "      <td>-0.000001</td>\n",
       "      <td>-0.020160</td>\n",
       "      <td>-0.028430</td>\n",
       "      <td>-0.038922</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.024961</td>\n",
       "      <td>-0.013839</td>\n",
       "      <td>-0.198912</td>\n",
       "      <td>-0.017209</td>\n",
       "      <td>-0.058169</td>\n",
       "      <td>-0.030193</td>\n",
       "      <td>-0.021030</td>\n",
       "      <td>-0.030924</td>\n",
       "      <td>0.040904</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.001343</td>\n",
       "      <td>0.000017</td>\n",
       "      <td>0.000685</td>\n",
       "      <td>0.000012</td>\n",
       "      <td>10</td>\n",
       "      <td>{'lasso__alpha': 10}</td>\n",
       "      <td>-0.000001</td>\n",
       "      <td>-0.020160</td>\n",
       "      <td>-0.028430</td>\n",
       "      <td>-0.038922</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.024961</td>\n",
       "      <td>-0.013839</td>\n",
       "      <td>-0.198912</td>\n",
       "      <td>-0.017209</td>\n",
       "      <td>-0.058169</td>\n",
       "      <td>-0.030193</td>\n",
       "      <td>-0.021030</td>\n",
       "      <td>-0.030924</td>\n",
       "      <td>0.040904</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.001351</td>\n",
       "      <td>0.000016</td>\n",
       "      <td>0.000701</td>\n",
       "      <td>0.000025</td>\n",
       "      <td>100</td>\n",
       "      <td>{'lasso__alpha': 100}</td>\n",
       "      <td>-0.000001</td>\n",
       "      <td>-0.020160</td>\n",
       "      <td>-0.028430</td>\n",
       "      <td>-0.038922</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.024961</td>\n",
       "      <td>-0.013839</td>\n",
       "      <td>-0.198912</td>\n",
       "      <td>-0.017209</td>\n",
       "      <td>-0.058169</td>\n",
       "      <td>-0.030193</td>\n",
       "      <td>-0.021030</td>\n",
       "      <td>-0.030924</td>\n",
       "      <td>0.040904</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6 rows × 29 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   mean_fit_time  std_fit_time  mean_score_time  std_score_time  \\\n",
       "0       0.003758      0.003261         0.000705        0.000052   \n",
       "1       0.001537      0.000020         0.000692        0.000013   \n",
       "2       0.001493      0.000023         0.000690        0.000010   \n",
       "3       0.001357      0.000014         0.000691        0.000007   \n",
       "4       0.001343      0.000017         0.000685        0.000012   \n",
       "5       0.001351      0.000016         0.000701        0.000025   \n",
       "\n",
       "  param_lasso__alpha                   params  split0_test_score  \\\n",
       "0              0.001  {'lasso__alpha': 0.001}           0.737480   \n",
       "1               0.01   {'lasso__alpha': 0.01}           0.736189   \n",
       "2                0.1    {'lasso__alpha': 0.1}           0.683930   \n",
       "3                  1      {'lasso__alpha': 1}          -0.000001   \n",
       "4                 10     {'lasso__alpha': 10}          -0.000001   \n",
       "5                100    {'lasso__alpha': 100}          -0.000001   \n",
       "\n",
       "   split1_test_score  split2_test_score  split3_test_score  ...  \\\n",
       "0           0.549968           0.745594           0.667290  ...   \n",
       "1           0.552228           0.740032           0.663569  ...   \n",
       "2           0.554172           0.654748           0.606068  ...   \n",
       "3          -0.020160          -0.028430          -0.038922  ...   \n",
       "4          -0.020160          -0.028430          -0.038922  ...   \n",
       "5          -0.020160          -0.028430          -0.038922  ...   \n",
       "\n",
       "   split13_test_score  split14_test_score  split15_test_score  \\\n",
       "0            0.687571            0.787872            0.721716   \n",
       "1            0.690830            0.790482            0.720442   \n",
       "2            0.670422            0.745071            0.651251   \n",
       "3           -0.024961           -0.013839           -0.198912   \n",
       "4           -0.024961           -0.013839           -0.198912   \n",
       "5           -0.024961           -0.013839           -0.198912   \n",
       "\n",
       "   split16_test_score  split17_test_score  split18_test_score  \\\n",
       "0            0.622673            0.702962            0.667604   \n",
       "1            0.624000            0.699477            0.664623   \n",
       "2            0.596415            0.635112            0.619380   \n",
       "3           -0.017209           -0.058169           -0.030193   \n",
       "4           -0.017209           -0.058169           -0.030193   \n",
       "5           -0.017209           -0.058169           -0.030193   \n",
       "\n",
       "   split19_test_score  mean_test_score  std_test_score  rank_test_score  \n",
       "0            0.690136         0.674172        0.080751                2  \n",
       "1            0.689167         0.674723        0.077819                1  \n",
       "2            0.652360         0.639161        0.054744                3  \n",
       "3           -0.021030        -0.030924        0.040904                4  \n",
       "4           -0.021030        -0.030924        0.040904                4  \n",
       "5           -0.021030        -0.030924        0.040904                4  \n",
       "\n",
       "[6 rows x 29 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Summarize the cv results\n",
    "\n",
    "pd.DataFrame(grid.cv_results_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doesn't seem all that different from the CV regression model we trained earlier.  Let's take a look at what the \"best\" Lasso model is.  What's in the pipeline defined are _named steps_ that can be accessed.  There are two: \"scaler\", and \"lasso\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best estimator:\n",
      "Pipeline(memory=None,\n",
      "         steps=[('scaler', MinMaxScaler(copy=True, feature_range=(0, 1))),\n",
      "                ('lasso',\n",
      "                 Lasso(alpha=0.01, copy_X=True, fit_intercept=True,\n",
      "                       max_iter=1000, normalize=False, positive=False,\n",
      "                       precompute=False, random_state=99, selection='cyclic',\n",
      "                       tol=0.0001, warm_start=False))],\n",
      "         verbose=False)\n"
     ]
    }
   ],
   "source": [
    "print(\"Best estimator:\\n{}\".format(grid.best_estimator_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"lasso\" step is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lasso step:\n",
      "Lasso(alpha=0.01, copy_X=True, fit_intercept=True, max_iter=1000,\n",
      "      normalize=False, positive=False, precompute=False, random_state=99,\n",
      "      selection='cyclic', tol=0.0001, warm_start=False)\n"
     ]
    }
   ],
   "source": [
    "print(\"lasso step:\\n{}\". \\\n",
    "      format(      grid.best_estimator_.named_steps[\"lasso\"]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "And the Lasso coefficients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Lasso coefs:\n",
      "[1.01168204 0.8140796  0.41591071 3.43746095 0.9112827  0.1551897\n",
      " 0.         0.12520995 0.87604514 1.57825472]\n"
     ]
    }
   ],
   "source": [
    "print(\"Lasso coefs:\\n{}\". \\\n",
    "      format(      grid.best_estimator_.named_steps[\"lasso\"].coef_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is there a _really_ true zero coefficient? That \"zero\" is the coefficient for the variable `q7`.  _Looks_ like a zero:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid.best_estimator_.named_steps['lasso'].coef_[6]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's worth taking a look at the correlation between `patSat` and `q7`and the other vars:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "patSat    1.000000\n",
       "q2        0.649388\n",
       "q3        0.613785\n",
       "q4        0.652300\n",
       "q5        0.741797\n",
       "q6        0.589858\n",
       "q7        0.564058\n",
       "q8        0.443638\n",
       "q9        0.452323\n",
       "ptCat1    0.061737\n",
       "ptCat2    0.588030\n",
       "Name: patSat, dtype: float64"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "patSatDF2.corr().iloc[:,0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What Happened?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lasso and LARS\n",
    "\n",
    "LARS is an acronym for _least angles regression_, a method that does regulatization and feature selection.  It has strengths and weaknesses relative to other methods. The latter include difficulty dealing with highly colinear features.  See [LARS on Wikipedia](https://en.wikipedia.org/wiki/Least-angle_regression)\n",
    "\n",
    "There is a version of the Lasso that uses LARS, [\"LassoLARS\"](https://en.wikipedia.org/wiki/Lasso_(statistics)) which is implemented in scikit-learn as a `linear_model` function as [LassoLARS](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LassoLars.html#sklearn.linear_model.LassoLars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A UDU: Replicate the Above Lasso Application Using LassoLARS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# elasticNet\n",
    "\n",
    "AKA \"elastic net,\" has some advantages over the Lasso.  It can select more features than there are cases (rows of data), and it tends to select just one feature from sets of highly intercorrelated features.  It does this by using a cost function that combines the penalizations of both Lasso (L1) and of Ridge (L2) regression.  It can do a little better than the Lasso when features are (multi)collinear\n",
    "\n",
    "Let's try elasticNet using the `radon` data.  We'll again do 20 fold CV with MinMax scaling, like with the Lasso, above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=20, error_score='raise-deprecating',\n",
       "             estimator=Pipeline(memory=None,\n",
       "                                steps=[('scaler',\n",
       "                                        MinMaxScaler(copy=True,\n",
       "                                                     feature_range=(0, 1))),\n",
       "                                       ('eNet',\n",
       "                                        ElasticNet(alpha=1.0, copy_X=True,\n",
       "                                                   fit_intercept=True,\n",
       "                                                   l1_ratio=0.5, max_iter=1000,\n",
       "                                                   normalize=False,\n",
       "                                                   positive=False,\n",
       "                                                   precompute=False,\n",
       "                                                   random_state=99,\n",
       "                                                   selection='cyclic',\n",
       "                                                   tol=0.0001,\n",
       "                                                   warm_start=False))],\n",
       "                                verbose=False),\n",
       "             iid='warn', n_jobs=None,\n",
       "             param_grid={'eNet__alpha': [0.001, 0.01, 0.1, 1, 10, 100]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=None, verbose=0)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best cross-validation accuracy: 0.6751\n",
      "Test set score: 0.7306\n",
      "Best parameters: {'eNet__alpha': 0.01}\n"
     ]
    }
   ],
   "source": [
    "eNetReg=linear_model.ElasticNet(random_state=99, normalize=False)\n",
    "pipe = Pipeline([(\"scaler\", MinMaxScaler()), (\"eNet\", eNetReg)])\n",
    "#\n",
    "param_grid={'eNet__alpha': [0.001, 0.01, 0.1, 1, 10, 100]} # alpha values\n",
    "#\n",
    "grid = GridSearchCV(pipe, param_grid=param_grid, cv=20)\n",
    "grid.fit(X_train, y_train)\n",
    "print(\"Best cross-validation accuracy: {:.4f}\".format(grid.best_score_))\n",
    "print(\"Test set score: {:.4f}\".format(grid.score(X_test, y_test)))\n",
    "print(\"Best parameters: {}\".format(grid.best_params_))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ElasticNet coefs:\n",
      "[0.99156813 0.82309054 0.57352466 3.08506985 0.90074559 0.2880325\n",
      " 0.         0.20559254 0.89634168 1.59854033]\n"
     ]
    }
   ],
   "source": [
    "print(\"ElasticNet coefs:\\n{}\". \\\n",
    "      format(      grid.best_estimator_.named_steps[\"eNet\"].coef_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the first coefficient is for the intercerpt, which was included by default.\n",
    "\n",
    "By comparison, the Lasso coefficients from above, were:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Lasso coefs:\n",
    "[1.01168204 0.8140796  0.41591071 3.43746095 0.9112827  0.1551897\n",
    " 0.         0.12520995 0.87604514 1.57825472]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# UDU: Apply elasticNet to the pt Sat Data, Find a \"Best\" shrinkage parameter value"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "250px"
   },
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}