2. Бостон.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:05:06.359703",
     "start_time": "2017-07-19T12:05:06.349392"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Загрузим данные"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:05:14.966064",
     "start_time": "2017-07-19T12:05:14.461429"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_boston"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:05:15.778886",
     "start_time": "2017-07-19T12:05:15.761514"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "bunch = load_boston()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:05:16.965402",
     "start_time": "2017-07-19T12:05:16.960434"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Boston House Prices dataset\n",
      "===========================\n",
      "\n",
      "Notes\n",
      "------\n",
      "Data Set Characteristics:  \n",
      "\n",
      "    :Number of Instances: 506 \n",
      "\n",
      "    :Number of Attributes: 13 numeric/categorical predictive\n",
      "    \n",
      "    :Median Value (attribute 14) is usually the target\n",
      "\n",
      "    :Attribute Information (in order):\n",
      "        - CRIM     per capita crime rate by town\n",
      "        - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.\n",
      "        - INDUS    proportion of non-retail business acres per town\n",
      "        - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)\n",
      "        - NOX      nitric oxides concentration (parts per 10 million)\n",
      "        - RM       average number of rooms per dwelling\n",
      "        - AGE      proportion of owner-occupied units built prior to 1940\n",
      "        - DIS      weighted distances to five Boston employment centres\n",
      "        - RAD      index of accessibility to radial highways\n",
      "        - TAX      full-value property-tax rate per $10,000\n",
      "        - PTRATIO  pupil-teacher ratio by town\n",
      "        - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town\n",
      "        - LSTAT    % lower status of the population\n",
      "        - MEDV     Median value of owner-occupied homes in $1000's\n",
      "\n",
      "    :Missing Attribute Values: None\n",
      "\n",
      "    :Creator: Harrison, D. and Rubinfeld, D.L.\n",
      "\n",
      "This is a copy of UCI ML housing dataset.\n",
      "http://archive.ics.uci.edu/ml/datasets/Housing\n",
      "\n",
      "\n",
      "This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.\n",
      "\n",
      "The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic\n",
      "prices and the demand for clean air', J. Environ. Economics & Management,\n",
      "vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics\n",
      "...', Wiley, 1980.   N.B. Various transformations are used in the table on\n",
      "pages 244-261 of the latter.\n",
      "\n",
      "The Boston house-price data has been used in many machine learning papers that address regression\n",
      "problems.   \n",
      "     \n",
      "**References**\n",
      "\n",
      "   - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.\n",
      "   - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.\n",
      "   - many more! (see http://archive.ics.uci.edu/ml/datasets/Housing)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(bunch.DESCR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:05:39.336807",
     "start_time": "2017-07-19T12:05:39.330614"
    }
   },
   "outputs": [],
   "source": [
    "X, y = pd.DataFrame(data=bunch.data, columns=bunch.feature_names.astype(str)), bunch.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:05:41.249948",
     "start_time": "2017-07-19T12:05:41.221702"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CRIM</th>\n",
       "      <th>ZN</th>\n",
       "      <th>INDUS</th>\n",
       "      <th>CHAS</th>\n",
       "      <th>NOX</th>\n",
       "      <th>RM</th>\n",
       "      <th>AGE</th>\n",
       "      <th>DIS</th>\n",
       "      <th>RAD</th>\n",
       "      <th>TAX</th>\n",
       "      <th>PTRATIO</th>\n",
       "      <th>B</th>\n",
       "      <th>LSTAT</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.00632</td>\n",
       "      <td>18.0</td>\n",
       "      <td>2.31</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.538</td>\n",
       "      <td>6.575</td>\n",
       "      <td>65.2</td>\n",
       "      <td>4.0900</td>\n",
       "      <td>1.0</td>\n",
       "      <td>296.0</td>\n",
       "      <td>15.3</td>\n",
       "      <td>396.90</td>\n",
       "      <td>4.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.02731</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>6.421</td>\n",
       "      <td>78.9</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>396.90</td>\n",
       "      <td>9.14</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.02729</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.469</td>\n",
       "      <td>7.185</td>\n",
       "      <td>61.1</td>\n",
       "      <td>4.9671</td>\n",
       "      <td>2.0</td>\n",
       "      <td>242.0</td>\n",
       "      <td>17.8</td>\n",
       "      <td>392.83</td>\n",
       "      <td>4.03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.03237</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>6.998</td>\n",
       "      <td>45.8</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>394.63</td>\n",
       "      <td>2.94</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.06905</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.18</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.458</td>\n",
       "      <td>7.147</td>\n",
       "      <td>54.2</td>\n",
       "      <td>6.0622</td>\n",
       "      <td>3.0</td>\n",
       "      <td>222.0</td>\n",
       "      <td>18.7</td>\n",
       "      <td>396.90</td>\n",
       "      <td>5.33</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      CRIM    ZN  INDUS  CHAS    NOX     RM   AGE     DIS  RAD    TAX  \\\n",
       "0  0.00632  18.0   2.31   0.0  0.538  6.575  65.2  4.0900  1.0  296.0   \n",
       "1  0.02731   0.0   7.07   0.0  0.469  6.421  78.9  4.9671  2.0  242.0   \n",
       "2  0.02729   0.0   7.07   0.0  0.469  7.185  61.1  4.9671  2.0  242.0   \n",
       "3  0.03237   0.0   2.18   0.0  0.458  6.998  45.8  6.0622  3.0  222.0   \n",
       "4  0.06905   0.0   2.18   0.0  0.458  7.147  54.2  6.0622  3.0  222.0   \n",
       "\n",
       "   PTRATIO       B  LSTAT  \n",
       "0     15.3  396.90   4.98  \n",
       "1     17.8  396.90   9.14  \n",
       "2     17.8  392.83   4.03  \n",
       "3     18.7  394.63   2.94  \n",
       "4     18.7  396.90   5.33  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Зафиксируем генератор случайных чисел для воспроизводимости:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:05:54.565451",
     "start_time": "2017-07-19T12:05:54.562199"
    }
   },
   "outputs": [],
   "source": [
    "SEED = 22\n",
    "np.random.seed = SEED"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Домашка!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Разделим данные на условно обучающую и отложенную выборки:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:06:06.969114",
     "start_time": "2017-07-19T12:06:06.937099"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:06:07.153370",
     "start_time": "2017-07-19T12:06:07.147084"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=SEED)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:06:07.443579",
     "start_time": "2017-07-19T12:06:07.438465"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((404, 13), (404,), (102, 13), (102,))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape, y_train.shape, X_test.shape, y_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Измерять качество будем с помощью метрики среднеквадратичной ошибки:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2017-07-19T12:06:47.313356",
     "start_time": "2017-07-19T12:06:47.310497"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import mean_squared_error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"panel panel-info\" style=\"margin: 50px 0 0 0\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Задача 1.</h3> \n",
    "    </div>\n",
    "    <div class=\"panel\">\n",
    "        Обучите <b>LinearRegression</b> из пакета <b>sklearn.linear_model</b> на обучающей выборке (<i>X_train, y_train</i>) и измерьте качество на <i>X_test</i>.\n",
    "        <br>\n",
    "        <br>\n",
    "        <i>P.s. Ошибка должна быть в районе 20. </i>\n",
    "    </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Вышла средняя ошибка, равная 21.7029\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "clf = LinearRegression()\n",
    "clf.fit(X_train, y_train);\n",
    "\n",
    "print('Вышла средняя ошибка, равная %5.4f' % \\\n",
    "            (-np.mean(cross_val_score(clf, X_test, y_test, cv=5, scoring='neg_mean_squared_error'))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"panel panel-info\" style=\"margin: 50px 0 0 0\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Задача 2. (с подвохом)</h3> \n",
    "    </div>\n",
    "    <div class=\"panel\">\n",
    "        Обучите <b>SGDRegressor</b> из пакета <b>sklearn.linear_model</b> на обучающей выборке (<i>X_train, y_train</i>) и измерьте качество на <i>X_test</i>.\n",
    "    </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Вышла средняя ошибка, равная 0.3137\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\preprocessing\\data.py:586: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample.\n",
      "  warnings.warn(DEPRECATION_MSG_1D, DeprecationWarning)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\preprocessing\\data.py:649: DeprecationWarning: Passing 1d arrays as data is deprecated in 0.17 and will raise ValueError in 0.19. Reshape your data either using X.reshape(-1, 1) if your data has a single feature or X.reshape(1, -1) if it contains a single sample.\n",
      "  warnings.warn(DEPRECATION_MSG_1D, DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import SGDRegressor\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "ss = StandardScaler()\n",
    "X_scaled = ss.fit_transform(X_train)\n",
    "y_scaled = ss.fit_transform(y_train)\n",
    "\n",
    "sgd = SGDRegressor()\n",
    "sgd.fit(X_scaled, y_scaled);\n",
    "\n",
    "print('Вышла средняя ошибка, равная %5.4f' % \\\n",
    "            (-np.mean(cross_val_score(sgd, X_scaled, y_scaled, cv=5, scoring='neg_mean_squared_error'))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"panel panel-info\" style=\"margin: 50px 0 0 0\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Задача 3.</h3>\n",
    "    </div>\n",
    "    <div class=\"panel\">\n",
    "        Попробуйте все остальные классы:\n",
    "        <ul>\n",
    "            <li>Ridge\n",
    "            <li>Lasso\n",
    "            <li>ElasticNet\n",
    "        </ul>\n",
    "\n",
    "        <br>\n",
    "\n",
    "        В них, как вам уже известно, используются параметры регуляризации <b>alpha</b>. Настройте его как с помощью <b>GridSearchCV</b>, так и с помощью готовых <b>-CV</b> классов (<b>RidgeCV</b>, <b>LassoCV</b> и т.д.).\n",
    "        \n",
    "        <br><br>\n",
    "\n",
    "        Найдите уже, в конце-концов, самую точную линейную модель!\n",
    "\n",
    "    </div>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Вышла средняя ошибка, равная 21.3110\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler, PolynomialFeatures\n",
    "from sklearn.pipeline import Pipeline, make_pipeline\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.linear_model import RidgeCV\n",
    "\n",
    "############Ridge\n",
    "params = { \n",
    "    'alpha': [10**x for x in range(-2,3)]\n",
    "}\n",
    "\n",
    "from sklearn.linear_model import Ridge\n",
    "\n",
    "gsR = RidgeCV() #GridSearchCV(Ridge(), param_grid=params)\n",
    "gsR.fit(X_train, y_train);\n",
    "\n",
    "print('Вышла средняя ошибка, равная %5.4f' % \\\n",
    "            (-np.mean(cross_val_score(gsR, X_test, y_test, cv=5, scoring='neg_mean_squared_error'))))\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Вышла средняя ошибка, равная 21.2454\n"
     ]
    }
   ],
   "source": [
    "############Lasso\n",
    "from sklearn.linear_model import Lasso\n",
    "from sklearn.linear_model import LassoCV\n",
    "\n",
    "gsL = GridSearchCV(Lasso(), param_grid=params) #LassoCV() - медленнее\n",
    "gsL.fit(X_train, y_train);\n",
    "\n",
    "print('Вышла средняя ошибка, равная %5.4f' % \\\n",
    "            (-np.mean(cross_val_score(gsL, X_test, y_test, cv=5, scoring='neg_mean_squared_error'))))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Вышла средняя ошибка, равная 21.3403\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import ElasticNet\n",
    "from sklearn.linear_model import ElasticNetCV\n",
    "\n",
    "gsE = GridSearchCV(ElasticNet(), param_grid=params) #ElasticNetCV() - просто заменить, не слишком точен\n",
    "gsE.fit(X_train, y_train);\n",
    "\n",
    "print('Вышла средняя ошибка, равная %5.4f' % \\\n",
    "            (-np.mean(cross_val_score(gsE, X_test, y_test, cv=5, scoring='neg_mean_squared_error'))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Итого самый точный среди этих трёх - GridSearchCV + Lasso"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"panel panel-info\" style=\"margin: 50px 0 0 0\">\n",
    "    <div class=\"panel-heading\">\n",
    "        <h3 class=\"panel-title\">Задача 4.</h3>\n",
    "    </div>\n",
    "    <div class=\"panel\">\n",
    "        Проверять качество правильно на кросс-валидации, как известно. Вы знаете, что делать: подключаем <b>cross_val_score</b> из <b>sklearn.model_selection</b>. Параметр <b>cv</b> установите равным 5.\n",
    "        <br><br>\n",
    "        Вспомните про все штуки, которым мы с вами научились.\n",
    "        <br><br>\n",
    "        Добейтесь <b>MSE < 27</b>.\n",
    "    </div>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops! Все случаи уже были рассмотрены для cross_val_score"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "12px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}