materials/compositions_practice.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Практическая часть семинара по композициям\n",
    "Данные: соревнование [Mercedes-Benz Greener Manufacturing](https://www.kaggle.com/c/mercedes-benz-greener-manufacturing)\n",
    "\n",
    "Данные загружаются сразу из гитхаба ниже.\n",
    "\n",
    "Кратко о задаче: объекты - автомобили, целевая переменная - время, за которое автомобиль проходит тест, задача регрессии, качество измеряется с помощью $R^2$-метрики.\n",
    "\n",
    "Скрипт загрузки и базовой предобработки данных взят с [Kaggle Kernels](https://www.kaggle.com/anokas/mercedes-eda-xgboost-starter-0-55/notebook). В этом же ядре приводится базовый exploration данных, можете почитать после семинара.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Загрузка данных"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_cell_guid": "45578147-cb22-cd2a-ccc7-1b81c82e137b",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "import gc\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "_cell_guid": "ba151bcc-e94e-3ff4-d213-4e80e7b4d956"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size of training set: 4209 rows and 378 columns\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ID</th>\n",
       "      <th>y</th>\n",
       "      <th>X0</th>\n",
       "      <th>X1</th>\n",
       "      <th>X2</th>\n",
       "      <th>X3</th>\n",
       "      <th>X4</th>\n",
       "      <th>X5</th>\n",
       "      <th>X6</th>\n",
       "      <th>X8</th>\n",
       "      <th>...</th>\n",
       "      <th>X375</th>\n",
       "      <th>X376</th>\n",
       "      <th>X377</th>\n",
       "      <th>X378</th>\n",
       "      <th>X379</th>\n",
       "      <th>X380</th>\n",
       "      <th>X382</th>\n",
       "      <th>X383</th>\n",
       "      <th>X384</th>\n",
       "      <th>X385</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>130.81</td>\n",
       "      <td>k</td>\n",
       "      <td>v</td>\n",
       "      <td>at</td>\n",
       "      <td>a</td>\n",
       "      <td>d</td>\n",
       "      <td>u</td>\n",
       "      <td>j</td>\n",
       "      <td>o</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6</td>\n",
       "      <td>88.53</td>\n",
       "      <td>k</td>\n",
       "      <td>t</td>\n",
       "      <td>av</td>\n",
       "      <td>e</td>\n",
       "      <td>d</td>\n",
       "      <td>y</td>\n",
       "      <td>l</td>\n",
       "      <td>o</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7</td>\n",
       "      <td>76.26</td>\n",
       "      <td>az</td>\n",
       "      <td>w</td>\n",
       "      <td>n</td>\n",
       "      <td>c</td>\n",
       "      <td>d</td>\n",
       "      <td>x</td>\n",
       "      <td>j</td>\n",
       "      <td>x</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>9</td>\n",
       "      <td>80.62</td>\n",
       "      <td>az</td>\n",
       "      <td>t</td>\n",
       "      <td>n</td>\n",
       "      <td>f</td>\n",
       "      <td>d</td>\n",
       "      <td>x</td>\n",
       "      <td>l</td>\n",
       "      <td>e</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>13</td>\n",
       "      <td>78.02</td>\n",
       "      <td>az</td>\n",
       "      <td>v</td>\n",
       "      <td>n</td>\n",
       "      <td>f</td>\n",
       "      <td>d</td>\n",
       "      <td>h</td>\n",
       "      <td>d</td>\n",
       "      <td>n</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 378 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   ID       y  X0 X1  X2 X3 X4 X5 X6 X8  ...   X375  X376  X377  X378  X379  \\\n",
       "0   0  130.81   k  v  at  a  d  u  j  o  ...      0     0     1     0     0   \n",
       "1   6   88.53   k  t  av  e  d  y  l  o  ...      1     0     0     0     0   \n",
       "2   7   76.26  az  w   n  c  d  x  j  x  ...      0     0     0     0     0   \n",
       "3   9   80.62  az  t   n  f  d  x  l  e  ...      0     0     0     0     0   \n",
       "4  13   78.02  az  v   n  f  d  h  d  n  ...      0     0     0     0     0   \n",
       "\n",
       "   X380  X382  X383  X384  X385  \n",
       "0     0     0     0     0     0  \n",
       "1     0     0     0     0     0  \n",
       "2     0     1     0     0     0  \n",
       "3     0     0     0     0     0  \n",
       "4     0     0     0     0     0  \n",
       "\n",
       "[5 rows x 378 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train = pd.read_csv('https://raw.githubusercontent.com/nadiinchi/hse_cs_ml_course_2017_FTAD/master/materials/train.csv')\n",
    "print('Size of training set: {} rows and {} columns'.format(*df_train.shape))\n",
    "df_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Распределение целевого признака:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "_cell_guid": "8aa66a00-591c-7cff-cc88-051bfcaa40c5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "min: 72.11 max: 265.32 mean: 100.66931812782134 std: 12.6778749695168\n",
      "Count of values above 180: 1\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4cAAAFKCAYAAACuBCrVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XucHXV9//HXJksIwU0aygLaIkipHynKLQHCJRARRUBL\npbUiRQV+YFQE6hUkXESjgJcAQRSNplEQWg2iNTZcFIkhCgiSShA/NHjtT6oRAwRDAgnbP2ZWTpfd\nzYHsmTPZfT0fjzyY+Z45M5+z350M73y/M6ejp6cHSZIkSdLINqrdBUiSJEmS2s9wKEmSJEkyHEqS\nJEmSDIeSJEmSJAyHkiRJkiQMh5IkSZIkDIeSpAYRsWNErI+IpeWfH0fEXRHx5oZtPtS4PsB+zo2I\nowZ47U/vj4ieiNj6Wda4d0RcUS5Pjoj5z+b9z0VEjI6Ib0TE/RHxzkHqmRYRyzbyWC+KiGs3Zh/P\n8nhzImLSEO3rloj4h6HYlySpep3tLkCSVDuPZ+YevSsRsQPwnYj4Y2Zem5nnNrGPQ4Cf9PdCk+8f\nzK7AX5b7uhOoIoz8BXAYsGVmrh+oniGyAxBDuL8NeSXw2QqPJ0mqqY6enp521yBJqomI2BFYlpnP\n69N+LHBaZk6JiHnlNp+IiPOB1wFPAA8BxwNHAxcBK4B3A0cBWwF/BSwAtm14fw/wOWBvitksZ2fm\ngog4HviHzHxNefzjKULg24ElwATga8AXgU9l5ksjYgJwObAH0AMsBM7KzHURsQa4kCIIvQC4NDMv\n6efzTwU+DowrP9PZ5fFupwhs9wB/n5kPlNtv308984DbgJcAY4GTM3NxRIwpfy4HA6OBu8uf6aMN\nxx8NJEUY/V5mHhYRZwF/V+5rS+C9mXldRHwQ2A94PvBj4K3AFcAU4GHKcJ6Zx0fEXwCfAl4IbAb8\na2Z+NCI+ArwP+Dnw5sy8vaGW7wOzMnN+uX4h0AF8CPgM8OKyX1cBx2ZmRsQt5XHupOH3qO/vVUT8\nP+AdZZ8/BLwzM3/atz8kSdVyWqkkqRn/CbyssaEMRv8M7J2Zk4EbgX0z83KKcPC+zLyu3HxcZu6a\nmWf0s++fZeZewHHAFyOie6AiMvPXwLnA4sw8oc/LsymCxsuAycDuwHvL1zYHfp+ZB1CEzAsjYmyf\nz/PnwHzg9MzcDXgLcBWwNXAE5YhqbzAcpJ6/BC4uR18/C3ywbD8TWAdMyszdgd9QBNbGz7ceOAl4\noAyGOwCHAgeXNc2gCGe9dgD2yszjgHMoZgS9pHzPng3bXQnMzcxJwD7AoRHxj5k5o6zjnxqDYWkO\nRdjvDa3HAZ8HDgcezswpmfli4IfAO2lSRBxM8bOdmpl7Ah+jCNaSpDYzHEqSmtEDrO7T9v8pQuOP\nIuITwNLM/PoA7791kH1fAZCZyyhGu/Z7jjUeTjGK2JOZa8v9Ht7w+jfK//6IIixu2ef9+wLLe0NS\nZt5LMSo47VnW8UBD0FoKbFMuv4ZiFPXuiFhKMRr4N4PtKDN/SRGk/qkcuXsb0Diqe1tmriuXjwC+\nkJlPlaORXwSIiC0pRis/XB73NooRxD0Y3FeA/SJiO4optcsz87/KkcR5EXFqRFxK8fN53iD76etI\nYGfg+2U9HwO2ioitnsU+JEktYDiUJDVjb4oplX+SmU9RhI7jKUbsLi7DQn8eG2TfjffwdQBPUoTR\njob2MU3U2PeaNopiCmWvxwEys/d+io5+tu9vn5v10z6YJxuWGz/HaIpRyT3KUcV92MD9khGxF/B9\nYDzFyOxFfepu/Lmu6/Na7891dNm+f8OxpwAfHezYmflH4KvAscAJFCOJRMTbgS9Q/GPB1cA1PPNn\nOVj/jQaubKhlL4qR3pWD1SNJaj3DoSRpUBHxYoopi5/s0747sAy4LzMvAC6mmMoJRVBpNlQdX+5v\nL+CvKe7vWwG8NCLGRkQn8NqG7Qfa9w3AKRHRERGbU9yDd1OTNUAxohYRsU9Zz67AQcAtG3hfs5/1\nBuCdETEmIkZRhK0LNrC/g4A7M3MWsIhitHH0APv/FnBCRIyKiHEUoa6nHEW8jeL+TyLizyhGRHuf\nJjtY/b1TS/cHep+gehgwLzO/QHF/5Gv7qelhYExE9I6Mvq7htRuBN0bE88v1twHfGeD4kqQKGQ4l\nSX1t0fBVFj+ieMDKBzLzW40bZeZ/Ukw9vDMi7gROBN5VvvxN4BMR8ZYmjrdTRNxNcT/bMZn5B4oA\nsQj4KbCY/ztq+QPgJRFxXZ/9nEYxhfOe8k8CH2nyM5OZvwdeD1wWEfdQjIqdkJn3b+CtA9XT14eB\nX1A8iOYnFCNr7+lnu3uB9RFxB8Wo3NYR8RPgLoqRwq0ioquf910ArKH47N8GfsfTU4GPBaaUn+t2\n4JrM/HL52teBf4uIV/XdYWbeRREer83MNWXzJ4Dp5ZTQ71BM0925z/seAd4PLIyIH1KMJPa+dgPF\nCOhNEfHjsrajG0Z0JUlt4tNKJUkaBiLiGODRzPyPcmTyWuDGzPxMm0uTJG0iHDmUJGl4WAbMKEf0\nllE8hfTz7S1JkrQpceRQkiRJkuTIoSRJkiTJcChJkiRJAjpbufOI2Be4KDOnRcQewGUU37u0Fnhz\nZv42Ik4GplM8DW1mZi6IiC2AqyieOrcKeEtmrmhlrZIkSZI0krXsnsOIeD/wJuCPmTklIhZRfPnv\n0oiYDgTwMYrvoJoMjAVuLZdPAcZn5gfLp6/tl5mnb+iYK1as2uRvoJw4cRwrV67e8IZqOfuiHuyH\nerAf6sO+qAf7oT7si3qwH+qju7ur47m+t5XTSh8Ajm5YPyYzl5bLnRTfxbQPsCQz15bfibQc2A04\nELi+3HYhcGgL66yVzs6BvttYVbMv6sF+qAf7oT7si3qwH+rDvqgH+2F4aFk4zMxrgScb1h8EiIj9\ngXcCFwPjgUca3rYKmNCnvbdNkiRJktQiLb3nsK+IeAMwAzgyM1dExKNAV8MmXcDDQGN7b9sGTZw4\nblj8q0V3d9eGN1Il7It6sB/qwX6oD/uiHuyH+rAv6sF+2PRVFg4j4jiKB89My8w/lM13AB+JiLHA\n5sAuFF/cuwQ4onz9cGBxM8cYDvOcu7u7WLFiVbvLEPZFXdgP9WA/1Id9UQ/2Q33YF/VgP9THxoT0\nSsJhRIwGZgO/Ar4WEQCLMvO8iJhNEf5GATMyc01EfAb4YkTcCjwBHFtFnZIkSZI0UrU0HGbmL4Ap\n5epWA2wzB5jTp2018PpW1iZJkiRJelorn1YqSZIkSdpEGA4lSZIkSYZDSZIkSZLhUJIkSZKE4VCS\nJEmShOFQkiRJkkRF33MoPRsnXnhzu0vo19wzD2l3CZIkSVLLOHIoSZIkSTIcSpIkSZIMh5IkSZIk\nDIeSJEmSJAyHkiRJkiQMh5IkSZIkDIeSJEmSJAyHkiRJkiQMh5IkSZIkDIeSJEmSJAyHkiRJkiQM\nh5IkSZIkDIeSJEmSJAyHkiRJkiQMh5IkSZIkDIeSJEmSJAyHkiRJkiQMh5IkSZIkDIeSJEmSJAyH\nkiRJkiQMh5IkSZIkoLPdBUibihMvvLndJfRr7pmHtLsESZIkDQOOHEqSJEmSDIeSJEmSJMOhJEmS\nJAnDoSRJkiQJw6EkSZIkCcOhJEmSJAnDoSRJkiQJv+dwRKvr9/ZJkiRJql5Lw2FE7AtclJnTImJn\nYB7QAywDTsnMpyLiZGA6sA6YmZkLImIL4CpgG2AV8JbMXNHKWiVJkiRpJGvZtNKIeD/weWBs2TQL\nODszpwIdwFERsR1wGnAAcBhwQURsDrwduKfc9kvA2a2qU5IkSZLU2nsOHwCOblifBCwqlxcChwL7\nAEsyc21mPgIsB3YDDgSu77OtJEmSJKlFWjatNDOvjYgdG5o6MrOnXF4FTADGA480bNNfe2/bBk2c\nOI7OztEbU3YtdHd3tbsEbUJGwu/LSPiMmwL7oT7si3qwH+rDvqgH+2HTV+UDaZ5qWO4CHgYeLZcH\na+9t26CVK1dvfJVt1t3dxYoVq9pdhjYhw/33xXOiHuyH+rAv6sF+qA/7oh7sh/rYmJBe5VdZ3B0R\n08rlw4HFwB3A1IgYGxETgF0oHlazBDiiz7aSJEmSpBapMhy+Bzg/In4AjAHmZ+b/ALMpwt/NwIzM\nXAN8Btg1Im4F3gqcX2GdkiRJkjTitHRaaWb+AphSLt8PHNzPNnOAOX3aVgOvb2VtkiRJkqSnVTly\nKEmSJEmqKcOhJEmSJMlwKEmSJEkyHEqSJEmSMBxKkiRJkjAcSpIkSZIwHEqSJEmSMBxKkiRJkjAc\nSpIkSZIwHEqSJEmSMBxKkiRJkjAcSpIkSZIwHEqSJEmSMBxKkiRJkjAcSpIkSZIwHEqSJEmSMBxK\nkiRJkjAcSpIkSZIwHEqSJEmSMBxKkiRJkjAcSpIkSZIwHEqSJEmSMBxKkiRJkjAcSpIkSZIwHEqS\nJEmSMBxKkiRJkjAcSpIkSZIwHEqSJEmSMBxKkiRJkjAcSpIkSZIwHEqSJEmSMBxKkiRJkjAcSpIk\nSZIwHEqSJEmSMBxKkiRJkjAcSpIkSZIwHEqSJEmSgM4qDxYRmwFfBHYE1gMnA+uAeUAPsAw4JTOf\nioiTgenl6zMzc0GVtUqSJEnSSFL1yOERQGdm7g98CPgIMAs4OzOnAh3AURGxHXAacABwGHBBRGxe\nca2SJEmSNGJUHQ7vBzojYhQwHngSmAQsKl9fCBwK7AMsycy1mfkIsBzYreJaJUmSJGnEqHRaKfAY\nxZTSnwJbA68BDsrMnvL1VcAEiuD4SMP7etsHNXHiODo7Rw9lvW3R3d3V7hK0CRkJvy8j4TNuCuyH\n+rAv6sF+qA/7oh7sh01f1eHwXcANmfmBiNgeuBkY0/B6F/Aw8Gi53Ld9UCtXrh7CUtuju7uLFStW\ntbsMbUKG+++L50Q92A/1YV/Ug/1QH/ZFPdgP9bExIb3qaaUreXpE8A/AZsDdETGtbDscWAzcAUyN\niLERMQHYheJhNZIkSZKkFqh65PBiYG5ELKYYMTwLuBOYExFjgPuA+Zm5PiJmUwTFUcCMzFxTca2S\nJEmSNGJUGg4z8zHgH/t56eB+tp0DzGl5UZIkSZKkyqeVSpIkSZJqyHAoSZIkSTIcSpIkSZIMh5Ik\nSZIkDIeSJEmSJAyHkiRJkiQMh5IkSZIkDIeSJEmSJAyHkiRJkiQMh5IkSZIkDIeSJEmSJJ5DOIyI\n8RGxayuKkSRJkiS1R2czG0XEScD+wBnA3cCqiLg2M89uZXGSJEmSpGo0O3L4duC9wBuBbwAvA17d\nqqIkSZIkSdVqelppZv4BOAL4VmauA7ZoWVWSJEmSpEo1Gw7vjYgFwE7AtyPiK8APW1eWJEmSJKlK\nzYbDE4GPAftm5hPAlcBJLatKkiRJklSpZsPhKGAqcElEjAf2fBbvlSRJkiTVXLMB73JgS2ASsA7Y\nGfhCq4qSJEmSJFWr2XA4KTPPAp7MzNXAWyhGDyVJkiRJw0Cz4bAnIsYAPeX61g3LkiRJkqRNXLPh\n8BLg28B2EXEJcBdwccuqkiRJkiRVqrOZjTLzyoi4C3g5MBo4MjPvaWllkiRJkqTKNDVyGBEvA87N\nzMuBm4DLIyJaWpkkSZIkqTLNTiudA8wDyMz7gA/j00olSZIkadhoNhxumZnX965k5k0UX20hSZIk\nSRoGmrrnEPhdRLwNuKpcPwb4bWtKkiRJkiRVrdmRwxOA1wAPAr8CjgROalVRkiRJkqRqNfu00l9R\nhENJkiRJ0jDUVDiMiMOAmcBWQEdve2bu1KK6JEmSJEkVavaew8uAdwPLgJ7WlSNJkiRJaodmw+Hv\nM3NBSyuRJEmSJLVNs+FwcUTMAq4H1vQ2Zub3WlKVJEmSJKlSzYbDfcr/7tnQ1gMcMrTlSJIkSZLa\nodmnlb681YVIkiRJktqn2aeV7gB8HtgRmApcDZyYmb9oWWWSJEmSpMqManK7zwIfBx4DfgtcA3yp\nVUVJkiRJkqrV7D2HW2fmjRFxUWb2AHMi4pTncsCI+ADwt8AY4NPAImAexT2My4BTMvOpiDgZmA6s\nA2b6tFRJkiRJap1mw+HjEfGXlN9xGBEHAmuf7cEiYhqwP3AAMA54LzALODszb4mIK4CjIuIHwGnA\nZGAscGtE3JSZz/qY0nB34oU3t7uEfs090+dVSZIkbUqaDYfvAhYAfxURS4GtgNc/h+MdBtwDXAeM\nB94HnEwxegiwEHgVsB5YUobBtRGxHNgN+OFzOKYkSZIkaQOaDYfbAnsDLwZGAz/NzCeew/G2BnYA\nXgO8CPh3YFQ5VRVgFTCBIjg+0vC+3vZBTZw4js7O0c+hrHrp7u5qdwnSRhvK32PPiXqwH+rDvqgH\n+6E+7It6sB82fc2Gw49l5reAezfyeA/xdLDMiFgDbN/wehfwMPBoudy3fVArV67eyPLar7u7ixUr\nVrW7DGmjDdXvsedEPdgP9WFf1IP9UB/2RT3YD/WxMSG92XD4QETMBW4HHu9tzMxn+8TSW4HTI2IW\n8HxgS+A7ETEtM28BDge+C9wBfCQixgKbA7tQPKxGkiRJktQCzYbDh4AOYEpDWw/P8ussMnNBRBxE\nEf5GAacAP6d4+ukY4D5gfmauj4jZwOJyuxmZuebZHEuSJEmS1LymwmFmnjBUB8zM9/fTfHA/280B\n5gzVcSVJkiRJA2sqHEbEzym/xqJRZu405BVJkiRJkirX7LTSaQ3LmwGvo7gXUJIkSZI0DDQ7rfSX\nfZo+HhF3AjOHviRJkiRJUtWanVZ6UMNqB7ArsEVLKpIkSZIkVa7ZaaXnNyz3AL8H3jL05UiSJEmS\n2qHZaaUvj4htMvN3ETEOeEFmLm9xbZIkSZKkioxqZqOIOBW4vlztBr4ZEW9tWVWSJEmSpEo1FQ6B\n6cBU+NPDaSYBp7aqKEmSJElStZoNh5sBaxvWn6Cf7z2UJEmSJG2amn0gzdeBmyPiK+X60cA3WlOS\nJEmSJKlqTY0cZuYZwGwggJ2ASzPznFYWJkmSJEmqTrMPpHkBsHdmngpcBvx9RGzb0sokSZIkSZVp\n9p7DLwM/K5d/AywGrmxJRZIkSZKkyjUbDrfKzM8CZObazJwDbN26siRJkiRJVWo2HD4eEYf3rkTE\nK4A/tqYkSZIkSVLVmn1a6XTgqojonUr6a+BNrSlJkiRJklS1DY4cRsRBwCeBFwJbAD8GTs/MZS2u\nTZIkSZJUkUHDYUQcAlwDXAvsDxwIzAeuiYhpLa9OkiRJklSJDU0rPQ84MjOXNrTdHRG3AxcDB7Ws\nMkmSJElSZTY0rXR8n2AIQGbeBWzVmpIkSZIkSVXbUDh8XkQ8Y3SxbGv2YTaSJEmSpJrbUDi8Abio\nsSEiRlNMKf1Wq4qSJEmSJFVrQ6N/ZwDfjIjlwJ3l9pOBe4GjW1ybJEmSJKkig4bDzPwjcEhEHAzs\nDfQAl2TmrVUUJ0mSJEmqRlP3DWbmImBRi2uRJEmSJLXJhu45lCRJkiSNAIZDSZIkSZLhUJIkSZJk\nOJQkSZIkYTiUJEmSJGE4lCRJkiRhOJQkSZIkYTiUJEmSJGE4lCRJkiRhOJQkSZIkYTiUJEmSJGE4\nlCRJkiQBne04aERsA9wFvBJYB8wDeoBlwCmZ+VREnAxML1+fmZkL2lGrJEmSJI0ElY8cRsRmwGeB\nx8umWcDZmTkV6ACOiojtgNOAA4DDgAsiYvOqa5UkSZKkkaId00o/AVwB/KZcnwQsKpcXAocC+wBL\nMnNtZj4CLAd2q7pQSZIkSRopKp1WGhHHAysy84aI+EDZ3JGZPeXyKmACMB54pOGtve2DmjhxHJ2d\no4ew4vbo7u5qdwnSRhvK32PPiXqwH+rDvqgH+6E+7It6sB82fVXfc3gi0BMRhwJ7AF8Ctml4vQt4\nGHi0XO7bPqiVK1cPXaVt0t3dxYoVq9pdhrTRhur32HOiHuyH+rAv6sF+qA/7oh7sh/rYmJBeaTjM\nzIN6lyPiFuBtwMcjYlpm3gIcDnwXuAP4SESMBTYHdqF4WI0kSZIkqQXa8rTSPt4DzImIMcB9wPzM\nXB8Rs4HFFPdFzsjMNe0sUpIkSZKGs7aFw8yc1rB6cD+vzwHmVFaQJEmSJI1g7XhaqSRJkiSpZgyH\nkiRJkiTDoSRJkiTJcChJkiRJwnAoSZIkScJwKEmSJEnCcChJkiRJwnAoSZIkScJwKEmSJEnCcChJ\nkiRJwnAoSZIkScJwKEmSJEnCcChJkiRJwnAoSZIkScJwKEmSJEnCcChJkiRJwnAoSZIkScJwKEmS\nJEnCcChJkiRJwnAoSZIkScJwKEmSJEnCcChJkiRJwnAoSZIkScJwKEmSJEnCcChJkiRJwnAoSZIk\nScJwKEmSJEnCcChJkiRJwnAoSZIkScJwKEmSJEnCcChJkiRJwnAoSZIkScJwKEmSJEnCcChJkiRJ\nwnAoSZIkScJwKEmSJEnCcChJkiRJAjqrPFhEbAbMBXYENgdmAj8B5gE9wDLglMx8KiJOBqYD64CZ\nmbmgylolSZIkaSSpeuTwOOChzJwKvBr4FDALOLts6wCOiojtgNOAA4DDgAsiYvOKa5UkSZKkEaPS\nkUPgq8D8crmDYlRwErCobFsIvApYDyzJzLXA2ohYDuwG/LDaciVJkiRpZKg0HGbmYwAR0UUREs8G\nPpGZPeUmq4AJwHjgkYa39rYPauLEcXR2jh7Smtuhu7ur3SVIG20of489J+rBfqgP+6Ie7If6sC/q\nwX7Y9FU9ckhEbA9cB3w6M6+OiI81vNwFPAw8Wi73bR/UypWrh7LUtuju7mLFilXtLkPaaEP1e+w5\nUQ/2Q33YF/VgP9SHfVEP9kN9bExIr/Sew4jYFrgROCMz55bNd0fEtHL5cGAxcAcwNSLGRsQEYBeK\nh9VIkiRJklqg6pHDs4CJwDkRcU7ZdjowOyLGAPcB8zNzfUTMpgiKo4AZmbmm4lolSZIkacSo+p7D\n0ynCYF8H97PtHGBOy4uSJEmSJFX+VRaSJEmSpBoyHEqSJEmSDIeSJEmSJMOhJEmSJAnDoSRJkiSJ\n6r/KYkQ68cKb212CJEmSJA3KkUNJkiRJkuFQkiRJkmQ4lCRJkiRhOJQkSZIk4QNpJLVIXR/ENPfM\nQ9pdgiRJUi05cihJkiRJMhxKkiRJkgyHkiRJkiQMh5IkSZIkDIeSJEmSJAyHkiRJkiQMh5IkSZIk\nDIeSJEmSJAyHkiRJkiQMh5IkSZIkDIeSJEmSJAyHkiRJkiQMh5IkSZIkDIeSJEmSJAyHkiRJkiQM\nh5IkSZIkDIeSJEmSJAyHkiRJkiSgs90FSFKVTrzw5naX0K+5Zx7S7hIkSdII58ihJEmSJMlwKEmS\nJEkyHEqSJEmSMBxKkiRJkjAcSpIkSZIwHEqSJEmS8KssJKkW/IoNSZLUbrUNhxExCvg0sDuwFjgp\nM5e3typJGlkMrZIkjRx1nlb6d8DYzNwPOBP4ZJvrkSRJkqRhq7Yjh8CBwPUAmXlbRExucz2SpJpw\nRFOSpKHX0dPT0+4a+hURnweuzcyF5fqvgJ0yc117K5MkSZKk4afO00ofBboa1kcZDCVJkiSpNeoc\nDpcARwBExBTgnvaWI0mSJEnDV53vObwOeGVEfB/oAE5ocz2SJEmSNGzV9p5DSZIkSVJ16jytVJIk\nSZJUEcOhJEmSJKnW9xwOexFxPHB8uToW2APYD1gA/FfZ/pnM/LfKixshImJf4KLMnBYROwPzgB5g\nGXBKZj4VEScD04F1wMzMXNC2goexPn2xB3AZsB5YC7w5M38bEZdSfAfqqvJtR2XmI+2peHjq0w97\n0s/fR54T1ejTF/8KbFe+tCNwW2Ye4znROhGxGTCX4ue9OTAT+AleJyo3QF/8Cq8TlRqgH36N14nK\nDdAXxzIE1wnDYRtl5jyKiwwRcTlFJ08CZmXmJ9tX2cgQEe8H3gT8sWyaBZydmbdExBXAURHxA+A0\nYDJFgL81Im7KzLVtKXqY6qcvLgVOzcylETEdOAN4N8X5cVhm/r49lQ5v/fTDM/4+iojt8Jxoub59\nkZnHlO0Tge8C7yo39ZxoneOAhzLzTRGxFbC0/ON1onr99cXP8TpRtf764UN4nWiHZ/RFZr4QNv46\n4bTSGoiIycCumfk5ig48MiK+FxFfiIiuDbxdz90DwNEN65OAReXyQuBQYB9gSWauLf+VZTmwW6VV\njgx9++KYzFxaLncCayJiFPDXwOciYklEnFh1kSNAf+dE37+PPCeq0bcvep0PXJaZD3pOtNxXgXPK\n5Q6KERCvE+3RX194najeQOeE14nq9dcXvTbqOmE4rIezKDoS4A7gfZl5EPAz4Ly2VTXMZea1wJMN\nTR2Z2fv43lXABGA80Dj03tuuIdS3LzLzQYCI2B94J3AxsCXFFKLjgFcD74gILzZDqJ9zor+/jzwn\nKtBPXxAR2wCvoJxxgudES2XmY5m5qvyf3fnA2XidaIv++sLrRPUGOCe8TrTBAH0xJNcJw2GbRcSf\nAZGZ3y2brsvMu3qXgT3bU9mI9FTDchfwMPBoudy3XS0WEW8ArgCOzMwVwGrg0sxcnZmrgJuB3dtZ\n4wjQ399HnhPt8w/A1Zm5vlz3nGixiNieYnrWlZl5NV4n2qafvvA60Qb99IPXiTbp75xgCK4ThsP2\nOwj4TsP6DRGxT7n8CuCuZ75FLXJ3REwrlw8HFlP8i9jUiBgbEROAXSgeQqAWiojjKP4leFpm/qxs\nfjGwJCJGlzdiHwj8qF01jhD9/X3kOdE+h1JMZezlOdFCEbEtcCNwRmbOLZu9TrRBf33hdaJ6A5wT\nXifaYIC+gCG4TvhAmvYLimH4Xm8HLouIJ4H/Ad7alqpGpvcAcyJiDHAfMD8z10fEbIr/ARgFzMjM\nNe0scriLiNHAbIon0X0tIgAWZeZ5EXElcBvFdLsvZea97at0RHjG30eZ+ajnRNv8n+tFZt7nOdFS\nZwETgXMiovfentOB2V4nKte3L0YDLwV+ideJKvV3TrwbuNjrROX664vDGYLrREdPT89gr0uSJEmS\nRgCnlUqSJEmSDIeSJEmSJMOhJEmSJAnDoSRJkiQJw6EkSZIkCb/KQpJUgYi4HDgAGAPsDPykfOnS\nzPyXFh0Z356IAAAFGUlEQVRzZ4rvgDp5I/dzFXB9Zl61kfu5AXhTZv5uY/ZTlYg4CZiSmSe1uxZJ\nUjUMh5KklsvMUwAiYkfglszco4LD7gi8qILjNCUzD2t3DZIkDcZwKElqq4jYHvg88GfAdsBVmTmj\nHLk6FugGriu3uarc7j+BaZm5fUR0AZcDu1LcLnFBZn4FmA1sHxGzM/O0huP9OzA3M79eri8F3gxs\nDXwY2KI8xnsz82sN79uZYgRx53J9JrAmM2dGxJHABymuqw9QfBH0H/p8zv8GpgCvBl5RHm8n4D8y\n89Q+276w/KzjgPXAqZl5R0TsC8wqa1xRHueXEbEXcEXZ/hBwbGb+pvxy5DeW+7geOKM85r8CCewO\nPAi8PjMfjojjgQ8AjwK/AlaW9VwMHAI8BVybmTMH6k9J0qbLew4lSe32T8CVmbkvsAdwWkRMLF97\nAbBHZp4LXEYRHHcD/h14frnNecBtmTkJmAacFxE7AKcBtzcGw9KVwDEAEbEL0JGZPwZOBY7PzL2A\ntwHnNFN8RGwLzARemZl7At8FPrqBt00BXgfsBvx9WUejk4HrMnMycBZwQERsDswB3lDWOBv4bLn9\nNcC5mfkyYD5wakT8LUUQ3QvYE9il3C/l+kWZ+VJgNfDGMqR/FDgQ2A+YUH6+vwIOyczdgf2Bl0TE\nmGZ+NpKkTYsjh5KkdrsIOCQi3kcx+jeGYsQM4K7MXF8uH0oxCkZmfjUiPtfQPiYi3lqubwn8DfDk\nAMf7JnBJRGxZ7u/LZfsbgddGxBspwtvzmqx/P2AH4JaIgOLa+tsNvGdJZj4GEBE/B7bq8/pNwPyI\nmAx8C/g0RbjbCVhQHqcD2CIitgMmZub1AJn5qXK/lwBXZ+aacv1fgDcA3wEeLAMxwLLy+AcAizNz\nRbn9l8u2XwPrI2IxsACYkZlPNPmzkSRtQhw5lCS12yXAO4CfU0zrXEkRfAAeb9hufUN7o9HAMZm5\nR3kv477Atwc6WBmWFgKvAV4PXB0RHcASYBLwQ+CCfo7V06dts4bj39Jw/MkUIWwwawbZL5n5PYqA\nexPF1Nqvl8e5v+E4ewEH0ycER8QWEfEinnmN7+DpfxTu7/g9fd6zrqzlCWAf4HxgG+C2cjRRkjTM\nGA4lSe32SoopjvMpHiCzLUUQ6uvbFEGJiHgtT4/s3Qy8vWz/C+Aeiumo6xh4hsyVFPffPZiZ/01x\nX+OLgPMycyHwqn5qWAl0R8SfR8RYoPcBMz8ApjYEpvOBC5v76P2LiFkUgXcexfTYvSie8LpdROxf\nbvZWium4DwErIuKQsv14iqm2NwPHRsTYiOgETqCY8jqQxRTTV58fEaOBfyxrmUwx2ngL8F7gfuDF\nG/P5JEn1ZDiUJLXbR4FrIuIu4F3A3fT/lNHTgGMi4m6K+/VWle3nAhMi4h6KkbZ3Z+YvgXspwty8\nfvb1PYqplFcBlF8vcSVwb7n/rYDxEbFF7xvKB8xcDNxVHue2sv03FPfyfS0ilgEvBd7/3H4Uf3Jp\n+VmXAl8B3pGZj1MEtksj4scU02B77yE8Fvhwuf3RFF/h8XXgRuDO8mfxX8BnBjpg+TneRREgbwce\nLtvvLD/zsvK/95f7lSQNMx09PT3trkGSpA2KiH8GFmZmRsQ+wGXlQ2wkSdIQ8IE0kqRNxXLgKxHx\nFMW9iNPbXI8kScOKI4eSJEmSJO85lCRJkiQZDiVJkiRJGA4lSZIkSRgOJUmSJEkYDiVJkiRJGA4l\nSZIkScD/Alr5mQ/8Xs/HAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10db930f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_train = df_train['y'].values\n",
    "plt.figure(figsize=(15, 5))\n",
    "plt.hist(y_train, bins=20)\n",
    "plt.xlabel('Target value in seconds')\n",
    "plt.ylabel('Occurences')\n",
    "plt.title('Distribution of the target value')\n",
    "\n",
    "print('min: {} max: {} mean: {} std: {}'.format(min(y_train), max(y_train), y_train.mean(), y_train.std()))\n",
    "print('Count of values above 180: {}'.format(np.sum(y_train > 200)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Число признаков и объектов:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "_cell_guid": "85c914e4-162a-e3be-ec91-1e687b8222e2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of features: 376\n",
      "Feature types:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "int64     368\n",
       "object      8\n",
       "dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cols = [c for c in df_train.columns if 'X' in c]\n",
    "print('Number of features: {}'.format(len(cols)))\n",
    "\n",
    "print('Feature types:')\n",
    "df_train[cols].dtypes.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Приведение данных к вещественному виду:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/user/anaconda/lib/python3.6/site-packages/ipykernel_launcher.py:10: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  # Remove the CWD from sys.path while we load stuff.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X63</th>\n",
       "      <th>X47</th>\n",
       "      <th>X370</th>\n",
       "      <th>X61</th>\n",
       "      <th>X212</th>\n",
       "      <th>X323</th>\n",
       "      <th>X368</th>\n",
       "      <th>X298</th>\n",
       "      <th>X211</th>\n",
       "      <th>X169</th>\n",
       "      <th>...</th>\n",
       "      <th>X241</th>\n",
       "      <th>X271</th>\n",
       "      <th>X263</th>\n",
       "      <th>X274</th>\n",
       "      <th>X117</th>\n",
       "      <th>X17</th>\n",
       "      <th>X16</th>\n",
       "      <th>X233</th>\n",
       "      <th>X366</th>\n",
       "      <th>X293</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 376 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   X63  X47  X370  X61  X212  X323  X368  X298  X211  X169  ...   X241  X271  \\\n",
       "0    0    0     0    0     0     0     0     0     0     0  ...      0     0   \n",
       "1    0    0     0    1     0     0     0     0     0     0  ...      0     0   \n",
       "2    0    0     0    1     0     0     0     0     0     1  ...      1     0   \n",
       "3    0    0     0    1     0     0     0     0     0     0  ...      1     0   \n",
       "4    0    0     0    1     0     0     0     0     0     0  ...      0     0   \n",
       "\n",
       "   X263  X274  X117  X17  X16  X233  X366  X293  \n",
       "0     1     0     0    0    0     0     0     0  \n",
       "1     1     0     0    0    0     0     0     0  \n",
       "2     0     1     0    1    0     0     0     0  \n",
       "3     0     0     0    0    0     0     0     0  \n",
       "4     0     0     0    0    0     0     0     0  \n",
       "\n",
       "[5 rows x 376 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "usable_columns = list(set(df_train.columns) - set(['ID', 'y']))\n",
    "y_train = df_train['y'].values\n",
    "x_train = df_train[usable_columns]\n",
    "for column in usable_columns:\n",
    "    cardinality = len(np.unique(x_train[column]))\n",
    "    if cardinality == 1:\n",
    "        x_train.drop(column, axis=1) # Column with only one value is useless so we drop it\n",
    "    if cardinality > 2: # Column is categorical\n",
    "        mapper = lambda x: sum([ord(digit) for digit in x])\n",
    "        x_train[column] = x_train[column].apply(mapper)\n",
    "        \n",
    "x_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Разделение данных\n",
    "Разделим данные на три части. Первую будем использовать для настройки случайного леса и градиентного бустинга, вторую - для настройки композиции, третью - для итогового контроля."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((4209, 376), (4209,))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape, y_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.utils import shuffle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x_train, y_train = shuffle(x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((2000, 376), (2000,))"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x1, y1 = x_train[:2000], y_train[:2000]\n",
    "x1.shape, y1.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1000, 376), (1000,))"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x2, y2 = x_train[2000:3000], y_train[2000:3000]\n",
    "x2.shape, y2.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1209, 376), (1209,))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x3, y3 = x_train[3000:], y_train[3000:]\n",
    "x3.shape, y3.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Настройка моделей"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/user/anaconda/lib/python3.6/site-packages/sklearn/cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.\n",
      "  \"This module will be removed in 0.20.\", DeprecationWarning)\n",
      "/Users/user/anaconda/lib/python3.6/site-packages/sklearn/grid_search.py:43: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. This module will be removed in 0.20.\n",
      "  DeprecationWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor\n",
    "from sklearn.metrics import r2_score, make_scorer\n",
    "from sklearn.grid_search import GridSearchCV"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "В этой части вам надо найти оптимальные гиперпараметры для случайного леса и градиентного бустинга, используя первую часть выборки (x1, y1).\n",
    "\n",
    "Начнем со случайного леса. \n",
    "1. (уже сделано ниже) Постройте график зависимости качества от числа деревьев в случайном лесе на кросс-валидации с 3 блоками. Выберите число деревьев для использования в дальнейшей настройке и при итоговом обучении модели.\n",
    "1. Постройте график зависимости качества от max_features на кросс-валидации. Выберите оптимальное значение гиперпараметра.\n",
    "1. С помощью кросс-валидации выберите оптимальные значения для max_depth, min_samples_leaf, min_samples_split.\n",
    "1. Обучите случайный лес с выбранными гиперпараметрами на всей выборке x1, y1.\n",
    "\n",
    "Указания: verbose=3 в GridSearchCV будет выводить значения качества на каждой обучаемой модели. Рекомендуется в экспериментах зафиксировать random_state у модели. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "param_grid = {\"n_estimators\":2**np.arange(10)} #  логарифмическая шкала\n",
    "gs = GridSearchCV(RandomForestRegressor(random_state=1, warm_start=True), param_grid, make_scorer(r2_score), verbose=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 10 candidates, totalling 30 fits\n",
      "[CV] n_estimators=1 ..................................................\n",
      "[CV] ........................ n_estimators=1, score=-0.684794 -   0.0s\n",
      "[CV] n_estimators=1 ..................................................\n",
      "[CV] ........................ n_estimators=1, score=-0.093801 -   0.1s\n",
      "[CV] n_estimators=1 ..................................................\n",
      "[CV] ......................... n_estimators=1, score=0.089762 -   0.0s\n",
      "[CV] n_estimators=2 ..................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.1s remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ......................... n_estimators=2, score=0.081357 -   0.1s\n",
      "[CV] n_estimators=2 ..................................................\n",
      "[CV] ......................... n_estimators=2, score=0.116615 -   0.1s\n",
      "[CV] n_estimators=2 ..................................................\n",
      "[CV] ......................... n_estimators=2, score=0.264400 -   0.1s\n",
      "[CV] n_estimators=4 ..................................................\n",
      "[CV] ......................... n_estimators=4, score=0.345549 -   0.1s\n",
      "[CV] n_estimators=4 ..................................................\n",
      "[CV] ......................... n_estimators=4, score=0.378655 -   0.2s\n",
      "[CV] n_estimators=4 ..................................................\n",
      "[CV] ......................... n_estimators=4, score=0.332547 -   0.2s\n",
      "[CV] n_estimators=8 ..................................................\n",
      "[CV] ......................... n_estimators=8, score=0.430955 -   0.3s\n",
      "[CV] n_estimators=8 ..................................................\n",
      "[CV] ......................... n_estimators=8, score=0.479509 -   0.3s\n",
      "[CV] n_estimators=8 ..................................................\n",
      "[CV] ......................... n_estimators=8, score=0.370495 -   0.3s\n",
      "[CV] n_estimators=16 .................................................\n",
      "[CV] ........................ n_estimators=16, score=0.435009 -   0.5s\n",
      "[CV] n_estimators=16 .................................................\n",
      "[CV] ........................ n_estimators=16, score=0.498499 -   0.6s\n",
      "[CV] n_estimators=16 .................................................\n",
      "[CV] ........................ n_estimators=16, score=0.364089 -   0.6s\n",
      "[CV] n_estimators=32 .................................................\n",
      "[CV] ........................ n_estimators=32, score=0.459496 -   1.2s\n",
      "[CV] n_estimators=32 .................................................\n",
      "[CV] ........................ n_estimators=32, score=0.516708 -   1.2s\n",
      "[CV] n_estimators=32 .................................................\n",
      "[CV] ........................ n_estimators=32, score=0.391338 -   1.2s\n",
      "[CV] n_estimators=64 .................................................\n",
      "[CV] ........................ n_estimators=64, score=0.457425 -   2.3s\n",
      "[CV] n_estimators=64 .................................................\n",
      "[CV] ........................ n_estimators=64, score=0.512309 -   2.2s\n",
      "[CV] n_estimators=64 .................................................\n",
      "[CV] ........................ n_estimators=64, score=0.404156 -   2.1s\n",
      "[CV] n_estimators=128 ................................................\n",
      "[CV] ....................... n_estimators=128, score=0.469284 -   4.1s\n",
      "[CV] n_estimators=128 ................................................\n",
      "[CV] ....................... n_estimators=128, score=0.508253 -   4.1s\n",
      "[CV] n_estimators=128 ................................................\n",
      "[CV] ....................... n_estimators=128, score=0.405360 -   4.3s\n",
      "[CV] n_estimators=256 ................................................\n",
      "[CV] ....................... n_estimators=256, score=0.474971 -   8.2s\n",
      "[CV] n_estimators=256 ................................................\n",
      "[CV] ....................... n_estimators=256, score=0.524991 -   8.3s\n",
      "[CV] n_estimators=256 ................................................\n",
      "[CV] ....................... n_estimators=256, score=0.407289 -   8.8s\n",
      "[CV] n_estimators=512 ................................................\n",
      "[CV] ....................... n_estimators=512, score=0.475730 -  16.1s\n",
      "[CV] n_estimators=512 ................................................\n",
      "[CV] ....................... n_estimators=512, score=0.525189 -  16.1s\n",
      "[CV] n_estimators=512 ................................................\n",
      "[CV] ....................... n_estimators=512, score=0.406250 -  16.6s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  30 out of  30 | elapsed:  1.7min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=None, error_score='raise',\n",
       "       estimator=RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,\n",
       "           max_features='auto', max_leaf_nodes=None,\n",
       "           min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "           min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "           n_estimators=10, n_jobs=1, oob_score=False, random_state=1,\n",
       "           verbose=0, warm_start=True),\n",
       "       fit_params={}, iid=True, n_jobs=1,\n",
       "       param_grid={'n_estimators': array([  1,   2,   4,   8,  16,  32,  64, 128, 256, 512])},\n",
       "       pre_dispatch='2*n_jobs', refit=True, scoring=make_scorer(r2_score),\n",
       "       verbose=3)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs.fit(x1, y1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x110fc1048>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAEPCAYAAABGP2P1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYHHd95/F3H3Mf0kgzkmzZ1mFJX4QPScb4wrGN4wMU\nWMzhJCgEcPASkl12l5DdBZaQTTZ5AiROQjZxHrwPWQ5DEocgMEuQbcA2WD4Ae2zJxv5KY122bEkz\n0mg0mru7a//oHql7LM1oemaq+vi8nmee7q6qrvp6fnJ9pupXVb9YEASIiIiMi0ddgIiIlBYFg4iI\nFFAwiIhIAQWDiIgUUDCIiEgBBYOIiBRIhrkxM4sDdwLrgBHgdnfvypv/MeB2oDs36bfd3cOsUUSk\n2oUaDMAtQL27X2lmVwB3AO/Im/8G4P3u/mTIdYmISE7YwXA1sAXA3R83s0snzH8D8EkzWwJ8z93/\nbKoVdnf3F32HXltbI729g8V+XUqA2rC8qf2i09HREjvdvLCDoRXoy/ucNrOku6dyn/8J+DvgGLDZ\nzN7m7v9vshW2tTWSTCaKLqijo6Xo70ppUBuWN7Vf6Qk7GI4B+f8K4uOhYGYx4K/dvS/3+XvABmDS\nYJjJXxsdHS10d/cX/X2JntqwvKn9ojNZIId9VdJWYCNAro9he968VuBZM2vOhcT1gPoaRERCFvYR\nw2bgRjN7FIgBt5nZJqDZ3e8ys08BD5K9YumH7v5vIdcnIlL1YuX+dNWZdD7rMLb8qQ3Lm9ovOpN1\nPusGNxERKaBgEBGRAgoGEREpEHbns4hIyQmCgOHRNIPDKQaGxxgYTjGYex0YHstOHzo5PZ2Jvm82\nmYjzrmtXsnxJ6+yve9bXKCISkbFUOrszP7ETP7mjHxjK7eBHxhgYyu7gj+d29IPDqZLY2U9HIh7j\nUO+QgkFEylMmEzCWzpBOZxhLB7nXDEPpgO6e47l5ecukAtKZDKl0hlQ6yL6mMgyPpV/z1/tA3s5/\nLJU545oS8RhN9Uma6mtY1NZAU30NTfVJGnOvTfU1NOZemxrypydJJmZ2Fn62IigeO+2FRTOiYBAp\nYZkgIJ0OyGSyO8p0Jsj+pAPSQXYHmxmflvvJZLLT0wXfHf/JTFiucHrBsnnbTWWCgp36iZ11esLO\nu+D9yWlzdVV8DE7svOe319HU8Nqde8HnE/OT1NUkiM3RjvVM6i5lCgaREnF8aIy9B/vZd6CfvQf7\n2Xugn4O9Q1GXNakYkEzGSSbiJBOxE6/1tTUTpp1cJpGIU5N7bWmuY2w0RU0ifmJ6Mu999jVOIhGj\nJhGntiZBU8PJHX59XXLO/mquZgoGkQgcPT7C3lwA7Dt4nL0H+jl8bLhgmca6JKvOmUddMk48HicR\nj2V/EtnXeDxGIm96PG9eIh4nHo+RPLHc+Hfjed8tXDaRiJGIZV+z342fXC63Y05MCIBEfGanVHSD\nW2lSMIjMoSAIOHxsmL0HjudCIHsk0DcwWrBcS2MNF65cwLLFLdmfJS20z6uP7FSHVDcFg8gsyQQB\n3b1DJ04Djb8ODKcKllvQWseG1e0sW9zCebkQmN9cqxCQkqFgEClCOpPh1cODJ08HHehn36HjDI+m\nC5ZbNL+BtcsXsGxxM8uWZIOgtbE2oqpFzoyCQWQKY6kMr/QMFBwJvHToeMGlkbEYnLWwifMWN584\nHXTe4mYa62sirFykOAoGqTpBEDCayjAymmZ4LM3IaO5nLM3waJqRsRQDQyle7s72C+zvHii4+SkR\nj7G0vYnzlpzsDzi3o5m62uJHEhQpJQoGKWmpdIaRsQk77rwdek3dEXqODJxcZjTN8FjqNTv94bF0\nwTJnell9TTLOsiUnjwCWLWlhaXszNUk9Zkwql4JBSkbXy3188+EXOdw3lPvLPU0qPbM7o2JAXW2C\nutoE9TUJ5jXWFnzOviZPTKurSVBfm/05u72JsxY2zviSTJFyo2CQyA0Mj/GvD73IQ0+/AsDC1jrm\nN9cV7KjH3+d/rq9J0L6wmdHhscJl8nb6tcm4rvYRmSYFg0QmCAJ++vwh/vGHOzk2MMrSjiY+cPPr\nWHXOvDNeh26QEpl9CgaJxKGjQ9x9n/Ps7iPUJOO8+9qV3HzZeTN+OJmIzJyCQUKVSme476f7uHfr\nHsZSGS5YsYDfvNlYNL8h6tJEJCfUYDCzOHAnsA4YAW53965TLHcXcMTdPxFmfTK3ul7u4yv3vcD+\n7gFam2r5rY2ruWztIvUBiJSYsI8YbgHq3f1KM7sCuAN4R/4CZvbbwEXAwyHXJnNkYufydevP5t3X\nnU+Tbv4SKUlhB8PVwBYAd3/czC7Nn2lmVwGXA18EXhdybTLLXtO53N7E+99irD5nftSlicgkwg6G\nVqAv73PazJLunjKzs4A/BN4J/GrIdcksU+eySPkKOxiOAS15n+PuPv7oyVuBduDfgCVAo5m94O5f\nnmyFbW2NJJPFP4qgo6Nl6oXkjKXSGTY/1MU/3e+MpjJsWNPB77x7HWe1N83ZNtWG5U3tV3rCDoat\nwNuBe3J9DNvHZ7j73wB/A2BmHwReN1UoAPT2DhZdjK6Bn11d+/v4ypZc53JjDbdtXJvtXA4yc/Z7\nVhuWN7VfdCYL5LCDYTNwo5k9SvZpBbeZ2Sag2d3vCrkWmSWDw2N88+FdPNS5H4Br15/Ne9S5LFK2\nYsFcjdIdku7u/qL/A/TXyswEQcDPXjjEN34QXeey2rC8qf2i09HRctrrxHWDmxSl++gQX7vfeXaX\nOpdFKo2CQaYllc5w/89e4t5HdjM6fufyTWtY1NYYdWkiMksUDHLGJnYuf3Dj67h87WLduSxSYRQM\nMqXxzuWHO/cToM5lkUqnYJDTGu9c/scf7KRPdy6LVA0Fg5ySOpdFqpeCQQq8pnN5eVv2sdjqXBap\nGgoGOaFrfx9f3fICL6tzWaSqKRgEgH95sIstT+wjAK5Zdza3vlmdyyLVSsEgPNPVw/ef2MfitgZu\n27iWNeeqc1mkmikYqtzoWJpv/GAH8ViM//CuizinoznqkkQkYrrEpMp9/4l9dB8d5oZLz1EoiAig\nYKhqh44O8b3H9jK/uZZ3XL0i6nJEpEQoGKpUEAR844EdpNIZfu361TTU6ayiiGQpGKrU0109bHvx\nMGuXtXHZ2kVRlyMiJUTBUIVGxtJ844GdJOIxfuPGNbpPQUQKKBiq0Pce28vhY8PcdNm5nD2HYzGL\nSHlSMFSZg0cG2fLEXtpa6nj7VcujLkdESpCCoYoEQcDdD+wglQ547y+vpr5WHc4i8loKhirypHfz\n3O4jXLBiAW+wjqjLEZESpWCoEiOjaf7xhztJJtThLCKTC/VcgpnFgTuBdcAIcLu7d+XNfzfwCSAA\nvu7uXwizvkr23Uf30Ns/wtuuWsaSBXqEtoicXthHDLcA9e5+JdkAuGN8hpklgM8CNwBXAr9rZu0h\n11eRXj08wH0/3cfC1np+5crlUZcjIiUu7GC4GtgC4O6PA5eOz3D3NLDW3fuAhUACGA25vooTBAF3\n37+DdCZg0w2rqatJRF2SiJS4sC9LaQX68j6nzSzp7ikAd0+Z2buAvwO+BwxMtcK2tkaSyeJ3dh0d\nLUV/txz8pHM/z+/t5dK1i7nxqhUV2bdQ6W1Y6dR+pSfsYDgG5P8riI+Hwjh3/5aZfRv4MvB+4P9O\ntsLe3sGii+noaKG7u7/o75e6oZEUd317G8lEnPdcs4KenuNRlzTrKr0NK53aLzqTBXLYp5K2AhsB\nzOwKYPv4DDNrNbOHzazO3TNkjxYyIddXUe7dupujx0fZeMV5GrNZRM5Y2EcMm4EbzexRIAbcZmab\ngGZ3v8vMvg782MzGgG3A3SHXVzFe7j7OAz97mY759Wy8YlnU5YhIGQk1GHJHAh+ZMPmFvPl3AXeF\nWVMlCoKAr9+/g0wQsOmGNdSqw1lEpkE3uFWgx39xEH/pKOtXtbNula74FZHpUTBUmMHhFPf8qIua\nZJxNN6yOuhwRKUMKhgrz7Ud20TcwytuuXEb7/IaoyxGRMqRgqCD7DvbzwydfZlFbA2+5XB3OIlIc\nBUOFyOQeqR0E8L4b11CTVNOKSHG096gQjz17gK6X+3iDdXDhyoVRlyMiZUzBUAEGhse458Euamvi\n/Pr16nAWkZlRMFSAzT/eRf/gGG+/ajkL59VHXY6IlDkFQ5nbe6CfBzv3s2RBIzdfdl7U5YhIBVAw\nlLFMEPC1+z3b4XzTGpIJNaeIzJz2JGXskW2vsuuVY1y2dhGvX74g6nJEpEIoGMrU8aExvvnQi9TV\nJvg1dTiLyCxSMJSpbz38IseHxnjHm1bQ1lIXdTkiUkEUDGVo96vHePjpV1ja3sQNl54TdTkiUmEU\nDGUmkwn42n1OgDqcRWRuaK9SZh5+5hX2HOjnigsWY+e1RV2OiFQgBUMZOTY4yrcefpGGugS/+uZV\nUZcjIhVKwVBG/vWhFxkYTnHL1SuZ36wOZxGZGwqGMtG1v4+fbHuVczqauf4NS6MuR0QqmIKhDGQy\nAXff50C2wzkRV7OJyNxJhrkxM4sDdwLrgBHgdnfvypv/XuC/AClgO/C77p4Js8ZS9GDnfvYdOs6b\nLlzCmnPnR12OiFS4sP/0vAWod/crgU8Ad4zPMLMG4E+AN7v7m4B5wNtCrq/k9A2M8q0f76KhLsmt\n6nAWkRCEHQxXA1sA3P1x4NK8eSPAVe4+mPucBIbDLa/0/MuDXQyNpHjXNStpbaqNuhwRqQKhnkoC\nWoG+vM9pM0u6eyp3yugggJl9FGgGHphqhW1tjSSTiaIL6uhoKfq7c+25XYd59NkDrFw6j1tveh2J\neCzqkkpSKbehTE3tV3rCDoZjQP6/gri7p8Y/5PogPg+sAd7t7sFUK+ztHZxqkdPq6Gihu7u/6O/P\npXQmw9/e0wnAe69fxZHDxyOuqDSVchvK1NR+0ZkskMM+lbQV2AhgZleQ7WDO90WgHrgl75RSVfrh\nk/t5uXuAX7r4LM5fOi/qckSkioR9xLAZuNHMHgViwG1mtonsaaOfAx8CfgL8yMwAvuDum0OuMXJH\nj4/w7Z/soqk+yXuuOz/qckSkyoQaDLl+hI9MmPxC3ntdoA/c86MuhkfTvP9mo6VRHc4iEi7tiEvM\nC3t7efwXB1lxVgvXrDs76nJEpAopGEpIKp3h7gd2EAPed5MR11VIIhIBBUMJ+cHPX+aVngGu3bCU\nFWe1Rl2OiFQpBUOJOHJsmO88spvmhhredc3KqMsRkSqmYCgR//yjLkbG0rznuvNpbqiJuhwRqWIK\nhhLw3J4j/OyFQ5x/ditXX3xW1OWISJVTMERsLJXh6/fvIBbLdTjH1OEsItFSMETsgZ+/xIEjg1y/\n4RyWLdEzY0QkegqGCAVBwA+ffJnGuiTvvGZF1OWIiAAKhkjtPdhPb/8I61a101ivDmcRKQ0Khgg9\ntaMHgEvWtEdciYjISQqGCHXu7KYmGefCFQujLkVE5AQFQ0QO9Q6yv3uAC5YvoK62+IGGRERm25TB\nYGbvMLOPmtn5E6Z/eO7KqnydO7Onkdav1mkkESktkwaDmX0W+CjZEdUeNbP35c2e+PhsmYbOHd3E\ngPWrFAwiUlqmOmL4FeAt7v5R4Grgf5nZrbl5uhOrSMcGR9m5v49V58yjtUnjLYhIaZkqGGJAAODu\nO4G3AV8ws+vGp8v0PdPVQxDAhtUdUZciIvIaUwXDvwAPmdllAO7+HHArcA+gMSeL1Jm7THWDLlMV\nkRI0aTC4+x8B/wzU5U3bClxCNhxkmkZG0zy35whL25tY3NYYdTkiIq8x6ZjPZvaHwH/Nvb8FeBD4\nfeBTwONzXl0Fenb3EcZSGR0tiEjJmjQYgA8Aq4GzgT8G/juwBPhVd79vuhszszhwJ7AOGAFud/eu\nCcs0Ag8AH3L3F6a7jVL39M5uQP0LIlK6pupj6Hf3V939SeAyYBuwvphQyLkFqHf3K4FPAHfkzzSz\nS4EfU6H9F+lMhqe7emhrqWO5nqQqIiVqqmDI5L3vcfePu3t6Btu7GtgC4O6PA5dOmF8HvBOouCMF\ngJ0v9TEwnGL96nZiGndBRErUVMGQf0nq0CxsrxXoy/ucNrMTp7Pcfau7vzQL2ylJT+VOI12i00gi\nUsKm6mO4wMx25d4vzXsfAwJ3n+6o9ceA/HMocXdPTXMdBdraGkkmi3/WUEdHOKd0giBg264jNNYn\nedMl51KT1GOqZktYbShzQ+1XeqYKhjWzvL2twNuBe8zsCmD7TFfY2ztY9Hc7Olro7u6faQlnZN/B\nfg4dGeTy1y/maO9AKNusBmG2ocw+tV90JgvkSYPB3ffOci2bgRvN7FGyRx23mdkmoNnd75rlbZWU\n8YfmbdBD80SkxE11xDCr3D3Dax++95qOZne/LpSCQtS5o5tkIsZFKzX2goiUNp3oDkFP3xD7Dh3n\ndcvaaKgLNYtFRKZNwRCC8dNIuhpJRMqBgiEEnTuyl6lqUB4RKQcKhjl2fGiMHS/1cf7Zrcxvrpv6\nCyIiEVMwzLFtL/aQCQIdLYhI2VAwzLHxsRcuWaP+BREpDwqGOTQ6lmb77sMsWdDIWQuboi5HROSM\nKBjm0C/29DI6prEXRKS8KBjmUKfGXhCRMqRgmCOZTMDTXT20NtWy8uzWqMsRETljCoY50rW/j/7B\nMTasbieusRdEpIwoGOaITiOJSLlSMMyBIAjo3NFDXW2Ctcvaoi5HRGRaFAxz4JWeAQ4dHeKilQs1\nII+IlB3ttebAUycemqfLVEWk/CgY5kDnjm4S8RgXn6+xF0Sk/CgYZtmRY8PsOdCPnTefxvqaqMsR\nEZk2BcMse7prfAhPXY0kIuVJwTDLxsde0NjOIlKuFAyzaHB4jBf2HWX5khYWtNZHXY6ISFEUDLNo\n267DpDOBjhZEpKyFOjK9mcWBO4F1wAhwu7t35c1/O/AZIAX8g7v/nzDrm6nxsRc2aOwFESljYR8x\n3ALUu/uVwCeAO8ZnmFkN8FfATcC1wIfNbHHI9RVtLJVh267DLJrfwNJ2jb0gIuUr7GC4GtgC4O6P\nA5fmzVsLdLl7r7uPAo8A14RcX9Ge39vLyGiaDWvaiemheSJSxkI9lQS0An15n9NmlnT31Cnm9QPz\nplphW1sjyWSi6II6OlqK/m6+Fx7eBcCb37hs1tYpZ0a/7/Km9is9YQfDMSD/X0E8FwqnmtcCHJ1q\nhb29g0UX09HRQnd3f9HfH5cJAh7b9grNDTW0N9XMyjrlzMxWG0o01H7RmSyQwz6VtBXYCGBmVwDb\n8+Y9D6w2swVmVkv2NNJjIddXlN2vHKNvYJT1q9uJx3UaSUTKW9hHDJuBG83sUSAG3GZmm4Bmd7/L\nzH4PuI9sYP2Du+8Pub6iPJUbe+ES3e0sIhUg1GBw9wzwkQmTX8ib/13gu2HWNBue3tlDbU2c1y/X\n2AsiUv50g9sMvXp4gFcPD3LhioXU1hTfCS4iUioUDDPUuXP8oXm621lEKoOCYYY6d3QTj8VYt0rB\nICKVQcEwA0ePj7DrlWOsOXcezQ0ae0FEKoOCYQae7uohQGMviEhlUTDMwImH5ql/QUQqiIKhSEMj\nKZ7fe4RzFzXTPr8h6nJERGaNgqFIz+4+QiqtsRdEpPIoGIo0PoTnJRp7QUQqjIKhCKl0hmdePMzC\n1nrOXdQcdTkiIrNKwVAE33eUoZEUG1Zr7AURqTwKhiJ05h6apyE8RaQSKRimKQgCOnf20FSfZM25\nU44jJCJSdhQM07TnQD+9/SOsW9VOIq5fn4hUHu3ZpunEaSRdpioiFUrBME2dO3uoSca5cMXCqEsR\nEZkTCoZpONg7yP7uAS5YvoC6Wo29ICKVScEwDXo2kohUAwXDNHTu7CYGGntBRCqaguEMHRsYpWt/\nH6vOmUdrU23U5YiIzJlkmBszswbgbmAR0A98wN27T7FcB7AVuNjdh8Os8XSe6eohCDT2gohUvrCP\nGH4H2O7uvwR8Ffj0xAXM7GbgfmBJyLVN6sTYzmt0GklEKlvYwXA1sCX3/vvADadYJpObfiSsoqYy\nMprmuT1HWNrexOK2xqjLERGZU3N2KsnMPgR8bMLkg0Bf7n0/8JpnSrj7A7nvn9F22toaSSaLv3S0\no6NlymUe2/4KY6kMb1q/9IyWl3CpTcqb2q/0zFkwuPuXgC/lTzOzbwHj/wpagKMz3U5v72DR3+3o\naKG7u3/K5R76+UsA2NLWM1pewnOmbSilSe0XnckCOexTSVuBjbn3bwV+EvL2py2dyfBMVw9tLXUs\nX6K/bESk8oV6VRLw98BXzOwRYBTYBGBmvwd0ufu9IdczpR0v9TEwnOKy1y/W2AsiUhVCDQZ3HwRu\nPcX0vzzFtOVh1DSV8YfmXaLLVEWkSugGt0kEQUDnjh4a6pLYefOjLkdEJBQKhkm8dOg4h48Ns+78\nhSQT+lWJSHXQ3m4ST+3InkZar4fmiUgVUTBM4umdPSQTMS5aqbEXRKR6KBhOo+foEPsOHWftsgU0\n1IV98ZaISHQUDKehZyOJSLVSMJzG+GWq6zX2gohUGQXDKRwfGmPHS32cf3Yr85vroi5HRCRUCoZT\neKarh0wQsGGNbmoTkeqjYDiFE/0LukxVRKqQgmGC0bE0z+4+zJIFjZy1sCnqckREQqdgmOAXe3oZ\nHcvoaiQRqVoKhgme0kPzRKTKKRjyZDIBT+/sYV5TLSvObo26HBGRSCgY8nTt7+P40BjrV7cT19gL\nIlKlFAx5xm9q26DTSCJSxRQMOeNjL9TVJli7rC3qckREIqNgyNnfM8Cho0NctHIhNUn9WkSkemkP\nmNO5Y/xqJF2mKiLVTcGQ07mzh0Q8xsXna+wFEaluoQ40YGYNwN3AIqAf+IC7d09Y5mPAr+c+/pu7\n/9Fc13Xk2DB7DvRzwfI2Gutr5npzIiIlLewjht8Btrv7LwFfBT6dP9PMVgK/AVwFXAHcZGYXz3VR\n489GWq+rkUREQg+Gq4EtufffB26YMP8l4C3unnb3AKgBhue6qJOXqap/QURkzk4lmdmHgI9NmHwQ\n6Mu97wfm5c909zGgx8xiwJ8Dne6+Y7LttLU1kkwmiq6zobke33eUVefOx87XEUM56uhoiboEmQG1\nX+mZs2Bw9y8BX8qfZmbfAsb/FbQARyd+z8zqgX8gGxy/O9V2ensHi66xo6OFB5/YQzoTcNGKBXR3\n9xe9LolGR0eL2q2Mqf2iM1kghz3K/VZgI/BT4K3AT/Jn5o4UvgP8yN0/F0ZBT2nsBRGRAmEHw98D\nXzGzR4BRYBOAmf0e0AUkgGuBOjN7a+47n3T3x+aimNGxNNt3HWbR/AaWtmvsBRERCDkY3H0QuPUU\n0/8y72N9WPVs6+phZDTNhvXtxPTQPBERoMpvcHv82VcBPTRPRCRf1QZDJgh44rkDtDTWsGrpvKm/\nICJSJao2GHa9coyj/SOsW9VOPK7TSCIi46o2GH6x+wigITxFRCYK+6qkknHxqoXEkgkuXLkg6lJE\nREpK1QbD8iWtvPGipbq5RkRkgqo9lSQiIqemYBARkQIKBhERKaBgEBGRAgoGEREpoGAQEZECCgYR\nESmgYBARkQKxIAiirkFEREqIjhhERKSAgkFERAooGEREpICCQURECigYRESkgIJBREQKKBhERKRA\n1Q7UcypmdhXw27mP/9ndj0ZZjxTHzK4HNrn77VHXItNjZr8M/DrQCHze3Z+JuKSqpCOGQh8mGwxf\nAn4t4lqkCGa2CtgA1EddixSlkez/h38B3BRxLVVLwVAo4e7DwKvAWVEXI9Pn7l3ufkfUdUhx3P27\nZMPhPwFfibicqqVgKDRoZnVkQ+FA1MWIVBszawf+N/AZdz8UdT3Vqmr6GMzscuBz7n6dmcWBO4F1\nwAhwu7t3AXcBXwRqONnXICXiDNtQStQZtt9fAh3An5nZt939m9FVXL2qIhjM7L8BvwkM5CbdAtS7\n+5VmdgVwB/AOd38S+GA0VcpkzrQNx5d39/eFX6WczjT+H3x/VDXKSdVyKulF4F15n68GtgC4++PA\npVEUJdOiNixvar8yUhXB4O7/CozlTWoF+vI+p82sKo6eypXasLyp/cpLVQTDKRwDWvI+x909FVUx\nUhS1YXlT+5Wwag2GrcBGgNz5ze3RliNFUBuWN7VfCavWQ7fNwI1m9igQA26LuB6ZPrVheVP7lTAN\n7SkiIgWq9VSSiIichoJBREQKKBhERKSAgkFERAooGEREpICCQURECigYRESkgIJBZAbM7DIz+1zu\n/b8zsz+ezXWKRKFa73wWmS2vBxYDuPu9wL2zuU6RKOjOZ6l4ZnYd8ClgEFhL9rk8m9x99DTLvwX4\nY7IDNu0G/r27HzazvwBuBNLAd4AvANuAZrLjCewHrnP3D5rZHuCfgbcBqdz2Pw6sBj7u7veY2YVk\nRytrBhbl1vHVCev8M+CvgV8GAuBr7v653H/T54EE8Gzue5/PLdMLvNfde2b2m5NqpVNJUi2uAv4j\n2WA4D7j5VAuZWQfwWeBmd98A3Ad8zsyWAW9193W5da0GhoHPAPe6+5+eYnWvuPsFwFPAJ8gObv8+\n4JO5+bcDf+LubwTeDPypux+dsM6PAOcCFwOXAe82s1/JfX8NcL27fwD4NPARd78U+C5wSRG/IxFA\nwSDV41l3f9ndM8DzwILTLHc52eB40MyeJhsmq8keDQyZ2VbgY8Cn3X14im1+P/e6F3g491jpvUBb\nbvrHgXoz+yTwp2SPEia6Hviyu6fdfRD4OtmjBwB39/ExDe4FNpvZ3wLPu/v9U9QmcloKBqkW+Tvx\ngOwTPU8lATzi7uvdfT3wRuA9uZ365cAfAAuBx8xszRTbzD9VdaqxBu4B3gn8guypplOZ+P9ojJN9\ng0PjE939r4DrgC7g82b2P6aoTeS0FAwihZ4Arszb6f8B8OdmtgF4GPixu/8+2Z25kd3hF3sRx43A\nZ9z9O8C1AGaWmLDOHwEfMLOEmTUCvwE8OHFFZvYE0OLufw38FTqVJDOgYBDJ4+4HgN8C7jGz7WR3\nsB93907gMeBZM3sK2EP2VNFPgSvM7LNFbO5/Ao/k1ndzbp0rJqzzi8DLwDNAJ9m+h82nWNengC+b\n2ZPAh4H4QR3qAAAATUlEQVQ/LKIeEUBXJYmIyAS6j0Gqjpk1kP3r/1Q+k7sfQaRq6YhBREQKqI9B\nREQKKBhERKSAgkFERAooGEREpICCQURECigYRESkwP8H0R1BMru2oLgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11aa35f60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot((param_grid[\"n_estimators\"]), [elem[1] for elem in gs.grid_scores_])\n",
    "plt.xscale(\"log\")\n",
    "plt.xlabel(\"n_estimators\")\n",
    "plt.ylabel(\"R2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Качество, как и ожидается, растет с увеличением числа базовых алгоритмов. Для валидации выберем 100 деревьев (быстро обучается и уже более-менее приналежит области стабильности качества), финальную модель будем обучать с 500 деревьями."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# базовая логарифмическая шкала для max_features\n",
    "param_grid = {\"max_features\":2**np.arange(1, 9)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# базовая шкала для остальных гиперпараметров\n",
    "param_grid = {\"max_depth\":np.arange(1, 9, 2), \"min_samples_leaf\":[1, 3, 9, 27], \"min_samples_split\":[3, 9, 27, 81]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Теперь настроим градиентный бустинг, используя ту же выборку x1, y1.\n",
    "\n",
    "1. Запустите кросс-валидацию для GradentBoostingRegressor по n_estimators и learning_rate. Постройте двумерный график или несколько одномерных (например, график качество-число деревьев при разных значениях длины шага). Выберите значения для использования при дальнейшей настройке и для финального обучения модели.\n",
    "1. Постройте график зависимости качества от max_depth на кросс-валидации.\n",
    "1. Выберите оптимальные значения для min_samples_split, min_samples_leaf, max_features с помощью кросс-валидации.\n",
    "1. Обучите итоговую модель с выбранными значениями гиперпараметров на всей выборке x1, y1.\n",
    "\n",
    "Примеры построения трехмерных графиков [здесь](https://matplotlib.org/mpl_toolkits/mplot3d/tutorial.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# базовая шкала для числа деревьев и длины шага\n",
    "param_grid = {\"n_estimators\":2**np.arange(10), \"learning_rate\":0.1**np.arange(5)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# базовая шкала для max_depth\n",
    "param_grid = {\"max_depth\":np.arange(1, 9)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# базовая шкала для остальных гиперпараметров\n",
    "param_grid = {\"max_features\":2**np.arange(1, 9, 2), \"min_samples_leaf\":[1, 3, 9, 27], \n",
    "              \"min_samples_split\":[3, 9, 27, 81]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Выведите качество случайного леса и градиентного бустинга на x2, y2, а также качество модели, представляющей собой усреднение предсказаний двух обученных моделей."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Реализуйте блендинг. Обучите линейную регрессию на признаках - выходах двух моделей по выборке x2, y2. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Выведите качество случайного леса и градиентного бустинга на x3, y3, качество модели, представляющей собой усреднение предсказаний двух обученных моделей, а также качество блендинга."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Что делать дальше (на выбор)\n",
    "* Построить двухмерный график для бустинга :)\n",
    "* Подобрать гиперпараметры более точно. Мы перебрали параметры только по грубой логарифмической шкале. На практике же перебор по более мелкой шкале может дать серьезный выигрыш в качестве. Сравнить итоговое качество на x2, y2 и x3, y3.\n",
    "* Подобрать гиперпараметры для xgboost. Сравнить итоговое качество.\n",
    "* Подобрать гиперпараметры для других моделей (например, линейной и метрической)\n",
    "* Почитать на форуме соревнования, как его решали, и посмотреть Kernels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "_change_revision": 113,
  "_is_fork": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}