Final_Submission.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](ltfs.jpg)\n",
    "\n",
    "# LTFS Data Science FinHack ( ML Hackathon)\n",
    "\n",
    "## L&T Financial Services & Analytics Vidhya presents **‘DataScience FinHack’**.\n",
    "\n",
    "In this FinHack, you will develop a model for our most common but real challenge **‘Loan Default Prediction’ & also, get a feel of our business!**\n",
    "\n",
    "If your solution adds good value to our organization, take it from us, Sky is the limit for you!\n",
    "\n",
    "## Problem Statement\n",
    "\n",
    "### Vehicle Loan Default Prediction\n",
    "\n",
    "Financial institutions incur significant losses due to the default of vehicle loans. This has led to the tightening up of vehicle loan underwriting and increased vehicle loan rejection rates. The need for a better credit risk scoring model is also raised by these institutions. This warrants a study to estimate the determinants of vehicle loan default.\n",
    "A financial institution has hired you to accurately predict the probability of loanee/borrower defaulting on a vehicle loan in the first EMI (Equated Monthly Instalments) on the due date. Following Information regarding the loan and loanee are provided in the datasets:\n",
    "\n",
    "* **Loanee Information (Demographic data like age, income, Identity proof etc.)**\n",
    "* **Loan Information (Disbursal details, amount, EMI, loan to value ratio etc.)**\n",
    "* **Bureau data & history (Bureau score, number of active accounts, the status of other loans, credit history etc.)**\n",
    "\n",
    "Doing so will ensure that clients capable of repayment are not rejected and important determinants can be identified which can be further used for minimising the default rates.\n",
    " \n",
    "\n",
    "### Data Description\n",
    "\n",
    "* train.zip contains train.csv and data_dictionary.csv.\n",
    "* train.csv contains the training data with details on loan as described in the last section\n",
    "* data_dictionary.csv contains a brief description on each variable provided in the training and test set.\n",
    "* test.csv contains details of all customers and loans for which the participants are to submit probability of default.\n",
    " \n",
    "\n",
    "\n",
    "* sample_submission.csv contains the submission format for the predictions against the test set. A single csv needs to be submitted as a solution.\n",
    " \n",
    "\n",
    "### Evaluation Metric\n",
    "\n",
    "Submissions are evaluated on **area under the ROC curve** between the predicted probability and the observed target.\n",
    " \n",
    "\n",
    "### Public and Private Split\n",
    "Test data is further randomly divided into **Public (25%) and Private (75%)** data.\n",
    "Your initial responses will be checked and scored on the Public data.\n",
    "The final rankings would be based on your private score which will be published once the competition is over."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UniqueID</th>\n",
       "      <th>disbursed_amount</th>\n",
       "      <th>asset_cost</th>\n",
       "      <th>ltv</th>\n",
       "      <th>branch_id</th>\n",
       "      <th>supplier_id</th>\n",
       "      <th>manufacturer_id</th>\n",
       "      <th>Current_pincode_ID</th>\n",
       "      <th>Date.of.Birth</th>\n",
       "      <th>Employment.Type</th>\n",
       "      <th>...</th>\n",
       "      <th>SEC.SANCTIONED.AMOUNT</th>\n",
       "      <th>SEC.DISBURSED.AMOUNT</th>\n",
       "      <th>PRIMARY.INSTAL.AMT</th>\n",
       "      <th>SEC.INSTAL.AMT</th>\n",
       "      <th>NEW.ACCTS.IN.LAST.SIX.MONTHS</th>\n",
       "      <th>DELINQUENT.ACCTS.IN.LAST.SIX.MONTHS</th>\n",
       "      <th>AVERAGE.ACCT.AGE</th>\n",
       "      <th>CREDIT.HISTORY.LENGTH</th>\n",
       "      <th>NO.OF_INQUIRIES</th>\n",
       "      <th>loan_default</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>420825</td>\n",
       "      <td>50578</td>\n",
       "      <td>58400</td>\n",
       "      <td>89.55</td>\n",
       "      <td>67</td>\n",
       "      <td>22807</td>\n",
       "      <td>45</td>\n",
       "      <td>1441</td>\n",
       "      <td>01-01-84</td>\n",
       "      <td>Salaried</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>537409</td>\n",
       "      <td>47145</td>\n",
       "      <td>65550</td>\n",
       "      <td>73.23</td>\n",
       "      <td>67</td>\n",
       "      <td>22807</td>\n",
       "      <td>45</td>\n",
       "      <td>1502</td>\n",
       "      <td>31-07-85</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1991</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1yrs 11mon</td>\n",
       "      <td>1yrs 11mon</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>417566</td>\n",
       "      <td>53278</td>\n",
       "      <td>61360</td>\n",
       "      <td>89.63</td>\n",
       "      <td>67</td>\n",
       "      <td>22807</td>\n",
       "      <td>45</td>\n",
       "      <td>1497</td>\n",
       "      <td>24-08-85</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>624493</td>\n",
       "      <td>57513</td>\n",
       "      <td>66113</td>\n",
       "      <td>88.48</td>\n",
       "      <td>67</td>\n",
       "      <td>22807</td>\n",
       "      <td>45</td>\n",
       "      <td>1501</td>\n",
       "      <td>30-12-93</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0yrs 8mon</td>\n",
       "      <td>1yrs 3mon</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>539055</td>\n",
       "      <td>52378</td>\n",
       "      <td>60300</td>\n",
       "      <td>88.39</td>\n",
       "      <td>67</td>\n",
       "      <td>22807</td>\n",
       "      <td>45</td>\n",
       "      <td>1495</td>\n",
       "      <td>09-12-77</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 41 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   UniqueID  disbursed_amount  asset_cost    ltv  branch_id  supplier_id  \\\n",
       "0    420825             50578       58400  89.55         67        22807   \n",
       "1    537409             47145       65550  73.23         67        22807   \n",
       "2    417566             53278       61360  89.63         67        22807   \n",
       "3    624493             57513       66113  88.48         67        22807   \n",
       "4    539055             52378       60300  88.39         67        22807   \n",
       "\n",
       "   manufacturer_id  Current_pincode_ID Date.of.Birth Employment.Type  ...  \\\n",
       "0               45                1441      01-01-84        Salaried  ...   \n",
       "1               45                1502      31-07-85   Self employed  ...   \n",
       "2               45                1497      24-08-85   Self employed  ...   \n",
       "3               45                1501      30-12-93   Self employed  ...   \n",
       "4               45                1495      09-12-77   Self employed  ...   \n",
       "\n",
       "  SEC.SANCTIONED.AMOUNT  SEC.DISBURSED.AMOUNT  PRIMARY.INSTAL.AMT  \\\n",
       "0                     0                     0                   0   \n",
       "1                     0                     0                1991   \n",
       "2                     0                     0                   0   \n",
       "3                     0                     0                  31   \n",
       "4                     0                     0                   0   \n",
       "\n",
       "   SEC.INSTAL.AMT  NEW.ACCTS.IN.LAST.SIX.MONTHS  \\\n",
       "0               0                             0   \n",
       "1               0                             0   \n",
       "2               0                             0   \n",
       "3               0                             0   \n",
       "4               0                             0   \n",
       "\n",
       "   DELINQUENT.ACCTS.IN.LAST.SIX.MONTHS  AVERAGE.ACCT.AGE  \\\n",
       "0                                    0         0yrs 0mon   \n",
       "1                                    1        1yrs 11mon   \n",
       "2                                    0         0yrs 0mon   \n",
       "3                                    0         0yrs 8mon   \n",
       "4                                    0         0yrs 0mon   \n",
       "\n",
       "   CREDIT.HISTORY.LENGTH  NO.OF_INQUIRIES  loan_default  \n",
       "0              0yrs 0mon                0             0  \n",
       "1             1yrs 11mon                0             1  \n",
       "2              0yrs 0mon                0             0  \n",
       "3              1yrs 3mon                1             1  \n",
       "4              0yrs 0mon                1             1  \n",
       "\n",
       "[5 rows x 41 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train=pd.read_csv('train.csv')\n",
    "test=pd.read_csv('test.csv')\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combining Train and Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Acer\\Anaconda3\\lib\\site-packages\\pandas\\core\\frame.py:6692: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version\n",
      "of pandas will change to not sort by default.\n",
      "\n",
      "To accept the future behavior, pass 'sort=False'.\n",
      "\n",
      "To retain the current behavior and silence the warning, pass 'sort=True'.\n",
      "\n",
      "  sort=sort)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AVERAGE.ACCT.AGE</th>\n",
       "      <th>Aadhar_flag</th>\n",
       "      <th>CREDIT.HISTORY.LENGTH</th>\n",
       "      <th>Current_pincode_ID</th>\n",
       "      <th>DELINQUENT.ACCTS.IN.LAST.SIX.MONTHS</th>\n",
       "      <th>Date.of.Birth</th>\n",
       "      <th>DisbursalDate</th>\n",
       "      <th>Driving_flag</th>\n",
       "      <th>Employee_code_ID</th>\n",
       "      <th>Employment.Type</th>\n",
       "      <th>...</th>\n",
       "      <th>State_ID</th>\n",
       "      <th>UniqueID</th>\n",
       "      <th>VoterID_flag</th>\n",
       "      <th>asset_cost</th>\n",
       "      <th>branch_id</th>\n",
       "      <th>disbursed_amount</th>\n",
       "      <th>loan_default</th>\n",
       "      <th>ltv</th>\n",
       "      <th>manufacturer_id</th>\n",
       "      <th>supplier_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>345541</th>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>1</td>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>2591</td>\n",
       "      <td>0</td>\n",
       "      <td>14-05-92</td>\n",
       "      <td>28-11-18</td>\n",
       "      <td>0</td>\n",
       "      <td>3823</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>755547</td>\n",
       "      <td>0</td>\n",
       "      <td>72509</td>\n",
       "      <td>74</td>\n",
       "      <td>44875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>67.58</td>\n",
       "      <td>86</td>\n",
       "      <td>16646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>345542</th>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>1</td>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>2593</td>\n",
       "      <td>0</td>\n",
       "      <td>24-04-84</td>\n",
       "      <td>19-11-18</td>\n",
       "      <td>0</td>\n",
       "      <td>3823</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>716994</td>\n",
       "      <td>0</td>\n",
       "      <td>72509</td>\n",
       "      <td>74</td>\n",
       "      <td>53549</td>\n",
       "      <td>NaN</td>\n",
       "      <td>78.61</td>\n",
       "      <td>86</td>\n",
       "      <td>16646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>345543</th>\n",
       "      <td>4yrs 10mon</td>\n",
       "      <td>1</td>\n",
       "      <td>9yrs 5mon</td>\n",
       "      <td>2593</td>\n",
       "      <td>0</td>\n",
       "      <td>01-01-86</td>\n",
       "      <td>17-11-18</td>\n",
       "      <td>0</td>\n",
       "      <td>3823</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>709352</td>\n",
       "      <td>0</td>\n",
       "      <td>72509</td>\n",
       "      <td>74</td>\n",
       "      <td>55243</td>\n",
       "      <td>NaN</td>\n",
       "      <td>81.37</td>\n",
       "      <td>86</td>\n",
       "      <td>16646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>345544</th>\n",
       "      <td>0yrs 3mon</td>\n",
       "      <td>1</td>\n",
       "      <td>0yrs 3mon</td>\n",
       "      <td>2593</td>\n",
       "      <td>1</td>\n",
       "      <td>01-01-73</td>\n",
       "      <td>15-11-18</td>\n",
       "      <td>0</td>\n",
       "      <td>3823</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>699795</td>\n",
       "      <td>0</td>\n",
       "      <td>72509</td>\n",
       "      <td>74</td>\n",
       "      <td>55243</td>\n",
       "      <td>NaN</td>\n",
       "      <td>81.37</td>\n",
       "      <td>86</td>\n",
       "      <td>16646</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>345545</th>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>1</td>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>2593</td>\n",
       "      <td>0</td>\n",
       "      <td>01-01-74</td>\n",
       "      <td>28-11-18</td>\n",
       "      <td>0</td>\n",
       "      <td>3823</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>...</td>\n",
       "      <td>4</td>\n",
       "      <td>755244</td>\n",
       "      <td>0</td>\n",
       "      <td>72509</td>\n",
       "      <td>74</td>\n",
       "      <td>40274</td>\n",
       "      <td>NaN</td>\n",
       "      <td>60.68</td>\n",
       "      <td>86</td>\n",
       "      <td>16646</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 41 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       AVERAGE.ACCT.AGE  Aadhar_flag CREDIT.HISTORY.LENGTH  \\\n",
       "345541        0yrs 0mon            1             0yrs 0mon   \n",
       "345542        0yrs 0mon            1             0yrs 0mon   \n",
       "345543       4yrs 10mon            1             9yrs 5mon   \n",
       "345544        0yrs 3mon            1             0yrs 3mon   \n",
       "345545        0yrs 0mon            1             0yrs 0mon   \n",
       "\n",
       "        Current_pincode_ID  DELINQUENT.ACCTS.IN.LAST.SIX.MONTHS Date.of.Birth  \\\n",
       "345541                2591                                    0      14-05-92   \n",
       "345542                2593                                    0      24-04-84   \n",
       "345543                2593                                    0      01-01-86   \n",
       "345544                2593                                    1      01-01-73   \n",
       "345545                2593                                    0      01-01-74   \n",
       "\n",
       "       DisbursalDate  Driving_flag  Employee_code_ID Employment.Type  ...  \\\n",
       "345541      28-11-18             0              3823   Self employed  ...   \n",
       "345542      19-11-18             0              3823   Self employed  ...   \n",
       "345543      17-11-18             0              3823   Self employed  ...   \n",
       "345544      15-11-18             0              3823   Self employed  ...   \n",
       "345545      28-11-18             0              3823   Self employed  ...   \n",
       "\n",
       "        State_ID  UniqueID  VoterID_flag  asset_cost  branch_id  \\\n",
       "345541         4    755547             0       72509         74   \n",
       "345542         4    716994             0       72509         74   \n",
       "345543         4    709352             0       72509         74   \n",
       "345544         4    699795             0       72509         74   \n",
       "345545         4    755244             0       72509         74   \n",
       "\n",
       "       disbursed_amount  loan_default    ltv  manufacturer_id  supplier_id  \n",
       "345541            44875           NaN  67.58               86        16646  \n",
       "345542            53549           NaN  78.61               86        16646  \n",
       "345543            55243           NaN  81.37               86        16646  \n",
       "345544            55243           NaN  81.37               86        16646  \n",
       "345545            40274           NaN  60.68               86        16646  \n",
       "\n",
       "[5 rows x 41 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df=train.append(test,ignore_index=True)\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Engineering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "1\n",
      "1\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 345546 entries, 0 to 345545\n",
      "Data columns (total 84 columns):\n",
      "AVERAGE.ACCT.AGE                       345546 non-null object\n",
      "Aadhar_flag                            345546 non-null object\n",
      "CREDIT.HISTORY.LENGTH                  345546 non-null object\n",
      "Current_pincode_ID                     345546 non-null object\n",
      "DELINQUENT.ACCTS.IN.LAST.SIX.MONTHS    345546 non-null int64\n",
      "Date.of.Birth                          345546 non-null datetime64[ns]\n",
      "DisbursalDate                          345546 non-null datetime64[ns]\n",
      "Driving_flag                           345546 non-null object\n",
      "Employee_code_ID                       345546 non-null object\n",
      "Employment.Type                        345546 non-null object\n",
      "MobileNo_Avl_Flag                      345546 non-null int64\n",
      "NEW.ACCTS.IN.LAST.SIX.MONTHS           345546 non-null int64\n",
      "NO.OF_INQUIRIES                        345546 non-null int64\n",
      "PAN_flag                               345546 non-null object\n",
      "PERFORM_CNS.SCORE                      345546 non-null int64\n",
      "PERFORM_CNS.SCORE.DESCRIPTION          345546 non-null object\n",
      "PRI.ACTIVE.ACCTS                       345546 non-null int64\n",
      "PRI.CURRENT.BALANCE                    345546 non-null int64\n",
      "PRI.DISBURSED.AMOUNT                   345546 non-null int64\n",
      "PRI.NO.OF.ACCTS                        345546 non-null int64\n",
      "PRI.OVERDUE.ACCTS                      345546 non-null int64\n",
      "PRI.SANCTIONED.AMOUNT                  345546 non-null int64\n",
      "PRIMARY.INSTAL.AMT                     345546 non-null int64\n",
      "Passport_flag                          345546 non-null object\n",
      "SEC.ACTIVE.ACCTS                       345546 non-null int64\n",
      "SEC.CURRENT.BALANCE                    345546 non-null int64\n",
      "SEC.DISBURSED.AMOUNT                   345546 non-null int64\n",
      "SEC.INSTAL.AMT                         345546 non-null int64\n",
      "SEC.NO.OF.ACCTS                        345546 non-null int64\n",
      "SEC.OVERDUE.ACCTS                      345546 non-null int64\n",
      "SEC.SANCTIONED.AMOUNT                  345546 non-null int64\n",
      "State_ID                               345546 non-null object\n",
      "UniqueID                               345546 non-null int64\n",
      "VoterID_flag                           345546 non-null object\n",
      "asset_cost                             345546 non-null int64\n",
      "branch_id                              345546 non-null object\n",
      "disbursed_amount                       345546 non-null int64\n",
      "loan_default                           233154 non-null float64\n",
      "ltv                                    345546 non-null float64\n",
      "manufacturer_id                        345546 non-null object\n",
      "supplier_id                            345546 non-null object\n",
      "age                                    345546 non-null int64\n",
      "acct_age                               345546 non-null int64\n",
      "credit_history_len                     345546 non-null int64\n",
      "credit_history_len_before_sixM         345546 non-null int64\n",
      "dob_y                                  345546 non-null int64\n",
      "dob_m                                  345546 non-null int64\n",
      "dob_d                                  345546 non-null int64\n",
      "dob_w                                  345546 non-null int64\n",
      "disb_m                                 345546 non-null int64\n",
      "disb_d                                 345546 non-null int64\n",
      "disb_w                                 345546 non-null int64\n",
      "PERFORM_CNS.SCORE.DESCRIPTION.APLHS    345546 non-null object\n",
      "PERFORM_CNS.SCORE.BINS                 345546 non-null object\n",
      "ltv.BINS                               345546 non-null object\n",
      "Not_Scored                             345546 non-null int32\n",
      "Very_Low                               345546 non-null int32\n",
      "Very_High                              345546 non-null int32\n",
      "No_History                             345546 non-null int32\n",
      "ACTIVE.ACCTS                           345546 non-null int64\n",
      "CURRENT.BALANCE                        345546 non-null int64\n",
      "DISBURSED.AMOUNT                       345546 non-null int64\n",
      "NO.OF.ACCTS                            345546 non-null int64\n",
      "OVERDUE.ACCTS                          345546 non-null int64\n",
      "SANCTIONED.AMOUNT                      345546 non-null int64\n",
      "INSTAL.AMT                             345546 non-null int64\n",
      "SANCTION_DISBURSED                     345546 non-null int64\n",
      "NO_DEACTIVE_ACCOUNTS                   345546 non-null int64\n",
      "NO.OF.ACC.BEF.SIX.MONTH                345546 non-null int64\n",
      "OVERDUE.ACC.BEF.SIX.MONTHS             345546 non-null int64\n",
      "CLEAN.ACC                              345546 non-null int64\n",
      "asset_value                            345546 non-null float64\n",
      "value_cost                             345546 non-null float64\n",
      "value_per_cost                         345546 non-null float64\n",
      "extra_finance                          345546 non-null float64\n",
      "asset_disburse                         345546 non-null float64\n",
      "sixmmonths_dfault                      345546 non-null int64\n",
      "AVERAGE.ACCT.AGE.YEAR                  345546 non-null int64\n",
      "AVERAGE.ACCT.AGE.MONTH                 345546 non-null int64\n",
      "CREDIT.HISTORY.LENGTH.YEAR             345546 non-null int64\n",
      "CREDIT.HISTORY.LENGTH.MONTH            345546 non-null int64\n",
      "Total_Flag                             345546 non-null object\n",
      "Branch_Emp_code                        345546 non-null object\n",
      "Pin_Emp_code                           345546 non-null object\n",
      "dtypes: datetime64[ns](2), float64(7), int32(4), int64(50), object(21)\n",
      "memory usage: 216.2+ MB\n"
     ]
    }
   ],
   "source": [
    "def calculateage(v):\n",
    "    return pd.to_datetime('today').year-pd.to_datetime(v).year\n",
    "df['age']=df['Date.of.Birth'].apply(calculateage)\n",
    "df['acct_age']=df['AVERAGE.ACCT.AGE'].apply(lambda x: int(x.split(\" \")[0][:-3])*12+int(x.split(\" \")[1][:-3]))\n",
    "df['credit_history_len']=df['CREDIT.HISTORY.LENGTH'].apply(lambda x: int(x.split(\" \")[0][:-3])*12+int(x.split(\" \")[1][:-3]))\n",
    "df['credit_history_len_before_sixM']=df['credit_history_len'].apply(lambda x: x if x-6<0 else x-6)\n",
    "def d(v):\n",
    "    if v.split(\"-\")[-1]=='00' or v.split(\"-\")[-1]=='18':\n",
    "        return \"-\".join(v.split(\"-\")[:-1])+'-20'+\"\".join(v.split(\"-\")[-1])\n",
    "    else:\n",
    "        return \"-\".join(v.split(\"-\")[:-1])+'-19'+\"\".join(v.split(\"-\")[-1])\n",
    "df['Date.of.Birth']=pd.to_datetime(df['Date.of.Birth'].apply(d),format='%d-%m-%Y',errors='coerce')\n",
    "print(1)\n",
    "df['dob_y']=df['Date.of.Birth'].dt.year\n",
    "df['dob_m']=df['Date.of.Birth'].dt.month\n",
    "df['dob_d']=df['Date.of.Birth'].dt.day\n",
    "df['dob_w']=df['Date.of.Birth'].dt.week\n",
    "df['DisbursalDate']=pd.to_datetime(df['DisbursalDate'].apply(d),format='%d-%m-%Y',errors='coerce')\n",
    "print(1)\n",
    "df['disb_m']=df['DisbursalDate'].dt.month\n",
    "df['disb_d']=df['DisbursalDate'].dt.day\n",
    "df['disb_w']=df['DisbursalDate'].dt.week\n",
    "df['Employment.Type'].fillna('Other',inplace=True)\n",
    "print(1)\n",
    "def alphs(v):\n",
    "    vv=v.split(\"-\")\n",
    "    if len(vv)!=1:\n",
    "        return vv[0]\n",
    "    else:\n",
    "        return 'Z'\n",
    "df['PERFORM_CNS.SCORE.DESCRIPTION.APLHS']=df['PERFORM_CNS.SCORE.DESCRIPTION'].apply(alphs).astype(np.object)\n",
    "df['PERFORM_CNS.SCORE.BINS']=pd.cut(df['PERFORM_CNS.SCORE'],5,labels=['Slim','Doubtful','Fair','Good','Great']).astype(np.object)\n",
    "df['ltv.BINS']=pd.cut(df['ltv'],3,labels=['Low','Medium','High']).astype(np.object)\n",
    "\n",
    "df['PERFORM_CNS.SCORE.DESCRIPTION'].replace({'C-Very Low Risk':'Very Low Risk','A-Very Low Risk':'Very Low Risk','D-Very Low Risk':'Very Low Risk','B-Very Low Risk':'Very Low Risk',\n",
    "'M-Very High Risk':'Very High Risk','L-Very High Risk':'Very High Risk',\n",
    "                                               'F-Low Risk':'Low Risk','E-Low Risk':'Low Risk',\n",
    "                                               'H-Medium Risk':'Medium Risk','I-Medium Risk':'Medium Risk',\n",
    "                                               'J-High Risk':'High Risk','K-High Risk':'High Risk'},inplace=True)\n",
    "\n",
    "df['Not_Scored']=np.where(df['PERFORM_CNS.SCORE.DESCRIPTION'].str.contains('Not Scored'),1,0)\n",
    "df['Very_Low']=np.where(df['PERFORM_CNS.SCORE.DESCRIPTION'].str.contains('Very Low'),1,0)\n",
    "df['Very_High']=np.where(df['PERFORM_CNS.SCORE.DESCRIPTION'].str.contains('Very High'),1,0)\n",
    "df['No_History']=np.where(df['PERFORM_CNS.SCORE.DESCRIPTION'].str.contains('No Bureau'),1,0)\n",
    "\n",
    "df['ACTIVE.ACCTS']=df['PRI.ACTIVE.ACCTS']+df['SEC.ACTIVE.ACCTS']\n",
    "df['CURRENT.BALANCE']=df['PRI.CURRENT.BALANCE']+df['SEC.CURRENT.BALANCE']\n",
    "df['DISBURSED.AMOUNT']=df['PRI.DISBURSED.AMOUNT']+df['SEC.DISBURSED.AMOUNT']\n",
    "df['NO.OF.ACCTS']=df['SEC.NO.OF.ACCTS']+df['PRI.NO.OF.ACCTS']\n",
    "df['OVERDUE.ACCTS']=df['PRI.OVERDUE.ACCTS']+df['SEC.OVERDUE.ACCTS']\n",
    "df['SANCTIONED.AMOUNT']=df['PRI.SANCTIONED.AMOUNT']+df['SEC.SANCTIONED.AMOUNT']\n",
    "df['INSTAL.AMT']=df['PRIMARY.INSTAL.AMT']+df['SEC.INSTAL.AMT']\n",
    "\n",
    "df['SANCTION_DISBURSED']=df['SANCTIONED.AMOUNT']-df['DISBURSED.AMOUNT']\n",
    "\n",
    "df['NO_DEACTIVE_ACCOUNTS']=df['NO.OF.ACCTS']-df['ACTIVE.ACCTS']\n",
    "df['NO.OF.ACC.BEF.SIX.MONTH']=df['NO.OF.ACCTS']-df['NEW.ACCTS.IN.LAST.SIX.MONTHS']\n",
    "df['OVERDUE.ACC.BEF.SIX.MONTHS']=df['OVERDUE.ACCTS']-df['DELINQUENT.ACCTS.IN.LAST.SIX.MONTHS']\n",
    "df['CLEAN.ACC']=df['NO.OF.ACCTS']-(df['ACTIVE.ACCTS']+df['OVERDUE.ACCTS'])\n",
    "\n",
    "\n",
    "df['asset_value']=df['disbursed_amount']*(df['ltv']/100)\n",
    "df['value_cost']=df['asset_cost']-df['asset_value']\n",
    "df['value_per_cost']=df['value_cost']/df['asset_value']\n",
    "df['extra_finance']=df['asset_cost']*(df['ltv']/100)-df['disbursed_amount']\n",
    "df['asset_disburse']=(df['asset_cost']-df['disbursed_amount'])/df['disbursed_amount']\n",
    "df['sixmmonths_dfault']=df['NEW.ACCTS.IN.LAST.SIX.MONTHS']-df['DELINQUENT.ACCTS.IN.LAST.SIX.MONTHS']\n",
    "\n",
    "df['Current_pincode_ID']=df['Current_pincode_ID'].astype(np.object)\n",
    "df['Employee_code_ID']=df['Employee_code_ID'].astype(np.object)\n",
    "df['State_ID']=df['State_ID'].astype(np.object)\n",
    "df['branch_id']=df['branch_id'].astype(np.object)\n",
    "df['manufacturer_id']=df['manufacturer_id'].astype(np.object)\n",
    "df['supplier_id']=df['supplier_id'].astype(np.object)\n",
    "\n",
    "# df['NEW.ACCTS.IN.LAST.SIX.MONTHS']=df['NEW.ACCTS.IN.LAST.SIX.MONTHS'].astype(np.object)\n",
    "# df['DELINQUENT.ACCTS.IN.LAST.SIX.MONTHS']=df['DELINQUENT.ACCTS.IN.LAST.SIX.MONTHS'].astype(np.object)\n",
    "# df['NO.OF_INQUIRIES']=df['NO.OF_INQUIRIES'].astype(np.object)\n",
    "\n",
    "for i in [ 'Aadhar_flag', 'Driving_flag', 'PAN_flag', 'Passport_flag', 'VoterID_flag']:\n",
    "    df[i]=df[i].astype(np.object)\n",
    "df['AVERAGE.ACCT.AGE.YEAR']=df['AVERAGE.ACCT.AGE'].apply(lambda x: int(x.split(\" \")[0][:-3]))\n",
    "df['AVERAGE.ACCT.AGE.MONTH']=df['AVERAGE.ACCT.AGE'].apply(lambda x: int(x.split(\" \")[1][:-3]))\n",
    "\n",
    "df['CREDIT.HISTORY.LENGTH.YEAR']=df['CREDIT.HISTORY.LENGTH'].apply(lambda x: int(x.split(\" \")[0][:-3]))\n",
    "df['CREDIT.HISTORY.LENGTH.MONTH']=df['CREDIT.HISTORY.LENGTH'].apply(lambda x: int(x.split(\" \")[1][:-3]))\n",
    "df['Total_Flag']=df['Driving_flag']+df['Aadhar_flag']+df['PAN_flag']+df['Passport_flag']+df['VoterID_flag']\n",
    "df['Branch_Emp_code']=(df['Employee_code_ID']+df['branch_id']).astype(np.object)\n",
    "df['Pin_Emp_code']=(df['Employee_code_ID']+df['Current_pincode_ID']).astype(np.object)\n",
    "df.info()\n",
    "# ------------------------------------------------------\n",
    "\n",
    "\n",
    "# 2nd try remove \n",
    "# df['Very_Low']=np.where(df['PERFORM_CNS.SCORE.DESCRIPTION'].str.contains('Very Low'),1,0)\n",
    "# df['No_History']=np.where(df['PERFORM_CNS.SCORE.DESCRIPTION'].str.contains('No Bureau'),1,0)\n",
    "# df['AVERAGE.ACCT.AGE.YEAR']=df['AVERAGE.ACCT.AGE'].apply(lambda x: int(x.split(\" \")[0][:-3]))\n",
    "# df['CREDIT.HISTORY.LENGTH.YEAR']=df['CREDIT.HISTORY.LENGTH'].apply(lambda x: int(x.split(\" \")[0][:-3]))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['SEC.ACTIVE.ACCTS', 'SEC.CURRENT.BALANCE', 'SEC.DISBURSED.AMOUNT',\n",
       "       'SEC.INSTAL.AMT', 'SEC.NO.OF.ACCTS', 'SEC.OVERDUE.ACCTS',\n",
       "       'SEC.SANCTIONED.AMOUNT'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns[df.columns.str.startswith('SEC')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(345546, 84)\n"
     ]
    }
   ],
   "source": [
    "print(df.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AVERAGE.ACCT.AGE</th>\n",
       "      <th>Aadhar_flag</th>\n",
       "      <th>CREDIT.HISTORY.LENGTH</th>\n",
       "      <th>Current_pincode_ID</th>\n",
       "      <th>DELINQUENT.ACCTS.IN.LAST.SIX.MONTHS</th>\n",
       "      <th>Driving_flag</th>\n",
       "      <th>Employee_code_ID</th>\n",
       "      <th>Employment.Type</th>\n",
       "      <th>NEW.ACCTS.IN.LAST.SIX.MONTHS</th>\n",
       "      <th>NO.OF_INQUIRIES</th>\n",
       "      <th>...</th>\n",
       "      <th>extra_finance</th>\n",
       "      <th>asset_disburse</th>\n",
       "      <th>sixmmonths_dfault</th>\n",
       "      <th>AVERAGE.ACCT.AGE.YEAR</th>\n",
       "      <th>AVERAGE.ACCT.AGE.MONTH</th>\n",
       "      <th>CREDIT.HISTORY.LENGTH.YEAR</th>\n",
       "      <th>CREDIT.HISTORY.LENGTH.MONTH</th>\n",
       "      <th>Total_Flag</th>\n",
       "      <th>Branch_Emp_code</th>\n",
       "      <th>Pin_Emp_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>1</td>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>1441</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1998</td>\n",
       "      <td>Salaried</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1719.2000</td>\n",
       "      <td>0.154652</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2065</td>\n",
       "      <td>3439</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1yrs 11mon</td>\n",
       "      <td>1</td>\n",
       "      <td>1yrs 11mon</td>\n",
       "      <td>1502</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1998</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>857.2650</td>\n",
       "      <td>0.390391</td>\n",
       "      <td>-1</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>2065</td>\n",
       "      <td>3500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>1</td>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>1497</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1998</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1718.9680</td>\n",
       "      <td>0.151695</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2065</td>\n",
       "      <td>3495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0yrs 8mon</td>\n",
       "      <td>1</td>\n",
       "      <td>1yrs 3mon</td>\n",
       "      <td>1501</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1998</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>983.7824</td>\n",
       "      <td>0.149531</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2065</td>\n",
       "      <td>3499</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>1</td>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>1495</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1998</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>921.1700</td>\n",
       "      <td>0.151247</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2065</td>\n",
       "      <td>3493</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 80 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  AVERAGE.ACCT.AGE Aadhar_flag CREDIT.HISTORY.LENGTH Current_pincode_ID  \\\n",
       "0        0yrs 0mon           1             0yrs 0mon               1441   \n",
       "1       1yrs 11mon           1            1yrs 11mon               1502   \n",
       "2        0yrs 0mon           1             0yrs 0mon               1497   \n",
       "3        0yrs 8mon           1             1yrs 3mon               1501   \n",
       "4        0yrs 0mon           1             0yrs 0mon               1495   \n",
       "\n",
       "   DELINQUENT.ACCTS.IN.LAST.SIX.MONTHS Driving_flag Employee_code_ID  \\\n",
       "0                                    0            0             1998   \n",
       "1                                    1            0             1998   \n",
       "2                                    0            0             1998   \n",
       "3                                    0            0             1998   \n",
       "4                                    0            0             1998   \n",
       "\n",
       "  Employment.Type  NEW.ACCTS.IN.LAST.SIX.MONTHS  NO.OF_INQUIRIES  ...  \\\n",
       "0        Salaried                             0                0  ...   \n",
       "1   Self employed                             0                0  ...   \n",
       "2   Self employed                             0                0  ...   \n",
       "3   Self employed                             0                1  ...   \n",
       "4   Self employed                             0                1  ...   \n",
       "\n",
       "  extra_finance  asset_disburse sixmmonths_dfault  AVERAGE.ACCT.AGE.YEAR  \\\n",
       "0     1719.2000        0.154652                 0                      0   \n",
       "1      857.2650        0.390391                -1                      1   \n",
       "2     1718.9680        0.151695                 0                      0   \n",
       "3      983.7824        0.149531                 0                      0   \n",
       "4      921.1700        0.151247                 0                      0   \n",
       "\n",
       "   AVERAGE.ACCT.AGE.MONTH  CREDIT.HISTORY.LENGTH.YEAR  \\\n",
       "0                       0                           0   \n",
       "1                      11                           1   \n",
       "2                       0                           0   \n",
       "3                       8                           1   \n",
       "4                       0                           0   \n",
       "\n",
       "   CREDIT.HISTORY.LENGTH.MONTH  Total_Flag  Branch_Emp_code  Pin_Emp_code  \n",
       "0                            0           1             2065          3439  \n",
       "1                           11           1             2065          3500  \n",
       "2                            0           1             2065          3495  \n",
       "3                            3           1             2065          3499  \n",
       "4                            0           1             2065          3493  \n",
       "\n",
       "[5 rows x 80 columns]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.drop(['Date.of.Birth','DisbursalDate','UniqueID','MobileNo_Avl_Flag'],axis=1,inplace=True)\n",
    "\n",
    "#-----------------------2nd try--------------------------------\n",
    "# df.drop(['Date.of.Birth','DisbursalDate','UniqueID','MobileNo_Avl_Flag','Passport_flag','Driving_flag','PAN_flag'\n",
    "#         ,'SEC.ACTIVE.ACCTS', 'SEC.CURRENT.BALANCE', 'SEC.DISBURSED.AMOUNT',\n",
    "#        'SEC.INSTAL.AMT', 'SEC.NO.OF.ACCTS', 'SEC.OVERDUE.ACCTS'],axis=1,inplace=True)\n",
    "\n",
    "\n",
    "# df.drop(pre_sec,axis=1,inplace=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>AVERAGE.ACCT.AGE</th>\n",
       "      <th>Aadhar_flag</th>\n",
       "      <th>CREDIT.HISTORY.LENGTH</th>\n",
       "      <th>Current_pincode_ID</th>\n",
       "      <th>DELINQUENT.ACCTS.IN.LAST.SIX.MONTHS</th>\n",
       "      <th>Driving_flag</th>\n",
       "      <th>Employee_code_ID</th>\n",
       "      <th>Employment.Type</th>\n",
       "      <th>NEW.ACCTS.IN.LAST.SIX.MONTHS</th>\n",
       "      <th>NO.OF_INQUIRIES</th>\n",
       "      <th>...</th>\n",
       "      <th>extra_finance</th>\n",
       "      <th>asset_disburse</th>\n",
       "      <th>sixmmonths_dfault</th>\n",
       "      <th>AVERAGE.ACCT.AGE.YEAR</th>\n",
       "      <th>AVERAGE.ACCT.AGE.MONTH</th>\n",
       "      <th>CREDIT.HISTORY.LENGTH.YEAR</th>\n",
       "      <th>CREDIT.HISTORY.LENGTH.MONTH</th>\n",
       "      <th>Total_Flag</th>\n",
       "      <th>Branch_Emp_code</th>\n",
       "      <th>Pin_Emp_code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>1</td>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>1441</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1998</td>\n",
       "      <td>Salaried</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1719.2000</td>\n",
       "      <td>0.154652</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2065</td>\n",
       "      <td>3439</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1yrs 11mon</td>\n",
       "      <td>1</td>\n",
       "      <td>1yrs 11mon</td>\n",
       "      <td>1502</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1998</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>857.2650</td>\n",
       "      <td>0.390391</td>\n",
       "      <td>-1</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>2065</td>\n",
       "      <td>3500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>1</td>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>1497</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1998</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1718.9680</td>\n",
       "      <td>0.151695</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2065</td>\n",
       "      <td>3495</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0yrs 8mon</td>\n",
       "      <td>1</td>\n",
       "      <td>1yrs 3mon</td>\n",
       "      <td>1501</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1998</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>983.7824</td>\n",
       "      <td>0.149531</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>2065</td>\n",
       "      <td>3499</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>1</td>\n",
       "      <td>0yrs 0mon</td>\n",
       "      <td>1495</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1998</td>\n",
       "      <td>Self employed</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>921.1700</td>\n",
       "      <td>0.151247</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2065</td>\n",
       "      <td>3493</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 80 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  AVERAGE.ACCT.AGE Aadhar_flag CREDIT.HISTORY.LENGTH Current_pincode_ID  \\\n",
       "0        0yrs 0mon           1             0yrs 0mon               1441   \n",
       "1       1yrs 11mon           1            1yrs 11mon               1502   \n",
       "2        0yrs 0mon           1             0yrs 0mon               1497   \n",
       "3        0yrs 8mon           1             1yrs 3mon               1501   \n",
       "4        0yrs 0mon           1             0yrs 0mon               1495   \n",
       "\n",
       "   DELINQUENT.ACCTS.IN.LAST.SIX.MONTHS Driving_flag Employee_code_ID  \\\n",
       "0                                    0            0             1998   \n",
       "1                                    1            0             1998   \n",
       "2                                    0            0             1998   \n",
       "3                                    0            0             1998   \n",
       "4                                    0            0             1998   \n",
       "\n",
       "  Employment.Type  NEW.ACCTS.IN.LAST.SIX.MONTHS  NO.OF_INQUIRIES  ...  \\\n",
       "0        Salaried                             0                0  ...   \n",
       "1   Self employed                             0                0  ...   \n",
       "2   Self employed                             0                0  ...   \n",
       "3   Self employed                             0                1  ...   \n",
       "4   Self employed                             0                1  ...   \n",
       "\n",
       "  extra_finance  asset_disburse sixmmonths_dfault  AVERAGE.ACCT.AGE.YEAR  \\\n",
       "0     1719.2000        0.154652                 0                      0   \n",
       "1      857.2650        0.390391                -1                      1   \n",
       "2     1718.9680        0.151695                 0                      0   \n",
       "3      983.7824        0.149531                 0                      0   \n",
       "4      921.1700        0.151247                 0                      0   \n",
       "\n",
       "   AVERAGE.ACCT.AGE.MONTH  CREDIT.HISTORY.LENGTH.YEAR  \\\n",
       "0                       0                           0   \n",
       "1                      11                           1   \n",
       "2                       0                           0   \n",
       "3                       8                           1   \n",
       "4                       0                           0   \n",
       "\n",
       "   CREDIT.HISTORY.LENGTH.MONTH  Total_Flag  Branch_Emp_code  Pin_Emp_code  \n",
       "0                            0           1             2065          3439  \n",
       "1                           11           1             2065          3500  \n",
       "2                            0           1             2065          3495  \n",
       "3                            3           1             2065          3499  \n",
       "4                            0           1             2065          3493  \n",
       "\n",
       "[5 rows x 80 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dftrain=df[df['loan_default'].isnull()!=True]\n",
    "dftest=df[df['loan_default'].isnull()==True]\n",
    "dftrain.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X,y=dftrain.drop('loan_default',axis=1),dftrain['loan_default']\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.metrics import accuracy_score,f1_score\n",
    "X_train,X_val,y_train,y_val = train_test_split(X,y,test_size=0.3,random_state = 1994)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0,  1,  2,  3,  5,  6,  7, 10, 12, 20, 28, 29, 31, 34, 35, 47, 48,\n",
       "       49, 76, 77, 78], dtype=int64)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "categorical_features_indices = np.where(X_train.dtypes =='object')[0]\n",
    "categorical_features_indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyperparameter - learning_rate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import catboost\n",
    "\n",
    "class ModelOptimizer:\n",
    "    best_score = None\n",
    "    opt = None\n",
    "    \n",
    "    def __init__(self, model, X_train, y_train, categorical_columns_indices=None, n_fold=3, seed=1994, early_stopping_rounds=30, is_stratified=True, is_shuffle=True):\n",
    "        self.model = model\n",
    "        self.X_train = X_train\n",
    "        self.y_train = y_train\n",
    "        self.categorical_columns_indices = categorical_columns_indices\n",
    "        self.n_fold = n_fold\n",
    "        self.seed = seed\n",
    "        self.early_stopping_rounds = early_stopping_rounds\n",
    "        self.is_stratified = is_stratified\n",
    "        self.is_shuffle = is_shuffle\n",
    "        \n",
    "        \n",
    "    def update_model(self, **kwargs):\n",
    "        for k, v in kwargs.items():\n",
    "            setattr(self.model, k, v)\n",
    "            \n",
    "    def evaluate_model(self):\n",
    "        pass\n",
    "    \n",
    "    def optimize(self, param_space, max_evals=10, n_random_starts=2):\n",
    "        start_time = time.time()\n",
    "        \n",
    "        @use_named_args(param_space)\n",
    "        def _minimize(**params):\n",
    "            self.model.set_params(**params)\n",
    "            return self.evaluate_model()\n",
    "        \n",
    "        opt = gp_minimize(_minimize, param_space, n_calls=max_evals, n_random_starts=n_random_starts, random_state=2405, n_jobs=-1)\n",
    "        best_values = opt.x\n",
    "        optimal_values = dict(zip([param.name for param in param_space], best_values))\n",
    "        best_score = opt.fun\n",
    "        self.best_score = best_score\n",
    "        self.opt = opt\n",
    "        \n",
    "        print('optimal_parameters: {}\\noptimal score: {}\\noptimization time: {}'.format(optimal_values, best_score, time.time() - start_time))\n",
    "        print('updating model with optimal values')\n",
    "        self.update_model(**optimal_values)\n",
    "        plot_convergence(opt)\n",
    "        return optimal_values\n",
    "    \n",
    "class CatboostOptimizer(ModelOptimizer):\n",
    "    def evaluate_model(self):\n",
    "        validation_scores = catboost.cv(\n",
    "        catboost.Pool(self.X_train, \n",
    "                      self.y_train, \n",
    "                      cat_features=self.categorical_columns_indices),\n",
    "        self.model.get_params(), \n",
    "        nfold=self.n_fold,\n",
    "        stratified=self.is_stratified,\n",
    "        seed=self.seed,\n",
    "        early_stopping_rounds=self.early_stopping_rounds,\n",
    "        shuffle=self.is_shuffle,\n",
    "        verbose=100,\n",
    "        plot=False)\n",
    "        self.scores = validation_scores\n",
    "        test_scores = validation_scores.iloc[:, 2]\n",
    "        best_metric = test_scores.max()\n",
    "        return 1 - best_metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from skopt import gp_minimize\n",
    "from skopt.space import Real, Integer\n",
    "from skopt.utils import use_named_args\n",
    "from skopt.plots import plot_convergence\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.5846680\ttest: 0.5834369\tbest: 0.5834369 (0)\ttotal: 3.93s\tremaining: 4h 22m 1s\n",
      "100:\tlearn: 0.6747684\ttest: 0.6704210\tbest: 0.6704210 (100)\ttotal: 4m 21s\tremaining: 2h 48m 1s\n",
      "200:\tlearn: 0.6860045\ttest: 0.6736056\tbest: 0.6736131 (199)\ttotal: 9m 8s\tremaining: 2h 52m 41s\n",
      "300:\tlearn: 0.6921693\ttest: 0.6744569\tbest: 0.6744569 (300)\ttotal: 13m 53s\tremaining: 2h 50m 43s\n",
      "400:\tlearn: 0.6974692\ttest: 0.6751834\tbest: 0.6751834 (400)\ttotal: 18m 39s\tremaining: 2h 47m 30s\n",
      "500:\tlearn: 0.7014065\ttest: 0.6754404\tbest: 0.6754697 (490)\ttotal: 22m 3s\tremaining: 2h 34m 5s\n",
      "600:\tlearn: 0.7047433\ttest: 0.6756045\tbest: 0.6756527 (575)\ttotal: 25m 26s\tremaining: 2h 23m 53s\n",
      "Stopped by overfitting detector  (30 iterations wait)\n",
      "0:\tlearn: 0.5846680\ttest: 0.5834369\tbest: 0.5834369 (0)\ttotal: 2.17s\tremaining: 2h 24m 31s\n",
      "100:\tlearn: 0.6601778\ttest: 0.6605716\tbest: 0.6605716 (100)\ttotal: 4m 29s\tremaining: 2h 53m 34s\n",
      "200:\tlearn: 0.6692627\ttest: 0.6678177\tbest: 0.6678177 (200)\ttotal: 8m 56s\tremaining: 2h 49m\n",
      "300:\tlearn: 0.6731880\ttest: 0.6702803\tbest: 0.6702803 (300)\ttotal: 12m 37s\tremaining: 2h 35m 8s\n",
      "400:\tlearn: 0.6764220\ttest: 0.6719515\tbest: 0.6719515 (400)\ttotal: 16m 30s\tremaining: 2h 28m 6s\n",
      "500:\tlearn: 0.6801027\ttest: 0.6732671\tbest: 0.6732671 (500)\ttotal: 21m 4s\tremaining: 2h 27m 14s\n",
      "600:\tlearn: 0.6829130\ttest: 0.6740291\tbest: 0.6740292 (598)\ttotal: 25m 50s\tremaining: 2h 26m 9s\n",
      "700:\tlearn: 0.6855362\ttest: 0.6745807\tbest: 0.6745807 (700)\ttotal: 30m 40s\tremaining: 2h 24m 21s\n",
      "800:\tlearn: 0.6876898\ttest: 0.6750496\tbest: 0.6750522 (799)\ttotal: 35m 51s\tremaining: 2h 23m 12s\n",
      "900:\tlearn: 0.6896322\ttest: 0.6753778\tbest: 0.6753778 (900)\ttotal: 40m 49s\tremaining: 2h 20m 24s\n",
      "1000:\tlearn: 0.6914878\ttest: 0.6757074\tbest: 0.6757074 (1000)\ttotal: 45m 47s\tremaining: 2h 17m 11s\n",
      "1100:\tlearn: 0.6931259\ttest: 0.6759807\tbest: 0.6759807 (1100)\ttotal: 49m 9s\tremaining: 2h 9m 25s\n",
      "1200:\tlearn: 0.6945920\ttest: 0.6761861\tbest: 0.6761868 (1199)\ttotal: 52m 27s\tremaining: 2h 2m 15s\n",
      "1300:\tlearn: 0.6958864\ttest: 0.6763849\tbest: 0.6763849 (1300)\ttotal: 55m 38s\tremaining: 1h 55m 26s\n",
      "1400:\tlearn: 0.6972064\ttest: 0.6765586\tbest: 0.6765586 (1400)\ttotal: 58m 45s\tremaining: 1h 49m\n",
      "1500:\tlearn: 0.6984790\ttest: 0.6766758\tbest: 0.6766776 (1498)\ttotal: 1h 4m 7s\tremaining: 1h 46m 45s\n",
      "1600:\tlearn: 0.6997102\ttest: 0.6767979\tbest: 0.6767979 (1600)\ttotal: 1h 9m 29s\tremaining: 1h 44m 8s\n",
      "1700:\tlearn: 0.7007124\ttest: 0.6768468\tbest: 0.6768468 (1700)\ttotal: 1h 14m 48s\tremaining: 1h 41m 6s\n",
      "1800:\tlearn: 0.7018291\ttest: 0.6769545\tbest: 0.6769545 (1800)\ttotal: 1h 20m 9s\tremaining: 1h 37m 52s\n",
      "1900:\tlearn: 0.7028785\ttest: 0.6770351\tbest: 0.6770351 (1900)\ttotal: 1h 24m 23s\tremaining: 1h 33m 10s\n",
      "2000:\tlearn: 0.7037043\ttest: 0.6771178\tbest: 0.6771178 (2000)\ttotal: 1h 27m 24s\tremaining: 1h 27m 18s\n",
      "2100:\tlearn: 0.7045327\ttest: 0.6771948\tbest: 0.6771960 (2090)\ttotal: 1h 30m 32s\tremaining: 1h 21m 50s\n",
      "2200:\tlearn: 0.7053711\ttest: 0.6772940\tbest: 0.6772940 (2200)\ttotal: 1h 33m 40s\tremaining: 1h 16m 33s\n",
      "2300:\tlearn: 0.7060860\ttest: 0.6773412\tbest: 0.6773415 (2299)\ttotal: 1h 36m 46s\tremaining: 1h 11m 27s\n",
      "2400:\tlearn: 0.7068715\ttest: 0.6773920\tbest: 0.6773957 (2399)\ttotal: 1h 41m 26s\tremaining: 1h 7m 33s\n",
      "2500:\tlearn: 0.7075235\ttest: 0.6774325\tbest: 0.6774336 (2495)\ttotal: 1h 46m 11s\tremaining: 1h 3m 38s\n",
      "Stopped by overfitting detector  (30 iterations wait)\n",
      "0:\tlearn: 0.5846680\ttest: 0.5834369\tbest: 0.5834369 (0)\ttotal: 2.09s\tremaining: 2h 19m 5s\n",
      "Stopped by overfitting detector  (30 iterations wait)\n",
      "0:\tlearn: 0.5846680\ttest: 0.5834369\tbest: 0.5834369 (0)\ttotal: 2.29s\tremaining: 2h 32m 45s\n",
      "100:\tlearn: 0.6404873\ttest: 0.6395678\tbest: 0.6395678 (100)\ttotal: 5m 27s\tremaining: 3h 30m 55s\n",
      "200:\tlearn: 0.6535273\ttest: 0.6539822\tbest: 0.6539822 (200)\ttotal: 10m 58s\tremaining: 3h 27m 18s\n",
      "300:\tlearn: 0.6604842\ttest: 0.6607351\tbest: 0.6607351 (300)\ttotal: 16m 37s\tremaining: 3h 24m 16s\n",
      "400:\tlearn: 0.6645557\ttest: 0.6642086\tbest: 0.6642086 (400)\ttotal: 19m 54s\tremaining: 2h 58m 40s\n",
      "500:\tlearn: 0.6673193\ttest: 0.6663282\tbest: 0.6663282 (500)\ttotal: 23m 5s\tremaining: 2h 41m 18s\n",
      "600:\tlearn: 0.6692305\ttest: 0.6676843\tbest: 0.6676843 (600)\ttotal: 26m 18s\tremaining: 2h 28m 46s\n",
      "700:\tlearn: 0.6709732\ttest: 0.6688732\tbest: 0.6688732 (700)\ttotal: 29m 27s\tremaining: 2h 18m 37s\n",
      "800:\tlearn: 0.6723604\ttest: 0.6697781\tbest: 0.6697781 (800)\ttotal: 33m 9s\tremaining: 2h 12m 25s\n",
      "900:\tlearn: 0.6735134\ttest: 0.6704353\tbest: 0.6704353 (900)\ttotal: 37m 50s\tremaining: 2h 10m 8s\n",
      "1000:\tlearn: 0.6746846\ttest: 0.6711162\tbest: 0.6711162 (1000)\ttotal: 42m 24s\tremaining: 2h 7m 4s\n",
      "1100:\tlearn: 0.6757269\ttest: 0.6716329\tbest: 0.6716329 (1100)\ttotal: 46m 54s\tremaining: 2h 3m 31s\n",
      "1200:\tlearn: 0.6769345\ttest: 0.6722014\tbest: 0.6722014 (1200)\ttotal: 51m 44s\tremaining: 2h 34s\n",
      "1300:\tlearn: 0.6782775\ttest: 0.6727480\tbest: 0.6727480 (1300)\ttotal: 56m 35s\tremaining: 1h 57m 23s\n",
      "1400:\tlearn: 0.6795050\ttest: 0.6731607\tbest: 0.6731607 (1400)\ttotal: 1h 1m 36s\tremaining: 1h 54m 16s\n",
      "1500:\tlearn: 0.6806738\ttest: 0.6735004\tbest: 0.6735004 (1500)\ttotal: 1h 5m 9s\tremaining: 1h 48m 29s\n",
      "1600:\tlearn: 0.6817776\ttest: 0.6738338\tbest: 0.6738338 (1600)\ttotal: 1h 8m 44s\tremaining: 1h 42m 59s\n",
      "1700:\tlearn: 0.6828037\ttest: 0.6741312\tbest: 0.6741312 (1700)\ttotal: 1h 12m 14s\tremaining: 1h 37m 38s\n",
      "1800:\tlearn: 0.6837205\ttest: 0.6743748\tbest: 0.6743748 (1800)\ttotal: 1h 17m 18s\tremaining: 1h 34m 23s\n",
      "1900:\tlearn: 0.6846098\ttest: 0.6745930\tbest: 0.6745936 (1898)\ttotal: 1h 22m 34s\tremaining: 1h 31m 10s\n",
      "2000:\tlearn: 0.6854444\ttest: 0.6747801\tbest: 0.6747802 (1999)\ttotal: 1h 27m 51s\tremaining: 1h 27m 46s\n",
      "2100:\tlearn: 0.6861989\ttest: 0.6749264\tbest: 0.6749264 (2100)\ttotal: 1h 32m 39s\tremaining: 1h 23m 44s\n",
      "2200:\tlearn: 0.6869479\ttest: 0.6750922\tbest: 0.6750922 (2200)\ttotal: 1h 37m 26s\tremaining: 1h 19m 38s\n",
      "2300:\tlearn: 0.6876511\ttest: 0.6752242\tbest: 0.6752242 (2300)\ttotal: 1h 42m 15s\tremaining: 1h 15m 30s\n",
      "2400:\tlearn: 0.6883661\ttest: 0.6753550\tbest: 0.6753550 (2400)\ttotal: 1h 47m 3s\tremaining: 1h 11m 17s\n",
      "2500:\tlearn: 0.6890650\ttest: 0.6754837\tbest: 0.6754837 (2500)\ttotal: 1h 51m 35s\tremaining: 1h 6m 52s\n",
      "2600:\tlearn: 0.6897493\ttest: 0.6756082\tbest: 0.6756082 (2600)\ttotal: 1h 56m 39s\tremaining: 1h 2m 44s\n",
      "2700:\tlearn: 0.6903320\ttest: 0.6757098\tbest: 0.6757098 (2700)\ttotal: 2h 1m 40s\tremaining: 58m 30s\n",
      "2800:\tlearn: 0.6909645\ttest: 0.6758086\tbest: 0.6758086 (2800)\ttotal: 2h 6m 25s\tremaining: 54m 6s\n",
      "2900:\tlearn: 0.6915311\ttest: 0.6758887\tbest: 0.6758887 (2900)\ttotal: 2h 11m 22s\tremaining: 49m 46s\n",
      "3000:\tlearn: 0.6921670\ttest: 0.6759862\tbest: 0.6759862 (3000)\ttotal: 2h 16m 21s\tremaining: 45m 23s\n",
      "3100:\tlearn: 0.6926994\ttest: 0.6760876\tbest: 0.6760876 (3100)\ttotal: 2h 21m 16s\tremaining: 40m 57s\n",
      "3200:\tlearn: 0.6933014\ttest: 0.6762055\tbest: 0.6762055 (3200)\ttotal: 2h 26m 12s\tremaining: 36m 29s\n",
      "3300:\tlearn: 0.6938615\ttest: 0.6762748\tbest: 0.6762748 (3300)\ttotal: 2h 31m 9s\tremaining: 32m\n",
      "3400:\tlearn: 0.6943966\ttest: 0.6763466\tbest: 0.6763466 (3400)\ttotal: 2h 36m 7s\tremaining: 27m 29s\n",
      "3500:\tlearn: 0.6949369\ttest: 0.6764113\tbest: 0.6764113 (3500)\ttotal: 2h 41m 6s\tremaining: 22m 57s\n",
      "3600:\tlearn: 0.6953663\ttest: 0.6764575\tbest: 0.6764578 (3599)\ttotal: 2h 45m 52s\tremaining: 18m 22s\n",
      "3700:\tlearn: 0.6958536\ttest: 0.6765158\tbest: 0.6765158 (3700)\ttotal: 2h 50m 50s\tremaining: 13m 48s\n",
      "3800:\tlearn: 0.6963466\ttest: 0.6765857\tbest: 0.6765857 (3800)\ttotal: 2h 55m 45s\tremaining: 9m 12s\n",
      "3900:\tlearn: 0.6968101\ttest: 0.6766318\tbest: 0.6766318 (3900)\ttotal: 3h 42s\tremaining: 4m 35s\n",
      "3999:\tlearn: 0.6972359\ttest: 0.6766923\tbest: 0.6766923 (3998)\ttotal: 3h 5m 38s\tremaining: 0us\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Acer\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:399: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.5846680\ttest: 0.5834369\tbest: 0.5834369 (0)\ttotal: 2.18s\tremaining: 2h 25m 36s\n",
      "100:\tlearn: 0.6404873\ttest: 0.6395678\tbest: 0.6395678 (100)\ttotal: 4m 33s\tremaining: 2h 56m 3s\n",
      "200:\tlearn: 0.6535273\ttest: 0.6539822\tbest: 0.6539822 (200)\ttotal: 9m 4s\tremaining: 2h 51m 33s\n",
      "300:\tlearn: 0.6604842\ttest: 0.6607351\tbest: 0.6607351 (300)\ttotal: 13m 56s\tremaining: 2h 51m 23s\n",
      "400:\tlearn: 0.6645557\ttest: 0.6642086\tbest: 0.6642086 (400)\ttotal: 18m 40s\tremaining: 2h 47m 36s\n",
      "500:\tlearn: 0.6673193\ttest: 0.6663282\tbest: 0.6663282 (500)\ttotal: 23m 11s\tremaining: 2h 42m 1s\n",
      "600:\tlearn: 0.6692305\ttest: 0.6676843\tbest: 0.6676843 (600)\ttotal: 27m 30s\tremaining: 2h 35m 35s\n",
      "700:\tlearn: 0.6709732\ttest: 0.6688732\tbest: 0.6688732 (700)\ttotal: 31m 50s\tremaining: 2h 29m 51s\n",
      "800:\tlearn: 0.6723604\ttest: 0.6697781\tbest: 0.6697781 (800)\ttotal: 35m 47s\tremaining: 2h 22m 55s\n",
      "900:\tlearn: 0.6735134\ttest: 0.6704353\tbest: 0.6704353 (900)\ttotal: 40m\tremaining: 2h 17m 37s\n",
      "1000:\tlearn: 0.6746846\ttest: 0.6711162\tbest: 0.6711162 (1000)\ttotal: 44m 8s\tremaining: 2h 12m 15s\n",
      "1100:\tlearn: 0.6757269\ttest: 0.6716329\tbest: 0.6716329 (1100)\ttotal: 48m 10s\tremaining: 2h 6m 51s\n",
      "1200:\tlearn: 0.6769345\ttest: 0.6722014\tbest: 0.6722014 (1200)\ttotal: 52m 40s\tremaining: 2h 2m 45s\n",
      "1300:\tlearn: 0.6782775\ttest: 0.6727480\tbest: 0.6727480 (1300)\ttotal: 57m 26s\tremaining: 1h 59m 9s\n",
      "1400:\tlearn: 0.6795050\ttest: 0.6731607\tbest: 0.6731607 (1400)\ttotal: 1h 2m 21s\tremaining: 1h 55m 40s\n",
      "1500:\tlearn: 0.6806738\ttest: 0.6735004\tbest: 0.6735004 (1500)\ttotal: 1h 7m 19s\tremaining: 1h 52m 5s\n",
      "1600:\tlearn: 0.6817776\ttest: 0.6738338\tbest: 0.6738338 (1600)\ttotal: 1h 12m 9s\tremaining: 1h 48m 7s\n",
      "1700:\tlearn: 0.6828037\ttest: 0.6741312\tbest: 0.6741312 (1700)\ttotal: 1h 17m 9s\tremaining: 1h 44m 17s\n",
      "1800:\tlearn: 0.6837205\ttest: 0.6743748\tbest: 0.6743748 (1800)\ttotal: 1h 22m 10s\tremaining: 1h 40m 20s\n",
      "1900:\tlearn: 0.6846098\ttest: 0.6745930\tbest: 0.6745936 (1898)\ttotal: 1h 27m 12s\tremaining: 1h 36m 17s\n",
      "2000:\tlearn: 0.6854444\ttest: 0.6747801\tbest: 0.6747802 (1999)\ttotal: 1h 32m 15s\tremaining: 1h 32m 9s\n",
      "2100:\tlearn: 0.6861989\ttest: 0.6749264\tbest: 0.6749264 (2100)\ttotal: 1h 37m 16s\tremaining: 1h 27m 55s\n",
      "2200:\tlearn: 0.6869479\ttest: 0.6750922\tbest: 0.6750922 (2200)\ttotal: 1h 42m 17s\tremaining: 1h 23m 36s\n",
      "2300:\tlearn: 0.6876511\ttest: 0.6752242\tbest: 0.6752242 (2300)\ttotal: 1h 47m 6s\tremaining: 1h 19m 5s\n",
      "2400:\tlearn: 0.6883661\ttest: 0.6753550\tbest: 0.6753550 (2400)\ttotal: 1h 52m 5s\tremaining: 1h 14m 39s\n",
      "2500:\tlearn: 0.6890650\ttest: 0.6754837\tbest: 0.6754837 (2500)\ttotal: 1h 57m 7s\tremaining: 1h 10m 11s\n",
      "2600:\tlearn: 0.6897493\ttest: 0.6756082\tbest: 0.6756082 (2600)\ttotal: 2h 2m 10s\tremaining: 1h 5m 42s\n",
      "2700:\tlearn: 0.6903320\ttest: 0.6757098\tbest: 0.6757098 (2700)\ttotal: 2h 7m 13s\tremaining: 1h 1m 11s\n",
      "2800:\tlearn: 0.6909645\ttest: 0.6758086\tbest: 0.6758086 (2800)\ttotal: 2h 12m 18s\tremaining: 56m 38s\n",
      "2900:\tlearn: 0.6915311\ttest: 0.6758887\tbest: 0.6758887 (2900)\ttotal: 2h 17m 21s\tremaining: 52m 2s\n",
      "3000:\tlearn: 0.6921670\ttest: 0.6759862\tbest: 0.6759862 (3000)\ttotal: 2h 22m 25s\tremaining: 47m 24s\n",
      "3100:\tlearn: 0.6926994\ttest: 0.6760876\tbest: 0.6760876 (3100)\ttotal: 2h 27m 17s\tremaining: 42m 41s\n",
      "3200:\tlearn: 0.6933014\ttest: 0.6762055\tbest: 0.6762055 (3200)\ttotal: 2h 32m 17s\tremaining: 38m\n",
      "3300:\tlearn: 0.6938615\ttest: 0.6762748\tbest: 0.6762748 (3300)\ttotal: 2h 37m 22s\tremaining: 33m 19s\n",
      "3400:\tlearn: 0.6943966\ttest: 0.6763466\tbest: 0.6763466 (3400)\ttotal: 2h 42m 23s\tremaining: 28m 36s\n",
      "3500:\tlearn: 0.6949369\ttest: 0.6764113\tbest: 0.6764113 (3500)\ttotal: 2h 47m 30s\tremaining: 23m 52s\n",
      "3600:\tlearn: 0.6953663\ttest: 0.6764575\tbest: 0.6764578 (3599)\ttotal: 2h 52m 35s\tremaining: 19m 7s\n",
      "3700:\tlearn: 0.6958536\ttest: 0.6765158\tbest: 0.6765158 (3700)\ttotal: 2h 57m 40s\tremaining: 14m 21s\n",
      "3800:\tlearn: 0.6963466\ttest: 0.6765857\tbest: 0.6765857 (3800)\ttotal: 3h 2m 44s\tremaining: 9m 34s\n",
      "3900:\tlearn: 0.6968101\ttest: 0.6766318\tbest: 0.6766318 (3900)\ttotal: 3h 7m 35s\tremaining: 4m 45s\n",
      "3999:\tlearn: 0.6972359\ttest: 0.6766923\tbest: 0.6766923 (3998)\ttotal: 3h 12m 39s\tremaining: 0us\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Acer\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:399: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.5846680\ttest: 0.5834369\tbest: 0.5834369 (0)\ttotal: 2.19s\tremaining: 2h 26m 14s\n",
      "100:\tlearn: 0.6404873\ttest: 0.6395678\tbest: 0.6395678 (100)\ttotal: 4m 32s\tremaining: 2h 55m 35s\n",
      "200:\tlearn: 0.6535273\ttest: 0.6539822\tbest: 0.6539822 (200)\ttotal: 9m 17s\tremaining: 2h 55m 29s\n",
      "300:\tlearn: 0.6604842\ttest: 0.6607351\tbest: 0.6607351 (300)\ttotal: 14m 11s\tremaining: 2h 54m 24s\n",
      "400:\tlearn: 0.6645557\ttest: 0.6642086\tbest: 0.6642086 (400)\ttotal: 18m 46s\tremaining: 2h 48m 27s\n",
      "500:\tlearn: 0.6673193\ttest: 0.6663282\tbest: 0.6663282 (500)\ttotal: 23m 19s\tremaining: 2h 42m 54s\n",
      "600:\tlearn: 0.6692305\ttest: 0.6676843\tbest: 0.6676843 (600)\ttotal: 27m 40s\tremaining: 2h 36m 33s\n",
      "700:\tlearn: 0.6709732\ttest: 0.6688732\tbest: 0.6688732 (700)\ttotal: 32m 2s\tremaining: 2h 30m 48s\n",
      "800:\tlearn: 0.6723604\ttest: 0.6697781\tbest: 0.6697781 (800)\ttotal: 36m 15s\tremaining: 2h 24m 49s\n",
      "900:\tlearn: 0.6735134\ttest: 0.6704353\tbest: 0.6704353 (900)\ttotal: 40m 32s\tremaining: 2h 19m 25s\n",
      "1000:\tlearn: 0.6746846\ttest: 0.6711162\tbest: 0.6711162 (1000)\ttotal: 44m 39s\tremaining: 2h 13m 48s\n",
      "1100:\tlearn: 0.6757269\ttest: 0.6716329\tbest: 0.6716329 (1100)\ttotal: 48m 43s\tremaining: 2h 8m 17s\n",
      "1200:\tlearn: 0.6769345\ttest: 0.6722014\tbest: 0.6722014 (1200)\ttotal: 53m 14s\tremaining: 2h 4m 5s\n",
      "1300:\tlearn: 0.6782775\ttest: 0.6727480\tbest: 0.6727480 (1300)\ttotal: 58m 4s\tremaining: 2h 27s\n",
      "1400:\tlearn: 0.6795050\ttest: 0.6731607\tbest: 0.6731607 (1400)\ttotal: 1h 2m 48s\tremaining: 1h 56m 30s\n",
      "1500:\tlearn: 0.6806738\ttest: 0.6735004\tbest: 0.6735004 (1500)\ttotal: 1h 7m 48s\tremaining: 1h 52m 53s\n",
      "1600:\tlearn: 0.6817776\ttest: 0.6738338\tbest: 0.6738338 (1600)\ttotal: 1h 12m 50s\tremaining: 1h 49m 9s\n",
      "1700:\tlearn: 0.6828037\ttest: 0.6741312\tbest: 0.6741312 (1700)\ttotal: 1h 17m 52s\tremaining: 1h 45m 15s\n",
      "1800:\tlearn: 0.6837205\ttest: 0.6743748\tbest: 0.6743748 (1800)\ttotal: 1h 22m 55s\tremaining: 1h 41m 15s\n",
      "1900:\tlearn: 0.6846098\ttest: 0.6745930\tbest: 0.6745936 (1898)\ttotal: 1h 27m 57s\tremaining: 1h 37m 7s\n",
      "2000:\tlearn: 0.6854444\ttest: 0.6747801\tbest: 0.6747802 (1999)\ttotal: 1h 33m 4s\tremaining: 1h 32m 58s\n",
      "2100:\tlearn: 0.6861989\ttest: 0.6749264\tbest: 0.6749264 (2100)\ttotal: 1h 37m 56s\tremaining: 1h 28m 31s\n",
      "2200:\tlearn: 0.6869479\ttest: 0.6750922\tbest: 0.6750922 (2200)\ttotal: 1h 42m 59s\tremaining: 1h 24m 11s\n",
      "2300:\tlearn: 0.6876511\ttest: 0.6752242\tbest: 0.6752242 (2300)\ttotal: 1h 48m 2s\tremaining: 1h 19m 46s\n",
      "2400:\tlearn: 0.6883661\ttest: 0.6753550\tbest: 0.6753550 (2400)\ttotal: 1h 53m 4s\tremaining: 1h 15m 18s\n",
      "2500:\tlearn: 0.6890650\ttest: 0.6754837\tbest: 0.6754837 (2500)\ttotal: 1h 58m 9s\tremaining: 1h 10m 49s\n",
      "2600:\tlearn: 0.6897493\ttest: 0.6756082\tbest: 0.6756082 (2600)\ttotal: 2h 3m 12s\tremaining: 1h 6m 16s\n",
      "2700:\tlearn: 0.6903320\ttest: 0.6757098\tbest: 0.6757098 (2700)\ttotal: 2h 8m 3s\tremaining: 1h 1m 35s\n",
      "2800:\tlearn: 0.6909645\ttest: 0.6758086\tbest: 0.6758086 (2800)\ttotal: 2h 13m 6s\tremaining: 56m 58s\n",
      "2900:\tlearn: 0.6915311\ttest: 0.6758887\tbest: 0.6758887 (2900)\ttotal: 2h 18m 9s\tremaining: 52m 20s\n",
      "3000:\tlearn: 0.6921670\ttest: 0.6759862\tbest: 0.6759862 (3000)\ttotal: 2h 22m 59s\tremaining: 47m 36s\n",
      "3100:\tlearn: 0.6926994\ttest: 0.6760876\tbest: 0.6760876 (3100)\ttotal: 2h 28m 10s\tremaining: 42m 57s\n",
      "3200:\tlearn: 0.6933014\ttest: 0.6762055\tbest: 0.6762055 (3200)\ttotal: 2h 33m 19s\tremaining: 38m 16s\n",
      "3300:\tlearn: 0.6938615\ttest: 0.6762748\tbest: 0.6762748 (3300)\ttotal: 2h 38m 26s\tremaining: 33m 33s\n",
      "3400:\tlearn: 0.6943966\ttest: 0.6763466\tbest: 0.6763466 (3400)\ttotal: 2h 43m 34s\tremaining: 28m 48s\n",
      "3500:\tlearn: 0.6949369\ttest: 0.6764113\tbest: 0.6764113 (3500)\ttotal: 2h 48m 46s\tremaining: 24m 3s\n",
      "3600:\tlearn: 0.6953663\ttest: 0.6764575\tbest: 0.6764578 (3599)\ttotal: 2h 53m 55s\tremaining: 19m 16s\n",
      "3700:\tlearn: 0.6958536\ttest: 0.6765158\tbest: 0.6765158 (3700)\ttotal: 2h 58m 52s\tremaining: 14m 27s\n",
      "3800:\tlearn: 0.6963466\ttest: 0.6765857\tbest: 0.6765857 (3800)\ttotal: 3h 3m 59s\tremaining: 9m 37s\n",
      "3900:\tlearn: 0.6968101\ttest: 0.6766318\tbest: 0.6766318 (3900)\ttotal: 3h 9m 13s\tremaining: 4m 48s\n",
      "3999:\tlearn: 0.6972359\ttest: 0.6766923\tbest: 0.6766923 (3998)\ttotal: 3h 14m 33s\tremaining: 0us\n",
      "0:\tlearn: 0.5846680\ttest: 0.5834369\tbest: 0.5834369 (0)\ttotal: 2.41s\tremaining: 2h 40m 48s\n",
      "100:\tlearn: 0.6946016\ttest: 0.6661463\tbest: 0.6665307 (71)\ttotal: 5m 11s\tremaining: 3h 20m 36s\n",
      "Stopped by overfitting detector  (30 iterations wait)\n",
      "0:\tlearn: 0.5846680\ttest: 0.5834369\tbest: 0.5834369 (0)\ttotal: 2.34s\tremaining: 2h 35m 47s\n",
      "Stopped by overfitting detector  (30 iterations wait)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Acer\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:399: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\tlearn: 0.5846680\ttest: 0.5834369\tbest: 0.5834369 (0)\ttotal: 2.26s\tremaining: 2h 30m 33s\n",
      "100:\tlearn: 0.6404873\ttest: 0.6395678\tbest: 0.6395678 (100)\ttotal: 4m 43s\tremaining: 3h 2m 27s\n",
      "200:\tlearn: 0.6535273\ttest: 0.6539822\tbest: 0.6539822 (200)\ttotal: 9m 18s\tremaining: 2h 56m 3s\n",
      "300:\tlearn: 0.6604842\ttest: 0.6607351\tbest: 0.6607351 (300)\ttotal: 13m 18s\tremaining: 2h 43m 36s\n",
      "400:\tlearn: 0.6645557\ttest: 0.6642086\tbest: 0.6642086 (400)\ttotal: 18m 8s\tremaining: 2h 42m 50s\n",
      "500:\tlearn: 0.6673193\ttest: 0.6663282\tbest: 0.6663282 (500)\ttotal: 22m 44s\tremaining: 2h 38m 49s\n",
      "600:\tlearn: 0.6692305\ttest: 0.6676843\tbest: 0.6676843 (600)\ttotal: 27m 6s\tremaining: 2h 33m 20s\n",
      "700:\tlearn: 0.6709732\ttest: 0.6688732\tbest: 0.6688732 (700)\ttotal: 31m 33s\tremaining: 2h 28m 32s\n",
      "800:\tlearn: 0.6723604\ttest: 0.6697781\tbest: 0.6697781 (800)\ttotal: 35m 47s\tremaining: 2h 22m 56s\n",
      "900:\tlearn: 0.6735134\ttest: 0.6704353\tbest: 0.6704353 (900)\ttotal: 40m 5s\tremaining: 2h 17m 54s\n",
      "1000:\tlearn: 0.6746846\ttest: 0.6711162\tbest: 0.6711162 (1000)\ttotal: 44m 17s\tremaining: 2h 12m 42s\n",
      "1100:\tlearn: 0.6757269\ttest: 0.6716329\tbest: 0.6716329 (1100)\ttotal: 48m 39s\tremaining: 2h 8m 6s\n",
      "1200:\tlearn: 0.6769345\ttest: 0.6722014\tbest: 0.6722014 (1200)\ttotal: 54m 13s\tremaining: 2h 6m 22s\n",
      "1300:\tlearn: 0.6782775\ttest: 0.6727480\tbest: 0.6727480 (1300)\ttotal: 1h 7s\tremaining: 2h 4m 43s\n",
      "1400:\tlearn: 0.6795050\ttest: 0.6731607\tbest: 0.6731607 (1400)\ttotal: 1h 6m 18s\tremaining: 2h 3m\n",
      "1500:\tlearn: 0.6806738\ttest: 0.6735004\tbest: 0.6735004 (1500)\ttotal: 1h 12m 41s\tremaining: 2h 1m 1s\n",
      "1600:\tlearn: 0.6817776\ttest: 0.6738338\tbest: 0.6738338 (1600)\ttotal: 1h 18m 57s\tremaining: 1h 58m 18s\n",
      "1700:\tlearn: 0.6828037\ttest: 0.6741312\tbest: 0.6741312 (1700)\ttotal: 1h 25m 16s\tremaining: 1h 55m 15s\n",
      "1800:\tlearn: 0.6837205\ttest: 0.6743748\tbest: 0.6743748 (1800)\ttotal: 1h 32m 21s\tremaining: 1h 52m 45s\n",
      "1900:\tlearn: 0.6846098\ttest: 0.6745930\tbest: 0.6745936 (1898)\ttotal: 1h 38m 17s\tremaining: 1h 48m 31s\n",
      "2000:\tlearn: 0.6854444\ttest: 0.6747801\tbest: 0.6747802 (1999)\ttotal: 1h 44m 28s\tremaining: 1h 44m 22s\n",
      "2100:\tlearn: 0.6861989\ttest: 0.6749264\tbest: 0.6749264 (2100)\ttotal: 1h 50m 52s\tremaining: 1h 40m 12s\n",
      "2200:\tlearn: 0.6869479\ttest: 0.6750922\tbest: 0.6750922 (2200)\ttotal: 1h 57m 9s\tremaining: 1h 35m 45s\n",
      "2300:\tlearn: 0.6876511\ttest: 0.6752242\tbest: 0.6752242 (2300)\ttotal: 2h 3m 29s\tremaining: 1h 31m 11s\n",
      "2400:\tlearn: 0.6883661\ttest: 0.6753550\tbest: 0.6753550 (2400)\ttotal: 2h 9m 45s\tremaining: 1h 26m 24s\n",
      "2500:\tlearn: 0.6890650\ttest: 0.6754837\tbest: 0.6754837 (2500)\ttotal: 2h 15m 58s\tremaining: 1h 21m 29s\n",
      "2600:\tlearn: 0.6897493\ttest: 0.6756082\tbest: 0.6756082 (2600)\ttotal: 2h 22m 15s\tremaining: 1h 16m 30s\n",
      "2700:\tlearn: 0.6903320\ttest: 0.6757098\tbest: 0.6757098 (2700)\ttotal: 2h 28m 28s\tremaining: 1h 11m 24s\n",
      "2800:\tlearn: 0.6909645\ttest: 0.6758086\tbest: 0.6758086 (2800)\ttotal: 2h 34m 48s\tremaining: 1h 6m 16s\n",
      "2900:\tlearn: 0.6915311\ttest: 0.6758887\tbest: 0.6758887 (2900)\ttotal: 2h 41m 8s\tremaining: 1h 1m 2s\n",
      "3000:\tlearn: 0.6921670\ttest: 0.6759862\tbest: 0.6759862 (3000)\ttotal: 2h 47m 25s\tremaining: 55m 44s\n",
      "3100:\tlearn: 0.6926994\ttest: 0.6760876\tbest: 0.6760876 (3100)\ttotal: 2h 53m 48s\tremaining: 50m 23s\n",
      "3200:\tlearn: 0.6933014\ttest: 0.6762055\tbest: 0.6762055 (3200)\ttotal: 2h 59m 51s\tremaining: 44m 53s\n",
      "3300:\tlearn: 0.6938615\ttest: 0.6762748\tbest: 0.6762748 (3300)\ttotal: 3h 6m 12s\tremaining: 39m 25s\n",
      "3400:\tlearn: 0.6943966\ttest: 0.6763466\tbest: 0.6763466 (3400)\ttotal: 3h 12m 31s\tremaining: 33m 54s\n",
      "3500:\tlearn: 0.6949369\ttest: 0.6764113\tbest: 0.6764113 (3500)\ttotal: 3h 18m 58s\tremaining: 28m 21s\n",
      "3600:\tlearn: 0.6953663\ttest: 0.6764575\tbest: 0.6764578 (3599)\ttotal: 3h 25m 16s\tremaining: 22m 44s\n",
      "3700:\tlearn: 0.6958536\ttest: 0.6765158\tbest: 0.6765158 (3700)\ttotal: 3h 31m 44s\tremaining: 17m 6s\n",
      "3800:\tlearn: 0.6963466\ttest: 0.6765857\tbest: 0.6765857 (3800)\ttotal: 3h 37m 46s\tremaining: 11m 24s\n",
      "3900:\tlearn: 0.6968101\ttest: 0.6766318\tbest: 0.6766318 (3900)\ttotal: 3h 44m 14s\tremaining: 5m 41s\n",
      "3999:\tlearn: 0.6972359\ttest: 0.6766923\tbest: 0.6766923 (3998)\ttotal: 3h 50m 43s\tremaining: 0us\n",
      "0:\tlearn: 0.5846680\ttest: 0.5834369\tbest: 0.5834369 (0)\ttotal: 2.28s\tremaining: 2h 31m 57s\n",
      "Stopped by overfitting detector  (30 iterations wait)\n",
      "optimal_parameters: {'learning_rate': 0.029503887680470645}\n",
      "optimal score: 0.2921053862868286\n",
      "optimization time: 57681.39850282669\n",
      "updating model with optimal values\n"
     ]
    }
   ],
   "source": [
    "cb = catboost.CatBoostClassifier(n_estimators=4000, # use large n_estimators deliberately to make use of the early stopping\n",
    "                         loss_function='Logloss',\n",
    "                         eval_metric='AUC',\n",
    "                         boosting_type='Ordered', # use permutations\n",
    "                         random_seed=1994, \n",
    "                         use_best_model=True)\n",
    "cb_optimizer = CatboostOptimizer(cb, X_train, y_train,categorical_columns_indices=categorical_features_indices)\n",
    "params_space = [Real(0.01, 0.8, name='learning_rate'),]\n",
    "cb_optimal_values = cb_optimizer.optimize(params_space)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tried and Evaluated\n",
    "\n",
    "## Journey\n",
    "\n",
    "* **Started with LGBM and XGB which has a CV of 67.7 and lb score as 65.1**\n",
    "* **Tried tuning it but could'nt increase the leaderboard score much.**\n",
    "* **Started with CatBoost for which a simple model without and Feature engineering scored 65.70 in Public LB**\n",
    "* **Tuned in, tried varied possible combination of features, using One- Hot Encoding, with categoricals as well as other methods the increment in Local CV dose'nt matched with the Public Leaderboard score.**\n",
    "* **Tried feature selection which also worked only till a point, after which 10-folds CV reached to 68.2X which helped to increase the score till 66.62 in Public Leaderboard.**\n",
    "* **One model training was taking at least 1hr, so my lapp**\n",
    "\n",
    "## Things Tried in Several Other Notebooks which did'nt made it to final Cut :p\n",
    "\n",
    "* **Model Tuning**\n",
    "* **Curbing the Overfitting**\n",
    "* **Tried many possible combinations of hyperparameters**\n",
    "* **Tried other models like XGB and LGBM which sadly didnt work**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\ttest: 0.6256996\ttest1: 0.6175770\tbest: 0.6175770 (0)\ttotal: 963ms\tremaining: 48m 8s\n",
      "100:\ttest: 0.6939269\ttest1: 0.6641209\tbest: 0.6641209 (100)\ttotal: 1m 47s\tremaining: 51m 19s\n",
      "200:\ttest: 0.7031226\ttest1: 0.6706726\tbest: 0.6706726 (200)\ttotal: 3m 35s\tremaining: 49m 58s\n",
      "300:\ttest: 0.7084578\ttest1: 0.6734228\tbest: 0.6734228 (300)\ttotal: 5m 21s\tremaining: 48m 7s\n",
      "400:\ttest: 0.7124937\ttest1: 0.6751649\tbest: 0.6751649 (400)\ttotal: 7m 10s\tremaining: 46m 32s\n",
      "500:\ttest: 0.7153664\ttest1: 0.6761197\tbest: 0.6761197 (500)\ttotal: 8m 59s\tremaining: 44m 51s\n",
      "600:\ttest: 0.7181662\ttest1: 0.6767696\tbest: 0.6767776 (599)\ttotal: 10m 47s\tremaining: 43m 5s\n",
      "700:\ttest: 0.7200585\ttest1: 0.6772081\tbest: 0.6772081 (700)\ttotal: 12m 36s\tremaining: 41m 20s\n",
      "800:\ttest: 0.7220448\ttest1: 0.6776259\tbest: 0.6776259 (800)\ttotal: 14m 26s\tremaining: 39m 37s\n",
      "900:\ttest: 0.7238743\ttest1: 0.6779664\tbest: 0.6779664 (899)\ttotal: 16m 15s\tremaining: 37m 52s\n",
      "1000:\ttest: 0.7254875\ttest1: 0.6782294\tbest: 0.6782387 (991)\ttotal: 18m 7s\tremaining: 36m 10s\n",
      "1100:\ttest: 0.7270149\ttest1: 0.6784862\tbest: 0.6784924 (1099)\ttotal: 19m 57s\tremaining: 34m 25s\n",
      "1200:\ttest: 0.7283432\ttest1: 0.6786890\tbest: 0.6786895 (1193)\ttotal: 21m 48s\tremaining: 32m 39s\n",
      "1300:\ttest: 0.7298933\ttest1: 0.6788012\tbest: 0.6788102 (1296)\ttotal: 23m 40s\tremaining: 30m 54s\n",
      "1400:\ttest: 0.7311990\ttest1: 0.6789832\tbest: 0.6789916 (1397)\ttotal: 25m 32s\tremaining: 29m 9s\n",
      "1500:\ttest: 0.7324833\ttest1: 0.6791462\tbest: 0.6791492 (1498)\ttotal: 27m 24s\tremaining: 27m 22s\n",
      "1600:\ttest: 0.7338910\ttest1: 0.6792541\tbest: 0.6792552 (1599)\ttotal: 29m 16s\tremaining: 25m 35s\n",
      "1700:\ttest: 0.7350924\ttest1: 0.6794064\tbest: 0.6794139 (1698)\ttotal: 31m 11s\tremaining: 23m 49s\n",
      "1800:\ttest: 0.7364752\ttest1: 0.6795302\tbest: 0.6795376 (1794)\ttotal: 33m 5s\tremaining: 22m 1s\n",
      "1900:\ttest: 0.7377443\ttest1: 0.6796490\tbest: 0.6796540 (1897)\ttotal: 34m 50s\tremaining: 20m 8s\n",
      "2000:\ttest: 0.7388384\ttest1: 0.6797035\tbest: 0.6797237 (1978)\ttotal: 36m 43s\tremaining: 18m 20s\n",
      "2100:\ttest: 0.7402475\ttest1: 0.6797466\tbest: 0.6797466 (2100)\ttotal: 38m 37s\tremaining: 16m 31s\n",
      "2200:\ttest: 0.7414329\ttest1: 0.6798616\tbest: 0.6798631 (2199)\ttotal: 40m 32s\tremaining: 14m 42s\n",
      "2300:\ttest: 0.7425011\ttest1: 0.6798803\tbest: 0.6798933 (2294)\ttotal: 42m 26s\tremaining: 12m 53s\n",
      "2400:\ttest: 0.7434070\ttest1: 0.6798968\tbest: 0.6799099 (2373)\ttotal: 44m 21s\tremaining: 11m 3s\n",
      "2500:\ttest: 0.7444364\ttest1: 0.6799454\tbest: 0.6799464 (2499)\ttotal: 46m 16s\tremaining: 9m 13s\n",
      "2600:\ttest: 0.7455244\ttest1: 0.6799713\tbest: 0.6799719 (2595)\ttotal: 48m 11s\tremaining: 7m 23s\n",
      "2700:\ttest: 0.7463090\ttest1: 0.6799123\tbest: 0.6799969 (2617)\ttotal: 50m 8s\tremaining: 5m 33s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6799969219\n",
      "bestIteration = 2617\n",
      "\n",
      "Shrink model to first 2618 iterations.\n"
     ]
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "from catboost import CatBoostClassifier\n",
    "from sklearn.metrics import accuracy_score,confusion_matrix,roc_auc_score\n",
    "m=CatBoostClassifier(n_estimators=3000,random_state=1994,eval_metric='AUC',max_depth=6,learning_rate=0.029,od_wait=50\n",
    "                     ,l2_leaf_reg=5,cat_features=categorical_features_indices,bagging_temperature=0.85,random_strength=100,\n",
    "                     use_best_model=True)\n",
    "m.fit(X_train,y_train,eval_set=[(X_train,y_train),(X_val, y_val)], early_stopping_rounds=100,verbose=100)\n",
    "p2=m.predict_proba(X_val)[:,-1]\n",
    "\n",
    "########### gave 68.00"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "n_estimators=3000,random_state=1994,eval_metric='AUC',max_depth=7,learning_rate=0.029,od_wait=50\n",
    "                     ,l2_leaf_reg=10,cat_features=categorical_features_indices,bagging_temperature=0.8,random_strength=100,\n",
    "                     use_best_model=True\n",
    "\n",
    "0:\ttest: 0.6298164\ttest1: 0.6206030\tbest: 0.6206030 (0)\ttotal: 1.51s\tremaining: 1h 15m 19s\n",
    "100:\ttest: 0.6967200\ttest1: 0.6657370\tbest: 0.6657370 (100)\ttotal: 2m 33s\tremaining: 1h 13m 20s\n",
    "200:\ttest: 0.7073686\ttest1: 0.6716921\tbest: 0.6716921 (200)\ttotal: 5m 21s\tremaining: 1h 14m 33s\n",
    "300:\ttest: 0.7132705\ttest1: 0.6740105\tbest: 0.6740105 (300)\ttotal: 7m 58s\tremaining: 1h 11m 26s\n",
    "400:\ttest: 0.7176570\ttest1: 0.6754946\tbest: 0.6754946 (400)\ttotal: 10m 44s\tremaining: 1h 9m 39s\n",
    "500:\ttest: 0.7209912\ttest1: 0.6764125\tbest: 0.6764168 (499)\ttotal: 13m 35s\tremaining: 1h 7m 49s\n",
    "600:\ttest: 0.7242409\ttest1: 0.6771109\tbest: 0.6771154 (599)\ttotal: 16m 23s\tremaining: 1h 5m 24s\n",
    "700:\ttest: 0.7267972\ttest1: 0.6775394\tbest: 0.6775394 (700)\ttotal: 19m 2s\tremaining: 1h 2m 26s\n",
    "800:\ttest: 0.7291595\ttest1: 0.6779639\tbest: 0.6779705 (799)\ttotal: 21m 47s\tremaining: 59m 48s\n",
    "900:\ttest: 0.7314468\ttest1: 0.6783770\tbest: 0.6783815 (899)\ttotal: 24m 34s\tremaining: 57m 15s\n",
    "1000:\ttest: 0.7334292\ttest1: 0.6784995\tbest: 0.6785201 (973)\ttotal: 27m 19s\tremaining: 54m 34s\n",
    "1100:\ttest: 0.7354691\ttest1: 0.6787310\tbest: 0.6787310 (1100)\ttotal: 30m 8s\tremaining: 51m 59s\n",
    "1200:\ttest: 0.7375465\ttest1: 0.6789353\tbest: 0.6789427 (1198)\ttotal: 32m 45s\tremaining: 49m 4s\n",
    "1300:\ttest: 0.7395440\ttest1: 0.6791223\tbest: 0.6791410 (1272)\ttotal: 35m 2s\tremaining: 45m 45s\n",
    "1400:\ttest: 0.7414093\ttest1: 0.6792434\tbest: 0.6792525 (1395)\ttotal: 37m 19s\tremaining: 42m 36s\n",
    "1500:\ttest: 0.7429491\ttest1: 0.6794451\tbest: 0.6794451 (1500)\ttotal: 39m 37s\tremaining: 39m 34s\n",
    "1600:\ttest: 0.7446167\ttest1: 0.6794840\tbest: 0.6795185 (1559)\ttotal: 41m 58s\tremaining: 36m 40s\n",
    "1700:\ttest: 0.7461050\ttest1: 0.6796050\tbest: 0.6796050 (1700)\ttotal: 44m 17s\tremaining: 33m 49s\n",
    "1800:\ttest: 0.7479902\ttest1: 0.6797087\tbest: 0.6797108 (1798)\ttotal: 46m 38s\tremaining: 31m 2s\n",
    "1900:\ttest: 0.7494178\ttest1: 0.6798259\tbest: 0.6798259 (1900)\ttotal: 48m 56s\tremaining: 28m 17s\n",
    "2000:\ttest: 0.7508585\ttest1: 0.6798631\tbest: 0.6799069 (1952)\ttotal: 51m 15s\tremaining: 25m 35s\n",
    "Stopped by overfitting detector  (100 iterations wait)\n",
    "\n",
    "bestTest = 0.6799069447\n",
    "bestIteration = 1952\n",
    "\n",
    "Shrink model to first 1953 iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\ttest: 0.6257002\ttest1: 0.6175802\tbest: 0.6175802 (0)\ttotal: 1.83s\tremaining: 1h 31m 17s\n",
      "100:\ttest: 0.6861999\ttest1: 0.6596716\tbest: 0.6596716 (100)\ttotal: 2m 29s\tremaining: 1h 11m 40s\n",
      "200:\ttest: 0.6984332\ttest1: 0.6675033\tbest: 0.6675033 (200)\ttotal: 5m\tremaining: 1h 9m 42s\n",
      "300:\ttest: 0.7034418\ttest1: 0.6708843\tbest: 0.6708843 (300)\ttotal: 7m 33s\tremaining: 1h 7m 44s\n",
      "400:\ttest: 0.7071229\ttest1: 0.6729311\tbest: 0.6729311 (400)\ttotal: 10m 36s\tremaining: 1h 8m 44s\n",
      "500:\ttest: 0.7099471\ttest1: 0.6741656\tbest: 0.6741656 (500)\ttotal: 14m 30s\tremaining: 1h 12m 22s\n",
      "600:\ttest: 0.7122577\ttest1: 0.6750352\tbest: 0.6750352 (600)\ttotal: 18m 27s\tremaining: 1h 13m 39s\n",
      "700:\ttest: 0.7144558\ttest1: 0.6758022\tbest: 0.6758022 (700)\ttotal: 22m 21s\tremaining: 1h 13m 19s\n",
      "800:\ttest: 0.7162215\ttest1: 0.6762848\tbest: 0.6762848 (800)\ttotal: 26m 14s\tremaining: 1h 12m 2s\n",
      "900:\ttest: 0.7176287\ttest1: 0.6766625\tbest: 0.6766633 (899)\ttotal: 30m 10s\tremaining: 1h 10m 16s\n",
      "1000:\ttest: 0.7191127\ttest1: 0.6770818\tbest: 0.6770818 (1000)\ttotal: 34m 6s\tremaining: 1h 8m 7s\n",
      "1100:\ttest: 0.7204655\ttest1: 0.6773460\tbest: 0.6773460 (1100)\ttotal: 38m 3s\tremaining: 1h 5m 38s\n",
      "1200:\ttest: 0.7217381\ttest1: 0.6776074\tbest: 0.6776074 (1200)\ttotal: 42m\tremaining: 1h 2m 55s\n",
      "1300:\ttest: 0.7232869\ttest1: 0.6778722\tbest: 0.6778755 (1299)\ttotal: 45m 57s\tremaining: 1h 1s\n",
      "1400:\ttest: 0.7244563\ttest1: 0.6781020\tbest: 0.6781039 (1395)\ttotal: 49m 57s\tremaining: 57m 1s\n",
      "1500:\ttest: 0.7254924\ttest1: 0.6783357\tbest: 0.6783357 (1500)\ttotal: 53m 56s\tremaining: 53m 52s\n",
      "1600:\ttest: 0.7267701\ttest1: 0.6785301\tbest: 0.6785311 (1597)\ttotal: 57m 57s\tremaining: 50m 38s\n",
      "1700:\ttest: 0.7279491\ttest1: 0.6787137\tbest: 0.6787187 (1695)\ttotal: 1h 1m 55s\tremaining: 47m 17s\n",
      "1800:\ttest: 0.7291158\ttest1: 0.6789020\tbest: 0.6789020 (1800)\ttotal: 1h 5m 54s\tremaining: 43m 52s\n",
      "1900:\ttest: 0.7299631\ttest1: 0.6790135\tbest: 0.6790184 (1887)\ttotal: 1h 9m 54s\tremaining: 40m 25s\n",
      "2000:\ttest: 0.7308862\ttest1: 0.6791179\tbest: 0.6791218 (1991)\ttotal: 1h 13m 54s\tremaining: 36m 53s\n",
      "2100:\ttest: 0.7317408\ttest1: 0.6792547\tbest: 0.6792626 (2098)\ttotal: 1h 17m 56s\tremaining: 33m 21s\n",
      "2200:\ttest: 0.7322979\ttest1: 0.6793683\tbest: 0.6793774 (2186)\ttotal: 1h 21m 56s\tremaining: 29m 44s\n",
      "2300:\ttest: 0.7330105\ttest1: 0.6794613\tbest: 0.6794613 (2300)\ttotal: 1h 25m 55s\tremaining: 26m 6s\n",
      "2400:\ttest: 0.7337666\ttest1: 0.6794959\tbest: 0.6794959 (2400)\ttotal: 1h 29m 58s\tremaining: 22m 26s\n",
      "2500:\ttest: 0.7347394\ttest1: 0.6795842\tbest: 0.6795842 (2500)\ttotal: 1h 34m\tremaining: 18m 45s\n",
      "2600:\ttest: 0.7355500\ttest1: 0.6797019\tbest: 0.6797019 (2600)\ttotal: 1h 38m 3s\tremaining: 15m 2s\n",
      "2700:\ttest: 0.7364411\ttest1: 0.6797377\tbest: 0.6797377 (2700)\ttotal: 1h 42m 3s\tremaining: 11m 17s\n",
      "2800:\ttest: 0.7375097\ttest1: 0.6797602\tbest: 0.6797660 (2781)\ttotal: 1h 45m 55s\tremaining: 7m 31s\n",
      "2900:\ttest: 0.7384537\ttest1: 0.6798284\tbest: 0.6798303 (2898)\ttotal: 1h 48m 44s\tremaining: 3m 42s\n",
      "2999:\ttest: 0.7391626\ttest1: 0.6799165\tbest: 0.6799171 (2998)\ttotal: 1h 51m 32s\tremaining: 0us\n",
      "\n",
      "bestTest = 0.6799170818\n",
      "bestIteration = 2998\n",
      "\n",
      "Shrink model to first 2999 iterations.\n"
     ]
    }
   ],
   "source": [
    "# import matplotlib.pyplot as plt\n",
    "# %matplotlib inline\n",
    "# from catboost import CatBoostClassifier\n",
    "# from sklearn.metrics import accuracy_score,confusion_matrix,roc_auc_score\n",
    "# m=CatBoostClassifier(n_estimators=3000,random_state=1994,eval_metric='AUC',max_depth=6,learning_rate=0.029,od_wait=50\n",
    "#                      ,l2_leaf_reg=10,cat_features=categorical_features_indices,bagging_temperature=0.85,random_strength=100,\n",
    "#                      use_best_model=True)\n",
    "# m.fit(X_train,y_train,eval_set=[(X_train,y_train),(X_val, y_val)], early_stopping_rounds=100,verbose=100)\n",
    "# p2=m.predict_proba(X_val)[:,-1]\n",
    "\n",
    "########### gave 68.00"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "from catboost import CatBoostClassifier\n",
    "from sklearn.metrics import accuracy_score,confusion_matrix,roc_auc_score\n",
    "m=CatBoostClassifier(n_estimators=3000,random_state=1994,eval_metric='AUC',max_depth=6,learning_rate=0.05,od_wait=50\n",
    "                     ,l2_leaf_reg=10,cat_features=categorical_features_indices,bagging_temperature=1,random_strength=100,\n",
    "                     use_best_model=True)\n",
    "m.fit(X_train,y_train,eval_set=[(X_train,y_train),(X_val, y_val)], early_stopping_rounds=100,verbose=100)\n",
    "p2=m.predict_proba(X_val)[:,-1]\n",
    "\n",
    "0:\ttest: 0.6238759\ttest1: 0.6166456\tbest: 0.6166456 (0)\ttotal: 1.08s\tremaining: 54m 7s\n",
    "100:\ttest: 0.7005969\ttest1: 0.6690637\tbest: 0.6690637 (100)\ttotal: 1m 35s\tremaining: 45m 28s\n",
    "200:\ttest: 0.7094896\ttest1: 0.6736557\tbest: 0.6736557 (200)\ttotal: 2m 39s\tremaining: 37m\n",
    "300:\ttest: 0.7143557\ttest1: 0.6756185\tbest: 0.6756235 (299)\ttotal: 3m 59s\tremaining: 35m 45s\n",
    "400:\ttest: 0.7180776\ttest1: 0.6767544\tbest: 0.6767544 (400)\ttotal: 5m 29s\tremaining: 35m 36s\n",
    "500:\ttest: 0.7209041\ttest1: 0.6773625\tbest: 0.6773625 (500)\ttotal: 7m 7s\tremaining: 35m 34s\n",
    "600:\ttest: 0.7235243\ttest1: 0.6779699\tbest: 0.6779718 (599)\ttotal: 8m 46s\tremaining: 35m 2s\n",
    "700:\ttest: 0.7265405\ttest1: 0.6783837\tbest: 0.6784066 (694)\ttotal: 10m 25s\tremaining: 34m 11s\n",
    "800:\ttest: 0.7292407\ttest1: 0.6785872\tbest: 0.6786010 (796)\ttotal: 12m 6s\tremaining: 33m 14s\n",
    "900:\ttest: 0.7318031\ttest1: 0.6788393\tbest: 0.6788449 (899)\ttotal: 13m 46s\tremaining: 32m 6s\n",
    "1000:\ttest: 0.7341236\ttest1: 0.6791289\tbest: 0.6791357 (993)\ttotal: 15m 26s\tremaining: 30m 49s\n",
    "1100:\ttest: 0.7360354\ttest1: 0.6792755\tbest: 0.6792755 (1100)\ttotal: 17m 5s\tremaining: 29m 28s\n",
    "1200:\ttest: 0.7378263\ttest1: 0.6793381\tbest: 0.6793531 (1199)\ttotal: 18m 43s\tremaining: 28m 3s\n",
    "1300:\ttest: 0.7397921\ttest1: 0.6795331\tbest: 0.6795331 (1300)\ttotal: 20m 19s\tremaining: 26m 33s\n",
    "1400:\ttest: 0.7417990\ttest1: 0.6796844\tbest: 0.6796990 (1392)\ttotal: 21m 58s\tremaining: 25m 4s\n",
    "1500:\ttest: 0.7435731\ttest1: 0.6798866\tbest: 0.6799281 (1493)\ttotal: 23m 37s\tremaining: 23m 35s\n",
    "1600:\ttest: 0.7453242\ttest1: 0.6799940\tbest: 0.6800087 (1599)\ttotal: 25m 16s\tremaining: 22m 5s\n",
    "1700:\ttest: 0.7468000\ttest1: 0.6800143\tbest: 0.6800710 (1651)\ttotal: 26m 54s\tremaining: 20m 33s\n",
    "Stopped by overfitting detector  (100 iterations wait)\n",
    "\n",
    "bestTest = 0.6800709593\n",
    "bestIteration = 1651\n",
    "\n",
    "Shrink model to first 1652 iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['AVERAGE.ACCT.AGE', 'Aadhar_flag', 'CREDIT.HISTORY.LENGTH',\n",
       "       'Current_pincode_ID', 'DELINQUENT.ACCTS.IN.LAST.SIX.MONTHS',\n",
       "       'Driving_flag', 'Employee_code_ID', 'Employment.Type',\n",
       "       'NEW.ACCTS.IN.LAST.SIX.MONTHS', 'NO.OF_INQUIRIES', 'PAN_flag',\n",
       "       'PERFORM_CNS.SCORE', 'PERFORM_CNS.SCORE.DESCRIPTION',\n",
       "       'PRI.ACTIVE.ACCTS', 'PRI.CURRENT.BALANCE', 'PRI.DISBURSED.AMOUNT',\n",
       "       'PRI.NO.OF.ACCTS', 'PRI.OVERDUE.ACCTS', 'PRI.SANCTIONED.AMOUNT',\n",
       "       'PRIMARY.INSTAL.AMT', 'Passport_flag', 'SEC.ACTIVE.ACCTS',\n",
       "       'SEC.CURRENT.BALANCE', 'SEC.DISBURSED.AMOUNT', 'SEC.INSTAL.AMT',\n",
       "       'SEC.NO.OF.ACCTS', 'SEC.OVERDUE.ACCTS', 'SEC.SANCTIONED.AMOUNT',\n",
       "       'State_ID', 'VoterID_flag', 'asset_cost', 'branch_id',\n",
       "       'disbursed_amount', 'loan_default', 'ltv', 'manufacturer_id',\n",
       "       'supplier_id', 'age', 'acct_age', 'credit_history_len',\n",
       "       'credit_history_len_before_sixM', 'dob_y', 'dob_m', 'dob_d', 'dob_w',\n",
       "       'disb_m', 'disb_d', 'disb_w', 'PERFORM_CNS.SCORE.DESCRIPTION.APLHS',\n",
       "       'PERFORM_CNS.SCORE.BINS', 'ltv.BINS', 'Not_Scored', 'Very_Low',\n",
       "       'Very_High', 'No_History', 'ACTIVE.ACCTS', 'CURRENT.BALANCE',\n",
       "       'DISBURSED.AMOUNT', 'NO.OF.ACCTS', 'OVERDUE.ACCTS', 'SANCTIONED.AMOUNT',\n",
       "       'INSTAL.AMT', 'SANCTION_DISBURSED', 'NO_DEACTIVE_ACCOUNTS',\n",
       "       'NO.OF.ACC.BEF.SIX.MONTH', 'OVERDUE.ACC.BEF.SIX.MONTHS', 'CLEAN.ACC',\n",
       "       'asset_value', 'value_cost', 'value_per_cost', 'extra_finance',\n",
       "       'asset_disburse', 'sixmmonths_dfault', 'AVERAGE.ACCT.AGE.YEAR',\n",
       "       'AVERAGE.ACCT.AGE.MONTH', 'CREDIT.HISTORY.LENGTH.YEAR',\n",
       "       'CREDIT.HISTORY.LENGTH.MONTH', 'Total_Flag', 'Branch_Emp_code',\n",
       "       'Pin_Emp_code'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(6.047410210517497, 'branch_id'),\n",
       " (5.787161448462624, 'asset_value'),\n",
       " (4.994259419681581, 'Employee_code_ID'),\n",
       " (4.312546992571475, 'supplier_id'),\n",
       " (3.72534727762862, 'dob_y'),\n",
       " (3.3864781168895046, 'Current_pincode_ID'),\n",
       " (3.267800040923283, 'ltv'),\n",
       " (3.1403483869080873, 'disb_d'),\n",
       " (3.0213980109815943, 'PERFORM_CNS.SCORE'),\n",
       " (2.990357229114383, 'extra_finance'),\n",
       " (2.680491400355188, 'value_cost'),\n",
       " (2.546525675276578, 'Employment.Type'),\n",
       " (2.531500881865133, 'disb_w'),\n",
       " (2.2421724086708004, 'Branch_Emp_code'),\n",
       " (2.2048309859899895, 'manufacturer_id'),\n",
       " (2.0455763646965544, 'State_ID'),\n",
       " (2.0288702476535763, 'dob_d'),\n",
       " (1.9659156720924318, 'Pin_Emp_code'),\n",
       " (1.9040263829153796, 'asset_cost'),\n",
       " (1.8898962924176472, 'CLEAN.ACC'),\n",
       " (1.8697323133686394, 'NO.OF_INQUIRIES'),\n",
       " (1.7532643125188854, 'PERFORM_CNS.SCORE.DESCRIPTION'),\n",
       " (1.6727839421650463, 'SANCTION_DISBURSED'),\n",
       " (1.612082124979268, 'disbursed_amount'),\n",
       " (1.5939762213367175, 'asset_disburse'),\n",
       " (1.5856357241341692, 'PRIMARY.INSTAL.AMT'),\n",
       " (1.396636034566479, 'SANCTIONED.AMOUNT'),\n",
       " (1.3450672183582506, 'value_per_cost'),\n",
       " (1.264770639405138, 'DELINQUENT.ACCTS.IN.LAST.SIX.MONTHS'),\n",
       " (1.2431966465070765, 'dob_w'),\n",
       " (1.182777701043443, 'PERFORM_CNS.SCORE.DESCRIPTION.APLHS'),\n",
       " (1.1397815145394976, 'disb_m'),\n",
       " (1.1147065020551328, 'PERFORM_CNS.SCORE.BINS'),\n",
       " (1.057199943478816, 'acct_age'),\n",
       " (1.0370380504951058, 'INSTAL.AMT'),\n",
       " (0.9708859017382261, 'credit_history_len'),\n",
       " (0.9202557953164427, 'age'),\n",
       " (0.9094142487853252, 'DISBURSED.AMOUNT'),\n",
       " (0.7851543935732402, 'PRI.CURRENT.BALANCE'),\n",
       " (0.7688978077764699, 'sixmmonths_dfault'),\n",
       " (0.7172965088142798, 'PRI.SANCTIONED.AMOUNT'),\n",
       " (0.7015428013385574, 'credit_history_len_before_sixM'),\n",
       " (0.6806583802671465, 'PRI.OVERDUE.ACCTS'),\n",
       " (0.677636482575073, 'PRI.DISBURSED.AMOUNT'),\n",
       " (0.673002438772955, 'dob_m'),\n",
       " (0.6511123561681275, 'CURRENT.BALANCE'),\n",
       " (0.5723560568377809, 'Total_Flag'),\n",
       " (0.5702334251628453, 'AVERAGE.ACCT.AGE.MONTH'),\n",
       " (0.504430208137175, 'ltv.BINS'),\n",
       " (0.4942293863784774, 'AVERAGE.ACCT.AGE'),\n",
       " (0.4776384527558254, 'CREDIT.HISTORY.LENGTH'),\n",
       " (0.4643665055113698, 'OVERDUE.ACC.BEF.SIX.MONTHS'),\n",
       " (0.46218089083855957, 'VoterID_flag'),\n",
       " (0.4254817811782662, 'NO_DEACTIVE_ACCOUNTS'),\n",
       " (0.4121799960633866, 'CREDIT.HISTORY.LENGTH.MONTH'),\n",
       " (0.36184826890093974, 'PRI.NO.OF.ACCTS'),\n",
       " (0.34083304306968065, 'NEW.ACCTS.IN.LAST.SIX.MONTHS'),\n",
       " (0.33025644380247, 'ACTIVE.ACCTS'),\n",
       " (0.31562483297596156, 'PRI.ACTIVE.ACCTS'),\n",
       " (0.290839034453973, 'OVERDUE.ACCTS'),\n",
       " (0.2739829422762936, 'NO.OF.ACC.BEF.SIX.MONTH'),\n",
       " (0.2677836575009171, 'NO.OF.ACCTS'),\n",
       " (0.16608296060963493, 'SEC.NO.OF.ACCTS'),\n",
       " (0.16194097370801713, 'SEC.SANCTIONED.AMOUNT'),\n",
       " (0.1570007055347178, 'Aadhar_flag'),\n",
       " (0.15454933334042334, 'SEC.INSTAL.AMT'),\n",
       " (0.14448339865159235, 'Not_Scored'),\n",
       " (0.11411397581092553, 'Very_High'),\n",
       " (0.10172068404943535, 'SEC.CURRENT.BALANCE'),\n",
       " (0.08905999514431655, 'SEC.ACTIVE.ACCTS'),\n",
       " (0.08353649753655877, 'PAN_flag'),\n",
       " (0.07250385703438575, 'CREDIT.HISTORY.LENGTH.YEAR'),\n",
       " (0.04274804486798096, 'AVERAGE.ACCT.AGE.YEAR'),\n",
       " (0.036352502297052953, 'SEC.DISBURSED.AMOUNT'),\n",
       " (0.030461165544757397, 'SEC.OVERDUE.ACCTS'),\n",
       " (0.025805769142637797, 'Driving_flag'),\n",
       " (0.01795576119422002, 'Passport_flag'),\n",
       " (0.0, 'Very_Low'),\n",
       " (0.0, 'No_History')]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(zip(m.feature_importances_,X_train),reverse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Acer\\Anaconda3\\lib\\site-packages\\pandas\\core\\frame.py:3940: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy\n",
      "  errors=errors)\n"
     ]
    }
   ],
   "source": [
    "dftest.drop('loan_default',axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 20- fold Cross Validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\ttest: 0.6255709\ttest1: 0.6235215\tbest: 0.6235215 (0)\ttotal: 617ms\tremaining: 30m 51s\n",
      "100:\ttest: 0.6912090\ttest1: 0.6663184\tbest: 0.6663184 (100)\ttotal: 1m 4s\tremaining: 30m 44s\n",
      "200:\ttest: 0.6996197\ttest1: 0.6710377\tbest: 0.6711387 (196)\ttotal: 2m 15s\tremaining: 31m 25s\n",
      "300:\ttest: 0.7048390\ttest1: 0.6731636\tbest: 0.6731636 (300)\ttotal: 3m 25s\tremaining: 30m 44s\n",
      "400:\ttest: 0.7080067\ttest1: 0.6743669\tbest: 0.6743669 (400)\ttotal: 4m 36s\tremaining: 29m 53s\n",
      "500:\ttest: 0.7102995\ttest1: 0.6752153\tbest: 0.6752209 (491)\ttotal: 5m 47s\tremaining: 28m 53s\n",
      "600:\ttest: 0.7122254\ttest1: 0.6756048\tbest: 0.6756048 (600)\ttotal: 6m 58s\tremaining: 27m 51s\n",
      "700:\ttest: 0.7143447\ttest1: 0.6761486\tbest: 0.6761524 (699)\ttotal: 8m 10s\tremaining: 26m 49s\n",
      "800:\ttest: 0.7161435\ttest1: 0.6765585\tbest: 0.6765585 (800)\ttotal: 9m 24s\tremaining: 25m 48s\n",
      "900:\ttest: 0.7176593\ttest1: 0.6769492\tbest: 0.6769492 (900)\ttotal: 10m 38s\tremaining: 24m 46s\n",
      "1000:\ttest: 0.7189799\ttest1: 0.6771910\tbest: 0.6771910 (1000)\ttotal: 11m 51s\tremaining: 23m 40s\n",
      "1100:\ttest: 0.7204417\ttest1: 0.6775184\tbest: 0.6775388 (1097)\ttotal: 13m 4s\tremaining: 22m 33s\n",
      "1200:\ttest: 0.7216731\ttest1: 0.6779027\tbest: 0.6779180 (1199)\ttotal: 14m 19s\tremaining: 21m 27s\n",
      "1300:\ttest: 0.7228720\ttest1: 0.6780715\tbest: 0.6780993 (1287)\ttotal: 15m 35s\tremaining: 20m 21s\n",
      "1400:\ttest: 0.7236892\ttest1: 0.6782556\tbest: 0.6782598 (1397)\ttotal: 16m 50s\tremaining: 19m 12s\n",
      "1500:\ttest: 0.7250784\ttest1: 0.6784203\tbest: 0.6784398 (1482)\ttotal: 18m 5s\tremaining: 18m 3s\n",
      "1600:\ttest: 0.7259855\ttest1: 0.6786090\tbest: 0.6786244 (1598)\ttotal: 19m 20s\tremaining: 16m 53s\n",
      "1700:\ttest: 0.7269920\ttest1: 0.6785874\tbest: 0.6786586 (1611)\ttotal: 20m 35s\tremaining: 15m 43s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6786586468\n",
      "bestIteration = 1611\n",
      "\n",
      "Shrink model to first 1612 iterations.\n",
      "err:  0.6786586468024367\n",
      "0:\ttest: 0.6228713\ttest1: 0.6195651\tbest: 0.6195651 (0)\ttotal: 600ms\tremaining: 29m 58s\n",
      "100:\ttest: 0.6913826\ttest1: 0.6682894\tbest: 0.6682894 (100)\ttotal: 1m 8s\tremaining: 32m 45s\n",
      "200:\ttest: 0.6997731\ttest1: 0.6752219\tbest: 0.6752219 (200)\ttotal: 2m 17s\tremaining: 31m 56s\n",
      "300:\ttest: 0.7042469\ttest1: 0.6785483\tbest: 0.6785483 (300)\ttotal: 3m 26s\tremaining: 30m 50s\n",
      "400:\ttest: 0.7077048\ttest1: 0.6804545\tbest: 0.6804574 (399)\ttotal: 4m 37s\tremaining: 29m 57s\n",
      "500:\ttest: 0.7099489\ttest1: 0.6815924\tbest: 0.6815924 (500)\ttotal: 5m 48s\tremaining: 28m 59s\n",
      "600:\ttest: 0.7121179\ttest1: 0.6823510\tbest: 0.6823663 (597)\ttotal: 7m 1s\tremaining: 28m\n",
      "700:\ttest: 0.7138255\ttest1: 0.6830116\tbest: 0.6830207 (696)\ttotal: 8m 13s\tremaining: 26m 58s\n",
      "800:\ttest: 0.7154276\ttest1: 0.6835545\tbest: 0.6835545 (800)\ttotal: 9m 26s\tremaining: 25m 54s\n",
      "900:\ttest: 0.7168601\ttest1: 0.6839613\tbest: 0.6839613 (900)\ttotal: 10m 39s\tremaining: 24m 48s\n",
      "1000:\ttest: 0.7181435\ttest1: 0.6841946\tbest: 0.6842037 (999)\ttotal: 11m 52s\tremaining: 23m 43s\n",
      "1100:\ttest: 0.7192711\ttest1: 0.6842007\tbest: 0.6842262 (1004)\ttotal: 13m 6s\tremaining: 22m 36s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6842261566\n",
      "bestIteration = 1004\n",
      "\n",
      "Shrink model to first 1005 iterations.\n",
      "err:  0.6842261565700131\n",
      "0:\ttest: 0.6229886\ttest1: 0.6157547\tbest: 0.6157547 (0)\ttotal: 615ms\tremaining: 30m 43s\n",
      "100:\ttest: 0.6914591\ttest1: 0.6683216\tbest: 0.6683216 (100)\ttotal: 1m 16s\tremaining: 36m 48s\n",
      "200:\ttest: 0.6992037\ttest1: 0.6750126\tbest: 0.6750126 (200)\ttotal: 2m 32s\tremaining: 35m 26s\n",
      "300:\ttest: 0.7040436\ttest1: 0.6778805\tbest: 0.6778805 (300)\ttotal: 3m 42s\tremaining: 33m 14s\n",
      "400:\ttest: 0.7072440\ttest1: 0.6794842\tbest: 0.6795496 (396)\ttotal: 4m 52s\tremaining: 31m 37s\n",
      "500:\ttest: 0.7098809\ttest1: 0.6805986\tbest: 0.6805986 (500)\ttotal: 6m 3s\tremaining: 30m 13s\n",
      "600:\ttest: 0.7124579\ttest1: 0.6813832\tbest: 0.6813832 (600)\ttotal: 7m 16s\tremaining: 29m 1s\n",
      "700:\ttest: 0.7150306\ttest1: 0.6820066\tbest: 0.6820075 (696)\ttotal: 8m 28s\tremaining: 27m 47s\n",
      "800:\ttest: 0.7169955\ttest1: 0.6823814\tbest: 0.6823814 (800)\ttotal: 9m 41s\tremaining: 26m 35s\n",
      "900:\ttest: 0.7186841\ttest1: 0.6828834\tbest: 0.6829101 (894)\ttotal: 10m 54s\tremaining: 25m 25s\n",
      "1000:\ttest: 0.7203683\ttest1: 0.6830135\tbest: 0.6830352 (987)\ttotal: 12m 19s\tremaining: 24m 36s\n",
      "1100:\ttest: 0.7215752\ttest1: 0.6833708\tbest: 0.6833730 (1099)\ttotal: 13m 33s\tremaining: 23m 23s\n",
      "1200:\ttest: 0.7229904\ttest1: 0.6837289\tbest: 0.6837289 (1200)\ttotal: 14m 47s\tremaining: 22m 9s\n",
      "1300:\ttest: 0.7241052\ttest1: 0.6839327\tbest: 0.6839327 (1300)\ttotal: 16m 1s\tremaining: 20m 56s\n",
      "1400:\ttest: 0.7252672\ttest1: 0.6841614\tbest: 0.6841614 (1400)\ttotal: 17m 18s\tremaining: 19m 45s\n",
      "1500:\ttest: 0.7261847\ttest1: 0.6841793\tbest: 0.6842413 (1444)\ttotal: 18m 31s\tremaining: 18m 29s\n",
      "1600:\ttest: 0.7272169\ttest1: 0.6842836\tbest: 0.6842850 (1582)\ttotal: 19m 46s\tremaining: 17m 16s\n",
      "1700:\ttest: 0.7282224\ttest1: 0.6843832\tbest: 0.6843908 (1688)\ttotal: 21m\tremaining: 16m 2s\n",
      "1800:\ttest: 0.7290798\ttest1: 0.6844621\tbest: 0.6844726 (1796)\ttotal: 22m 18s\tremaining: 14m 51s\n",
      "1900:\ttest: 0.7300609\ttest1: 0.6844805\tbest: 0.6845871 (1858)\ttotal: 23m 33s\tremaining: 13m 37s\n",
      "2000:\ttest: 0.7309624\ttest1: 0.6845467\tbest: 0.6846442 (1939)\ttotal: 24m 47s\tremaining: 12m 22s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6846441548\n",
      "bestIteration = 1939\n",
      "\n",
      "Shrink model to first 1940 iterations.\n",
      "err:  0.6846441548116243\n",
      "0:\ttest: 0.6234385\ttest1: 0.6130140\tbest: 0.6130140 (0)\ttotal: 606ms\tremaining: 30m 17s\n",
      "100:\ttest: 0.6909422\ttest1: 0.6634531\tbest: 0.6634531 (100)\ttotal: 1m 6s\tremaining: 31m 38s\n",
      "200:\ttest: 0.7007506\ttest1: 0.6698937\tbest: 0.6698937 (200)\ttotal: 2m 14s\tremaining: 31m 13s\n",
      "300:\ttest: 0.7053667\ttest1: 0.6727347\tbest: 0.6727347 (300)\ttotal: 3m 23s\tremaining: 30m 20s\n",
      "400:\ttest: 0.7085797\ttest1: 0.6745104\tbest: 0.6745104 (400)\ttotal: 4m 32s\tremaining: 29m 28s\n",
      "500:\ttest: 0.7111159\ttest1: 0.6756648\tbest: 0.6756648 (500)\ttotal: 5m 44s\tremaining: 28m 36s\n",
      "600:\ttest: 0.7131710\ttest1: 0.6766818\tbest: 0.6766818 (600)\ttotal: 6m 56s\tremaining: 27m 42s\n",
      "700:\ttest: 0.7152707\ttest1: 0.6772654\tbest: 0.6772654 (700)\ttotal: 8m 8s\tremaining: 26m 42s\n",
      "800:\ttest: 0.7170695\ttest1: 0.6779277\tbest: 0.6779277 (800)\ttotal: 9m 22s\tremaining: 25m 44s\n",
      "900:\ttest: 0.7183297\ttest1: 0.6783302\tbest: 0.6783302 (900)\ttotal: 10m 35s\tremaining: 24m 40s\n",
      "1000:\ttest: 0.7194993\ttest1: 0.6785906\tbest: 0.6786032 (999)\ttotal: 11m 48s\tremaining: 23m 34s\n",
      "1100:\ttest: 0.7208086\ttest1: 0.6789744\tbest: 0.6789744 (1100)\ttotal: 13m 1s\tremaining: 22m 28s\n",
      "1200:\ttest: 0.7222438\ttest1: 0.6792644\tbest: 0.6792778 (1197)\ttotal: 14m 15s\tremaining: 21m 21s\n",
      "1300:\ttest: 0.7233298\ttest1: 0.6794654\tbest: 0.6794684 (1291)\ttotal: 15m 28s\tremaining: 20m 13s\n",
      "1400:\ttest: 0.7243792\ttest1: 0.6797508\tbest: 0.6797508 (1400)\ttotal: 16m 42s\tremaining: 19m 4s\n",
      "1500:\ttest: 0.7255983\ttest1: 0.6799975\tbest: 0.6799975 (1500)\ttotal: 17m 56s\tremaining: 17m 55s\n",
      "1600:\ttest: 0.7265774\ttest1: 0.6800948\tbest: 0.6800974 (1598)\ttotal: 19m 12s\tremaining: 16m 47s\n",
      "1700:\ttest: 0.7276253\ttest1: 0.6803033\tbest: 0.6803064 (1691)\ttotal: 20m 27s\tremaining: 15m 37s\n",
      "1800:\ttest: 0.7286874\ttest1: 0.6805540\tbest: 0.6805575 (1793)\ttotal: 21m 42s\tremaining: 14m 26s\n",
      "1900:\ttest: 0.7297789\ttest1: 0.6807274\tbest: 0.6807323 (1899)\ttotal: 22m 57s\tremaining: 13m 16s\n",
      "2000:\ttest: 0.7308218\ttest1: 0.6808348\tbest: 0.6808378 (1955)\ttotal: 24m 13s\tremaining: 12m 5s\n",
      "2100:\ttest: 0.7316558\ttest1: 0.6809912\tbest: 0.6810132 (2085)\ttotal: 25m 28s\tremaining: 10m 54s\n",
      "2200:\ttest: 0.7325682\ttest1: 0.6811786\tbest: 0.6811921 (2188)\ttotal: 26m 43s\tremaining: 9m 42s\n",
      "2300:\ttest: 0.7333412\ttest1: 0.6812351\tbest: 0.6813292 (2274)\ttotal: 27m 59s\tremaining: 8m 30s\n",
      "2400:\ttest: 0.7343594\ttest1: 0.6813981\tbest: 0.6814097 (2399)\ttotal: 29m 16s\tremaining: 7m 18s\n",
      "2500:\ttest: 0.7352866\ttest1: 0.6814227\tbest: 0.6814227 (2500)\ttotal: 30m 31s\tremaining: 6m 5s\n",
      "2600:\ttest: 0.7361799\ttest1: 0.6815484\tbest: 0.6815652 (2593)\ttotal: 31m 48s\tremaining: 4m 52s\n",
      "2700:\ttest: 0.7370400\ttest1: 0.6816752\tbest: 0.6816794 (2696)\ttotal: 33m 5s\tremaining: 3m 39s\n",
      "2800:\ttest: 0.7378688\ttest1: 0.6819328\tbest: 0.6819402 (2777)\ttotal: 34m 20s\tremaining: 2m 26s\n",
      "2900:\ttest: 0.7387581\ttest1: 0.6821185\tbest: 0.6821303 (2897)\ttotal: 35m 36s\tremaining: 1m 12s\n",
      "2999:\ttest: 0.7395797\ttest1: 0.6821672\tbest: 0.6822089 (2954)\ttotal: 36m 52s\tremaining: 0us\n",
      "\n",
      "bestTest = 0.6822089123\n",
      "bestIteration = 2954\n",
      "\n",
      "Shrink model to first 2955 iterations.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "err:  0.6822089123249053\n",
      "0:\ttest: 0.6252943\ttest1: 0.6254041\tbest: 0.6254041 (0)\ttotal: 619ms\tremaining: 30m 56s\n",
      "100:\ttest: 0.6900003\ttest1: 0.6697481\tbest: 0.6697481 (100)\ttotal: 1m 6s\tremaining: 32m 2s\n",
      "200:\ttest: 0.7003300\ttest1: 0.6770288\tbest: 0.6770288 (200)\ttotal: 2m 16s\tremaining: 31m 46s\n",
      "300:\ttest: 0.7049339\ttest1: 0.6800258\tbest: 0.6800286 (299)\ttotal: 3m 25s\tremaining: 30m 46s\n",
      "400:\ttest: 0.7083498\ttest1: 0.6819269\tbest: 0.6819396 (399)\ttotal: 4m 35s\tremaining: 29m 45s\n",
      "500:\ttest: 0.7111396\ttest1: 0.6831513\tbest: 0.6831523 (497)\ttotal: 5m 45s\tremaining: 28m 44s\n",
      "600:\ttest: 0.7131556\ttest1: 0.6839218\tbest: 0.6839259 (595)\ttotal: 6m 56s\tremaining: 27m 41s\n",
      "700:\ttest: 0.7150111\ttest1: 0.6843911\tbest: 0.6843911 (700)\ttotal: 8m 8s\tremaining: 26m 40s\n",
      "800:\ttest: 0.7163601\ttest1: 0.6848077\tbest: 0.6848531 (796)\ttotal: 9m 19s\tremaining: 25m 35s\n",
      "900:\ttest: 0.7178451\ttest1: 0.6851528\tbest: 0.6851528 (900)\ttotal: 10m 30s\tremaining: 24m 29s\n",
      "1000:\ttest: 0.7192884\ttest1: 0.6856108\tbest: 0.6856108 (1000)\ttotal: 11m 42s\tremaining: 23m 23s\n",
      "1100:\ttest: 0.7205646\ttest1: 0.6859939\tbest: 0.6860254 (1092)\ttotal: 12m 55s\tremaining: 22m 17s\n",
      "1200:\ttest: 0.7217220\ttest1: 0.6862425\tbest: 0.6862425 (1200)\ttotal: 14m 8s\tremaining: 21m 11s\n",
      "1300:\ttest: 0.7229635\ttest1: 0.6864624\tbest: 0.6864776 (1298)\ttotal: 15m 22s\tremaining: 20m 5s\n",
      "1400:\ttest: 0.7239092\ttest1: 0.6866222\tbest: 0.6866452 (1396)\ttotal: 16m 38s\tremaining: 18m 59s\n",
      "1500:\ttest: 0.7250586\ttest1: 0.6867666\tbest: 0.6867896 (1496)\ttotal: 17m 51s\tremaining: 17m 49s\n",
      "1600:\ttest: 0.7260018\ttest1: 0.6869591\tbest: 0.6869659 (1599)\ttotal: 19m 5s\tremaining: 16m 40s\n",
      "1700:\ttest: 0.7271042\ttest1: 0.6871426\tbest: 0.6871464 (1699)\ttotal: 20m 18s\tremaining: 15m 30s\n",
      "1800:\ttest: 0.7279844\ttest1: 0.6870575\tbest: 0.6871541 (1704)\ttotal: 21m 31s\tremaining: 14m 19s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6871541434\n",
      "bestIteration = 1704\n",
      "\n",
      "Shrink model to first 1705 iterations.\n",
      "err:  0.6871541434475893\n",
      "0:\ttest: 0.6247567\ttest1: 0.6360112\tbest: 0.6360112 (0)\ttotal: 630ms\tremaining: 31m 30s\n",
      "100:\ttest: 0.6900712\ttest1: 0.6854201\tbest: 0.6854201 (100)\ttotal: 1m 15s\tremaining: 36m 18s\n",
      "200:\ttest: 0.6997716\ttest1: 0.6902737\tbest: 0.6902737 (200)\ttotal: 2m 27s\tremaining: 34m 15s\n",
      "300:\ttest: 0.7041861\ttest1: 0.6919992\tbest: 0.6919992 (300)\ttotal: 3m 37s\tremaining: 32m 27s\n",
      "400:\ttest: 0.7078358\ttest1: 0.6931758\tbest: 0.6931758 (400)\ttotal: 4m 46s\tremaining: 30m 55s\n",
      "500:\ttest: 0.7104795\ttest1: 0.6938607\tbest: 0.6938607 (500)\ttotal: 5m 56s\tremaining: 29m 38s\n",
      "600:\ttest: 0.7129303\ttest1: 0.6942475\tbest: 0.6942475 (600)\ttotal: 7m 7s\tremaining: 28m 24s\n",
      "700:\ttest: 0.7146693\ttest1: 0.6946318\tbest: 0.6946340 (696)\ttotal: 8m 18s\tremaining: 27m 16s\n",
      "800:\ttest: 0.7164550\ttest1: 0.6952559\tbest: 0.6952642 (799)\ttotal: 9m 30s\tremaining: 26m 5s\n",
      "900:\ttest: 0.7177533\ttest1: 0.6955983\tbest: 0.6956110 (897)\ttotal: 10m 41s\tremaining: 24m 55s\n",
      "1000:\ttest: 0.7189805\ttest1: 0.6958890\tbest: 0.6959037 (997)\ttotal: 11m 53s\tremaining: 23m 44s\n",
      "1100:\ttest: 0.7203090\ttest1: 0.6960390\tbest: 0.6960646 (1096)\ttotal: 13m 5s\tremaining: 22m 35s\n",
      "1200:\ttest: 0.7213779\ttest1: 0.6963125\tbest: 0.6963125 (1200)\ttotal: 14m 18s\tremaining: 21m 26s\n",
      "1300:\ttest: 0.7224211\ttest1: 0.6964497\tbest: 0.6964518 (1298)\ttotal: 15m 32s\tremaining: 20m 17s\n",
      "1400:\ttest: 0.7235405\ttest1: 0.6966815\tbest: 0.6967031 (1398)\ttotal: 16m 44s\tremaining: 19m 6s\n",
      "1500:\ttest: 0.7244726\ttest1: 0.6968692\tbest: 0.6968692 (1500)\ttotal: 17m 58s\tremaining: 17m 57s\n",
      "1600:\ttest: 0.7254062\ttest1: 0.6970160\tbest: 0.6970277 (1590)\ttotal: 19m 12s\tremaining: 16m 46s\n",
      "1700:\ttest: 0.7263777\ttest1: 0.6970907\tbest: 0.6971084 (1688)\ttotal: 20m 25s\tremaining: 15m 36s\n",
      "1800:\ttest: 0.7272793\ttest1: 0.6970901\tbest: 0.6971278 (1777)\ttotal: 21m 40s\tremaining: 14m 25s\n",
      "1900:\ttest: 0.7281827\ttest1: 0.6971599\tbest: 0.6972143 (1877)\ttotal: 22m 54s\tremaining: 13m 14s\n",
      "2000:\ttest: 0.7290246\ttest1: 0.6973345\tbest: 0.6973733 (1997)\ttotal: 24m 9s\tremaining: 12m 3s\n",
      "2100:\ttest: 0.7301179\ttest1: 0.6972631\tbest: 0.6973760 (2044)\ttotal: 25m 25s\tremaining: 10m 52s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6973759847\n",
      "bestIteration = 2044\n",
      "\n",
      "Shrink model to first 2045 iterations.\n",
      "err:  0.6973759847053976\n",
      "0:\ttest: 0.6230540\ttest1: 0.6188687\tbest: 0.6188687 (0)\ttotal: 614ms\tremaining: 30m 42s\n",
      "100:\ttest: 0.6895544\ttest1: 0.6764603\tbest: 0.6764603 (100)\ttotal: 1m 6s\tremaining: 31m 35s\n",
      "200:\ttest: 0.7006606\ttest1: 0.6827842\tbest: 0.6827842 (200)\ttotal: 2m 14s\tremaining: 31m 19s\n",
      "300:\ttest: 0.7055054\ttest1: 0.6852040\tbest: 0.6852040 (300)\ttotal: 3m 23s\tremaining: 30m 25s\n",
      "400:\ttest: 0.7089273\ttest1: 0.6866990\tbest: 0.6867130 (396)\ttotal: 4m 32s\tremaining: 29m 28s\n",
      "500:\ttest: 0.7115005\ttest1: 0.6875124\tbest: 0.6875235 (497)\ttotal: 5m 42s\tremaining: 28m 27s\n",
      "600:\ttest: 0.7136035\ttest1: 0.6883870\tbest: 0.6883870 (600)\ttotal: 6m 52s\tremaining: 27m 27s\n",
      "700:\ttest: 0.7155532\ttest1: 0.6889358\tbest: 0.6889358 (700)\ttotal: 8m 4s\tremaining: 26m 29s\n",
      "800:\ttest: 0.7170401\ttest1: 0.6893323\tbest: 0.6893589 (788)\ttotal: 9m 16s\tremaining: 25m 28s\n",
      "900:\ttest: 0.7184136\ttest1: 0.6897234\tbest: 0.6897234 (900)\ttotal: 10m 28s\tremaining: 24m 23s\n",
      "1000:\ttest: 0.7196234\ttest1: 0.6899886\tbest: 0.6900231 (993)\ttotal: 11m 40s\tremaining: 23m 19s\n",
      "1100:\ttest: 0.7208660\ttest1: 0.6902686\tbest: 0.6902825 (1071)\ttotal: 12m 53s\tremaining: 22m 13s\n",
      "1200:\ttest: 0.7220628\ttest1: 0.6905165\tbest: 0.6905300 (1196)\ttotal: 14m 5s\tremaining: 21m 6s\n",
      "1300:\ttest: 0.7230854\ttest1: 0.6906976\tbest: 0.6906976 (1300)\ttotal: 15m 18s\tremaining: 19m 58s\n",
      "1400:\ttest: 0.7240963\ttest1: 0.6908346\tbest: 0.6908346 (1400)\ttotal: 16m 30s\tremaining: 18m 50s\n",
      "1500:\ttest: 0.7251745\ttest1: 0.6909207\tbest: 0.6909388 (1499)\ttotal: 17m 43s\tremaining: 17m 42s\n",
      "1600:\ttest: 0.7261561\ttest1: 0.6912180\tbest: 0.6912301 (1599)\ttotal: 18m 56s\tremaining: 16m 33s\n",
      "1700:\ttest: 0.7273048\ttest1: 0.6912569\tbest: 0.6912615 (1698)\ttotal: 20m 10s\tremaining: 15m 24s\n",
      "1800:\ttest: 0.7284899\ttest1: 0.6914740\tbest: 0.6915052 (1791)\ttotal: 21m 23s\tremaining: 14m 14s\n",
      "1900:\ttest: 0.7293100\ttest1: 0.6915921\tbest: 0.6916040 (1897)\ttotal: 22m 37s\tremaining: 13m 5s\n",
      "2000:\ttest: 0.7304487\ttest1: 0.6916008\tbest: 0.6916492 (1955)\ttotal: 23m 51s\tremaining: 11m 54s\n",
      "2100:\ttest: 0.7314175\ttest1: 0.6918492\tbest: 0.6918492 (2100)\ttotal: 25m 5s\tremaining: 10m 44s\n",
      "2200:\ttest: 0.7323797\ttest1: 0.6920217\tbest: 0.6920492 (2196)\ttotal: 26m 18s\tremaining: 9m 33s\n",
      "2300:\ttest: 0.7331981\ttest1: 0.6920563\tbest: 0.6920595 (2281)\ttotal: 27m 33s\tremaining: 8m 22s\n",
      "2400:\ttest: 0.7342739\ttest1: 0.6919467\tbest: 0.6921836 (2357)\ttotal: 28m 48s\tremaining: 7m 11s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6921835721\n",
      "bestIteration = 2357\n",
      "\n",
      "Shrink model to first 2358 iterations.\n",
      "err:  0.6921835721116445\n",
      "0:\ttest: 0.6243128\ttest1: 0.6094128\tbest: 0.6094128 (0)\ttotal: 622ms\tremaining: 31m 5s\n",
      "100:\ttest: 0.6904969\ttest1: 0.6644498\tbest: 0.6644498 (100)\ttotal: 1m 7s\tremaining: 32m 15s\n",
      "200:\ttest: 0.7003803\ttest1: 0.6702988\tbest: 0.6702988 (200)\ttotal: 2m 16s\tremaining: 31m 43s\n",
      "300:\ttest: 0.7046420\ttest1: 0.6722936\tbest: 0.6722936 (300)\ttotal: 3m 34s\tremaining: 32m 1s\n",
      "400:\ttest: 0.7076852\ttest1: 0.6733395\tbest: 0.6733547 (399)\ttotal: 4m 44s\tremaining: 30m 45s\n",
      "500:\ttest: 0.7101706\ttest1: 0.6742147\tbest: 0.6742147 (500)\ttotal: 5m 54s\tremaining: 29m 26s\n",
      "600:\ttest: 0.7122093\ttest1: 0.6748339\tbest: 0.6748339 (600)\ttotal: 7m 4s\tremaining: 28m 14s\n",
      "700:\ttest: 0.7140427\ttest1: 0.6752180\tbest: 0.6752404 (681)\ttotal: 8m 15s\tremaining: 27m 5s\n",
      "800:\ttest: 0.7155666\ttest1: 0.6753740\tbest: 0.6753929 (796)\ttotal: 9m 28s\tremaining: 26m\n",
      "900:\ttest: 0.7171826\ttest1: 0.6758111\tbest: 0.6758111 (900)\ttotal: 10m 40s\tremaining: 24m 51s\n",
      "1000:\ttest: 0.7185823\ttest1: 0.6761017\tbest: 0.6761017 (1000)\ttotal: 11m 53s\tremaining: 23m 44s\n",
      "1100:\ttest: 0.7196504\ttest1: 0.6762338\tbest: 0.6762440 (1093)\ttotal: 13m 7s\tremaining: 22m 38s\n",
      "1200:\ttest: 0.7207646\ttest1: 0.6762598\tbest: 0.6763229 (1181)\ttotal: 14m 20s\tremaining: 21m 28s\n",
      "1300:\ttest: 0.7218248\ttest1: 0.6764919\tbest: 0.6765378 (1270)\ttotal: 15m 33s\tremaining: 20m 19s\n",
      "1400:\ttest: 0.7231523\ttest1: 0.6769343\tbest: 0.6769343 (1400)\ttotal: 16m 47s\tremaining: 19m 9s\n",
      "1500:\ttest: 0.7241847\ttest1: 0.6770613\tbest: 0.6770884 (1494)\ttotal: 18m 1s\tremaining: 18m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1600:\ttest: 0.7250679\ttest1: 0.6771310\tbest: 0.6771372 (1598)\ttotal: 19m 15s\tremaining: 16m 49s\n",
      "1700:\ttest: 0.7260419\ttest1: 0.6773691\tbest: 0.6774362 (1689)\ttotal: 20m 30s\tremaining: 15m 39s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6774362321\n",
      "bestIteration = 1689\n",
      "\n",
      "Shrink model to first 1690 iterations.\n",
      "err:  0.677436232050502\n",
      "0:\ttest: 0.6216150\ttest1: 0.6103711\tbest: 0.6103711 (0)\ttotal: 1.01s\tremaining: 50m 21s\n",
      "100:\ttest: 0.6919819\ttest1: 0.6546138\tbest: 0.6546138 (100)\ttotal: 1m 16s\tremaining: 36m 25s\n",
      "200:\ttest: 0.7023167\ttest1: 0.6595422\tbest: 0.6595422 (200)\ttotal: 2m 24s\tremaining: 33m 36s\n",
      "300:\ttest: 0.7067608\ttest1: 0.6617835\tbest: 0.6618193 (298)\ttotal: 3m 33s\tremaining: 31m 51s\n",
      "400:\ttest: 0.7100749\ttest1: 0.6630082\tbest: 0.6630311 (399)\ttotal: 4m 41s\tremaining: 30m 27s\n",
      "500:\ttest: 0.7128757\ttest1: 0.6640254\tbest: 0.6640254 (500)\ttotal: 5m 51s\tremaining: 29m 13s\n",
      "600:\ttest: 0.7149214\ttest1: 0.6646851\tbest: 0.6646967 (595)\ttotal: 7m 2s\tremaining: 28m 4s\n",
      "700:\ttest: 0.7165412\ttest1: 0.6649999\tbest: 0.6649999 (700)\ttotal: 8m 13s\tremaining: 26m 58s\n",
      "800:\ttest: 0.7179761\ttest1: 0.6654177\tbest: 0.6654188 (799)\ttotal: 9m 26s\tremaining: 25m 54s\n",
      "900:\ttest: 0.7193205\ttest1: 0.6657364\tbest: 0.6657594 (889)\ttotal: 10m 39s\tremaining: 24m 48s\n",
      "1000:\ttest: 0.7207173\ttest1: 0.6660712\tbest: 0.6660885 (992)\ttotal: 11m 52s\tremaining: 23m 42s\n",
      "1100:\ttest: 0.7221167\ttest1: 0.6663235\tbest: 0.6663393 (1096)\ttotal: 13m 4s\tremaining: 22m 33s\n",
      "1200:\ttest: 0.7233947\ttest1: 0.6664088\tbest: 0.6664267 (1184)\ttotal: 14m 17s\tremaining: 21m 24s\n",
      "1300:\ttest: 0.7243672\ttest1: 0.6666021\tbest: 0.6666039 (1292)\ttotal: 15m 30s\tremaining: 20m 14s\n",
      "1400:\ttest: 0.7253311\ttest1: 0.6668970\tbest: 0.6668970 (1400)\ttotal: 16m 44s\tremaining: 19m 5s\n",
      "1500:\ttest: 0.7263014\ttest1: 0.6671617\tbest: 0.6671701 (1497)\ttotal: 17m 57s\tremaining: 17m 56s\n",
      "1600:\ttest: 0.7272627\ttest1: 0.6672857\tbest: 0.6673055 (1596)\ttotal: 19m 11s\tremaining: 16m 46s\n",
      "1700:\ttest: 0.7282932\ttest1: 0.6674717\tbest: 0.6674747 (1699)\ttotal: 20m 25s\tremaining: 15m 36s\n",
      "1800:\ttest: 0.7292289\ttest1: 0.6677319\tbest: 0.6677320 (1799)\ttotal: 21m 40s\tremaining: 14m 25s\n",
      "1900:\ttest: 0.7300760\ttest1: 0.6678204\tbest: 0.6678493 (1887)\ttotal: 22m 54s\tremaining: 13m 14s\n",
      "2000:\ttest: 0.7308875\ttest1: 0.6678665\tbest: 0.6678917 (1988)\ttotal: 24m 9s\tremaining: 12m 3s\n",
      "2100:\ttest: 0.7316671\ttest1: 0.6680911\tbest: 0.6680911 (2100)\ttotal: 25m 24s\tremaining: 10m 52s\n",
      "2200:\ttest: 0.7325390\ttest1: 0.6684326\tbest: 0.6684327 (2196)\ttotal: 26m 39s\tremaining: 9m 40s\n",
      "2300:\ttest: 0.7334061\ttest1: 0.6686653\tbest: 0.6686653 (2300)\ttotal: 27m 56s\tremaining: 8m 29s\n",
      "2400:\ttest: 0.7342472\ttest1: 0.6687681\tbest: 0.6687845 (2363)\ttotal: 29m 11s\tremaining: 7m 17s\n",
      "2500:\ttest: 0.7350159\ttest1: 0.6690366\tbest: 0.6690644 (2489)\ttotal: 30m 26s\tremaining: 6m 4s\n",
      "2600:\ttest: 0.7358398\ttest1: 0.6691637\tbest: 0.6691986 (2593)\ttotal: 31m 42s\tremaining: 4m 51s\n",
      "2700:\ttest: 0.7366786\ttest1: 0.6692350\tbest: 0.6692405 (2680)\ttotal: 32m 58s\tremaining: 3m 39s\n",
      "2800:\ttest: 0.7373244\ttest1: 0.6692062\tbest: 0.6692856 (2713)\ttotal: 34m 15s\tremaining: 2m 26s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.669285607\n",
      "bestIteration = 2713\n",
      "\n",
      "Shrink model to first 2714 iterations.\n",
      "err:  0.6692856070212005\n",
      "0:\ttest: 0.6235837\ttest1: 0.6147848\tbest: 0.6147848 (0)\ttotal: 607ms\tremaining: 30m 19s\n",
      "100:\ttest: 0.6897658\ttest1: 0.6661599\tbest: 0.6661599 (100)\ttotal: 1m 12s\tremaining: 34m 52s\n",
      "200:\ttest: 0.7007338\ttest1: 0.6725971\tbest: 0.6725971 (200)\ttotal: 2m 23s\tremaining: 33m 23s\n",
      "300:\ttest: 0.7055659\ttest1: 0.6753634\tbest: 0.6753634 (300)\ttotal: 3m 32s\tremaining: 31m 43s\n",
      "400:\ttest: 0.7094383\ttest1: 0.6770263\tbest: 0.6770263 (400)\ttotal: 4m 42s\tremaining: 30m 30s\n",
      "500:\ttest: 0.7120554\ttest1: 0.6781470\tbest: 0.6781470 (500)\ttotal: 5m 53s\tremaining: 29m 24s\n",
      "600:\ttest: 0.7142240\ttest1: 0.6788857\tbest: 0.6788869 (598)\ttotal: 7m 5s\tremaining: 28m 17s\n",
      "700:\ttest: 0.7162538\ttest1: 0.6794349\tbest: 0.6794455 (699)\ttotal: 8m 17s\tremaining: 27m 12s\n",
      "800:\ttest: 0.7178106\ttest1: 0.6797740\tbest: 0.6798009 (790)\ttotal: 9m 29s\tremaining: 26m 3s\n",
      "900:\ttest: 0.7192330\ttest1: 0.6802339\tbest: 0.6802421 (897)\ttotal: 10m 45s\tremaining: 25m 4s\n",
      "1000:\ttest: 0.7206600\ttest1: 0.6804793\tbest: 0.6804863 (999)\ttotal: 11m 58s\tremaining: 23m 54s\n",
      "1100:\ttest: 0.7220528\ttest1: 0.6807566\tbest: 0.6807566 (1100)\ttotal: 13m 11s\tremaining: 22m 44s\n",
      "1200:\ttest: 0.7233309\ttest1: 0.6809917\tbest: 0.6810262 (1196)\ttotal: 14m 24s\tremaining: 21m 34s\n",
      "1300:\ttest: 0.7245792\ttest1: 0.6811454\tbest: 0.6811454 (1300)\ttotal: 15m 37s\tremaining: 20m 24s\n",
      "1400:\ttest: 0.7258042\ttest1: 0.6814092\tbest: 0.6814601 (1391)\ttotal: 16m 51s\tremaining: 19m 13s\n",
      "1500:\ttest: 0.7271638\ttest1: 0.6816443\tbest: 0.6816443 (1500)\ttotal: 18m 4s\tremaining: 18m 3s\n",
      "1600:\ttest: 0.7280478\ttest1: 0.6818113\tbest: 0.6818145 (1598)\ttotal: 19m 17s\tremaining: 16m 51s\n",
      "1700:\ttest: 0.7291547\ttest1: 0.6818826\tbest: 0.6819274 (1695)\ttotal: 20m 31s\tremaining: 15m 40s\n",
      "1800:\ttest: 0.7302508\ttest1: 0.6820355\tbest: 0.6820355 (1800)\ttotal: 21m 45s\tremaining: 14m 29s\n",
      "1900:\ttest: 0.7310714\ttest1: 0.6821980\tbest: 0.6821982 (1893)\ttotal: 22m 58s\tremaining: 13m 16s\n",
      "2000:\ttest: 0.7320807\ttest1: 0.6823161\tbest: 0.6823489 (1982)\ttotal: 24m 12s\tremaining: 12m 5s\n",
      "2100:\ttest: 0.7329850\ttest1: 0.6825174\tbest: 0.6825174 (2100)\ttotal: 25m 27s\tremaining: 10m 53s\n",
      "2200:\ttest: 0.7340289\ttest1: 0.6827129\tbest: 0.6827325 (2193)\ttotal: 26m 51s\tremaining: 9m 45s\n",
      "2300:\ttest: 0.7349613\ttest1: 0.6828207\tbest: 0.6828394 (2294)\ttotal: 28m 9s\tremaining: 8m 33s\n",
      "2400:\ttest: 0.7359474\ttest1: 0.6828810\tbest: 0.6829283 (2379)\ttotal: 29m 24s\tremaining: 7m 20s\n",
      "2500:\ttest: 0.7367339\ttest1: 0.6830167\tbest: 0.6830167 (2500)\ttotal: 30m 39s\tremaining: 6m 7s\n",
      "2600:\ttest: 0.7376261\ttest1: 0.6829939\tbest: 0.6830500 (2580)\ttotal: 31m 56s\tremaining: 4m 54s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.683050022\n",
      "bestIteration = 2580\n",
      "\n",
      "Shrink model to first 2581 iterations.\n",
      "err:  0.68305002195413\n",
      "0:\ttest: 0.6233738\ttest1: 0.6299678\tbest: 0.6299678 (0)\ttotal: 630ms\tremaining: 31m 27s\n",
      "100:\ttest: 0.6903185\ttest1: 0.6765974\tbest: 0.6765974 (100)\ttotal: 1m 6s\tremaining: 31m 38s\n",
      "200:\ttest: 0.7008313\ttest1: 0.6819870\tbest: 0.6819870 (200)\ttotal: 2m 15s\tremaining: 31m 22s\n",
      "300:\ttest: 0.7053277\ttest1: 0.6838793\tbest: 0.6838793 (300)\ttotal: 3m 24s\tremaining: 30m 29s\n",
      "400:\ttest: 0.7087473\ttest1: 0.6853756\tbest: 0.6854063 (397)\ttotal: 4m 33s\tremaining: 29m 29s\n",
      "500:\ttest: 0.7114961\ttest1: 0.6861461\tbest: 0.6861461 (500)\ttotal: 5m 43s\tremaining: 28m 31s\n",
      "600:\ttest: 0.7136005\ttest1: 0.6865466\tbest: 0.6865466 (600)\ttotal: 6m 53s\tremaining: 27m 31s\n",
      "700:\ttest: 0.7153875\ttest1: 0.6869022\tbest: 0.6869041 (697)\ttotal: 8m 5s\tremaining: 26m 32s\n",
      "800:\ttest: 0.7169123\ttest1: 0.6871011\tbest: 0.6871011 (800)\ttotal: 9m 16s\tremaining: 25m 29s\n",
      "900:\ttest: 0.7182269\ttest1: 0.6873019\tbest: 0.6873067 (898)\ttotal: 10m 29s\tremaining: 24m 26s\n",
      "1000:\ttest: 0.7196531\ttest1: 0.6875162\tbest: 0.6875201 (990)\ttotal: 11m 41s\tremaining: 23m 20s\n",
      "1100:\ttest: 0.7209455\ttest1: 0.6878071\tbest: 0.6878622 (1091)\ttotal: 12m 55s\tremaining: 22m 18s\n",
      "1200:\ttest: 0.7219567\ttest1: 0.6879277\tbest: 0.6879406 (1196)\ttotal: 14m 9s\tremaining: 21m 11s\n",
      "1300:\ttest: 0.7230302\ttest1: 0.6882080\tbest: 0.6882080 (1300)\ttotal: 15m 22s\tremaining: 20m 4s\n",
      "1400:\ttest: 0.7241702\ttest1: 0.6883368\tbest: 0.6883549 (1397)\ttotal: 16m 36s\tremaining: 18m 57s\n",
      "1500:\ttest: 0.7251819\ttest1: 0.6885198\tbest: 0.6885798 (1473)\ttotal: 17m 54s\tremaining: 17m 52s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6885797875\n",
      "bestIteration = 1473\n",
      "\n",
      "Shrink model to first 1474 iterations.\n",
      "err:  0.6885797874732846\n",
      "0:\ttest: 0.6251041\ttest1: 0.6277555\tbest: 0.6277555 (0)\ttotal: 613ms\tremaining: 30m 37s\n",
      "100:\ttest: 0.6893551\ttest1: 0.6748233\tbest: 0.6748233 (100)\ttotal: 1m 5s\tremaining: 31m 24s\n",
      "200:\ttest: 0.6988401\ttest1: 0.6796480\tbest: 0.6796480 (200)\ttotal: 2m 14s\tremaining: 31m 17s\n",
      "300:\ttest: 0.7039333\ttest1: 0.6817267\tbest: 0.6817267 (300)\ttotal: 3m 23s\tremaining: 30m 28s\n",
      "400:\ttest: 0.7077832\ttest1: 0.6830035\tbest: 0.6830035 (400)\ttotal: 4m 33s\tremaining: 29m 33s\n",
      "500:\ttest: 0.7104540\ttest1: 0.6837084\tbest: 0.6837084 (500)\ttotal: 5m 44s\tremaining: 28m 40s\n",
      "600:\ttest: 0.7125534\ttest1: 0.6842535\tbest: 0.6842634 (591)\ttotal: 6m 54s\tremaining: 27m 36s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "700:\ttest: 0.7148588\ttest1: 0.6848948\tbest: 0.6848948 (700)\ttotal: 8m 5s\tremaining: 26m 31s\n",
      "800:\ttest: 0.7163820\ttest1: 0.6853115\tbest: 0.6853352 (796)\ttotal: 9m 16s\tremaining: 25m 28s\n",
      "900:\ttest: 0.7178133\ttest1: 0.6856260\tbest: 0.6856352 (898)\ttotal: 10m 31s\tremaining: 24m 30s\n",
      "1000:\ttest: 0.7192064\ttest1: 0.6857925\tbest: 0.6858110 (995)\ttotal: 11m 45s\tremaining: 23m 28s\n",
      "1100:\ttest: 0.7206271\ttest1: 0.6860887\tbest: 0.6861007 (1091)\ttotal: 12m 59s\tremaining: 22m 23s\n",
      "1200:\ttest: 0.7220098\ttest1: 0.6862627\tbest: 0.6863069 (1149)\ttotal: 14m 12s\tremaining: 21m 17s\n",
      "1300:\ttest: 0.7232216\ttest1: 0.6864481\tbest: 0.6864481 (1300)\ttotal: 15m 27s\tremaining: 20m 10s\n",
      "1400:\ttest: 0.7243796\ttest1: 0.6865786\tbest: 0.6865828 (1399)\ttotal: 16m 42s\tremaining: 19m 3s\n",
      "1500:\ttest: 0.7255056\ttest1: 0.6866996\tbest: 0.6867163 (1494)\ttotal: 17m 56s\tremaining: 17m 55s\n",
      "1600:\ttest: 0.7264869\ttest1: 0.6867906\tbest: 0.6867971 (1568)\ttotal: 19m 10s\tremaining: 16m 45s\n",
      "1700:\ttest: 0.7276142\ttest1: 0.6868797\tbest: 0.6869295 (1688)\ttotal: 20m 25s\tremaining: 15m 35s\n",
      "1800:\ttest: 0.7287893\ttest1: 0.6869955\tbest: 0.6870201 (1793)\ttotal: 21m 40s\tremaining: 14m 25s\n",
      "1900:\ttest: 0.7296126\ttest1: 0.6870706\tbest: 0.6870706 (1900)\ttotal: 22m 54s\tremaining: 13m 14s\n",
      "2000:\ttest: 0.7306576\ttest1: 0.6872884\tbest: 0.6873086 (1994)\ttotal: 24m 10s\tremaining: 12m 3s\n",
      "2100:\ttest: 0.7316111\ttest1: 0.6873561\tbest: 0.6873590 (2097)\ttotal: 25m 25s\tremaining: 10m 52s\n",
      "2200:\ttest: 0.7325102\ttest1: 0.6874556\tbest: 0.6874604 (2187)\ttotal: 26m 40s\tremaining: 9m 40s\n",
      "2300:\ttest: 0.7335041\ttest1: 0.6875825\tbest: 0.6876313 (2259)\ttotal: 27m 57s\tremaining: 8m 29s\n",
      "2400:\ttest: 0.7345123\ttest1: 0.6877901\tbest: 0.6878108 (2395)\ttotal: 29m 12s\tremaining: 7m 17s\n",
      "2500:\ttest: 0.7354495\ttest1: 0.6878270\tbest: 0.6879538 (2448)\ttotal: 30m 28s\tremaining: 6m 4s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6879538233\n",
      "bestIteration = 2448\n",
      "\n",
      "Shrink model to first 2449 iterations.\n",
      "err:  0.687953823321414\n",
      "0:\ttest: 0.6246365\ttest1: 0.6072429\tbest: 0.6072429 (0)\ttotal: 950ms\tremaining: 47m 29s\n",
      "100:\ttest: 0.6930981\ttest1: 0.6570899\tbest: 0.6570899 (100)\ttotal: 1m 12s\tremaining: 34m 39s\n",
      "200:\ttest: 0.7010468\ttest1: 0.6629737\tbest: 0.6629737 (200)\ttotal: 2m 21s\tremaining: 32m 52s\n",
      "300:\ttest: 0.7051729\ttest1: 0.6654919\tbest: 0.6654919 (300)\ttotal: 3m 29s\tremaining: 31m 22s\n",
      "400:\ttest: 0.7086159\ttest1: 0.6671334\tbest: 0.6671334 (400)\ttotal: 4m 39s\tremaining: 30m 14s\n",
      "500:\ttest: 0.7112534\ttest1: 0.6682194\tbest: 0.6682346 (498)\ttotal: 5m 49s\tremaining: 29m 2s\n",
      "600:\ttest: 0.7130563\ttest1: 0.6688789\tbest: 0.6688789 (600)\ttotal: 6m 59s\tremaining: 27m 53s\n",
      "700:\ttest: 0.7147493\ttest1: 0.6694631\tbest: 0.6694631 (700)\ttotal: 8m 13s\tremaining: 26m 57s\n",
      "800:\ttest: 0.7161574\ttest1: 0.6698387\tbest: 0.6698387 (800)\ttotal: 9m 24s\tremaining: 25m 48s\n",
      "900:\ttest: 0.7174115\ttest1: 0.6702330\tbest: 0.6702370 (899)\ttotal: 10m 35s\tremaining: 24m 41s\n",
      "1000:\ttest: 0.7187272\ttest1: 0.6705124\tbest: 0.6705247 (995)\ttotal: 11m 48s\tremaining: 23m 34s\n",
      "1100:\ttest: 0.7198719\ttest1: 0.6707488\tbest: 0.6707773 (1084)\ttotal: 13m\tremaining: 22m 26s\n",
      "1200:\ttest: 0.7209810\ttest1: 0.6710750\tbest: 0.6710852 (1197)\ttotal: 14m 14s\tremaining: 21m 20s\n",
      "1300:\ttest: 0.7221005\ttest1: 0.6713376\tbest: 0.6713376 (1300)\ttotal: 15m 28s\tremaining: 20m 12s\n",
      "1400:\ttest: 0.7232625\ttest1: 0.6714004\tbest: 0.6714265 (1398)\ttotal: 16m 42s\tremaining: 19m 3s\n",
      "1500:\ttest: 0.7246499\ttest1: 0.6717327\tbest: 0.6717347 (1497)\ttotal: 17m 55s\tremaining: 17m 54s\n",
      "1600:\ttest: 0.7258695\ttest1: 0.6718728\tbest: 0.6719076 (1556)\ttotal: 19m 10s\tremaining: 16m 45s\n",
      "1700:\ttest: 0.7268676\ttest1: 0.6720756\tbest: 0.6720924 (1681)\ttotal: 20m 25s\tremaining: 15m 35s\n",
      "1800:\ttest: 0.7278113\ttest1: 0.6721768\tbest: 0.6721768 (1800)\ttotal: 21m 39s\tremaining: 14m 24s\n",
      "1900:\ttest: 0.7287281\ttest1: 0.6722991\tbest: 0.6723202 (1896)\ttotal: 22m 54s\tremaining: 13m 14s\n",
      "2000:\ttest: 0.7296024\ttest1: 0.6724563\tbest: 0.6724766 (1995)\ttotal: 24m 9s\tremaining: 12m 3s\n",
      "2100:\ttest: 0.7304800\ttest1: 0.6727454\tbest: 0.6727568 (2098)\ttotal: 25m 25s\tremaining: 10m 52s\n",
      "2200:\ttest: 0.7313154\ttest1: 0.6729058\tbest: 0.6729058 (2200)\ttotal: 26m 40s\tremaining: 9m 41s\n",
      "2300:\ttest: 0.7322147\ttest1: 0.6730577\tbest: 0.6730598 (2299)\ttotal: 27m 56s\tremaining: 8m 29s\n",
      "2400:\ttest: 0.7330413\ttest1: 0.6731058\tbest: 0.6731163 (2398)\ttotal: 29m 13s\tremaining: 7m 17s\n",
      "2500:\ttest: 0.7337915\ttest1: 0.6731901\tbest: 0.6732494 (2465)\ttotal: 30m 29s\tremaining: 6m 5s\n",
      "2600:\ttest: 0.7346456\ttest1: 0.6733066\tbest: 0.6733148 (2588)\ttotal: 31m 45s\tremaining: 4m 52s\n",
      "2700:\ttest: 0.7354323\ttest1: 0.6733623\tbest: 0.6733953 (2662)\ttotal: 33m 2s\tremaining: 3m 39s\n",
      "2800:\ttest: 0.7364415\ttest1: 0.6734658\tbest: 0.6734732 (2799)\ttotal: 34m 17s\tremaining: 2m 26s\n",
      "2900:\ttest: 0.7373953\ttest1: 0.6736085\tbest: 0.6736192 (2890)\ttotal: 35m 33s\tremaining: 1m 12s\n",
      "2999:\ttest: 0.7383856\ttest1: 0.6735414\tbest: 0.6736278 (2930)\ttotal: 36m 50s\tremaining: 0us\n",
      "\n",
      "bestTest = 0.673627828\n",
      "bestIteration = 2930\n",
      "\n",
      "Shrink model to first 2931 iterations.\n",
      "err:  0.6736278279577902\n",
      "0:\ttest: 0.6222534\ttest1: 0.6192051\tbest: 0.6192051 (0)\ttotal: 609ms\tremaining: 30m 27s\n",
      "100:\ttest: 0.6899515\ttest1: 0.6679447\tbest: 0.6679447 (100)\ttotal: 1m 12s\tremaining: 34m 28s\n",
      "200:\ttest: 0.7004432\ttest1: 0.6745070\tbest: 0.6745070 (200)\ttotal: 2m 23s\tremaining: 33m 14s\n",
      "300:\ttest: 0.7053112\ttest1: 0.6771332\tbest: 0.6771332 (300)\ttotal: 3m 32s\tremaining: 31m 40s\n",
      "400:\ttest: 0.7086791\ttest1: 0.6790192\tbest: 0.6790192 (400)\ttotal: 4m 41s\tremaining: 30m 21s\n",
      "500:\ttest: 0.7109890\ttest1: 0.6801934\tbest: 0.6801934 (500)\ttotal: 5m 51s\tremaining: 29m 13s\n",
      "600:\ttest: 0.7134927\ttest1: 0.6811825\tbest: 0.6811825 (600)\ttotal: 7m 2s\tremaining: 28m 6s\n",
      "700:\ttest: 0.7152040\ttest1: 0.6817779\tbest: 0.6817779 (700)\ttotal: 8m 14s\tremaining: 27m 3s\n",
      "800:\ttest: 0.7165963\ttest1: 0.6824729\tbest: 0.6824729 (800)\ttotal: 9m 27s\tremaining: 25m 57s\n",
      "900:\ttest: 0.7182006\ttest1: 0.6829995\tbest: 0.6830046 (897)\ttotal: 10m 40s\tremaining: 24m 51s\n",
      "1000:\ttest: 0.7195018\ttest1: 0.6836139\tbest: 0.6836225 (995)\ttotal: 11m 53s\tremaining: 23m 44s\n",
      "1100:\ttest: 0.7207985\ttest1: 0.6840378\tbest: 0.6840424 (1097)\ttotal: 13m 6s\tremaining: 22m 36s\n",
      "1200:\ttest: 0.7221482\ttest1: 0.6844836\tbest: 0.6844836 (1200)\ttotal: 14m 19s\tremaining: 21m 28s\n",
      "1300:\ttest: 0.7231970\ttest1: 0.6846816\tbest: 0.6847192 (1280)\ttotal: 15m 33s\tremaining: 20m 19s\n",
      "1400:\ttest: 0.7243952\ttest1: 0.6849692\tbest: 0.6849692 (1400)\ttotal: 16m 47s\tremaining: 19m 9s\n",
      "1500:\ttest: 0.7255362\ttest1: 0.6852895\tbest: 0.6853250 (1495)\ttotal: 18m 1s\tremaining: 18m\n",
      "1600:\ttest: 0.7264979\ttest1: 0.6856090\tbest: 0.6856090 (1600)\ttotal: 19m 15s\tremaining: 16m 49s\n",
      "1700:\ttest: 0.7274319\ttest1: 0.6856539\tbest: 0.6856622 (1697)\ttotal: 20m 27s\tremaining: 15m 37s\n",
      "1800:\ttest: 0.7283649\ttest1: 0.6858916\tbest: 0.6859056 (1791)\ttotal: 21m 41s\tremaining: 14m 26s\n",
      "1900:\ttest: 0.7292224\ttest1: 0.6860411\tbest: 0.6860700 (1888)\ttotal: 22m 55s\tremaining: 13m 15s\n",
      "2000:\ttest: 0.7302691\ttest1: 0.6862749\tbest: 0.6862749 (2000)\ttotal: 24m 9s\tremaining: 12m 3s\n",
      "2100:\ttest: 0.7312891\ttest1: 0.6863208\tbest: 0.6863473 (2094)\ttotal: 25m 25s\tremaining: 10m 52s\n",
      "2200:\ttest: 0.7321577\ttest1: 0.6864863\tbest: 0.6865186 (2195)\ttotal: 26m 40s\tremaining: 9m 41s\n",
      "2300:\ttest: 0.7331956\ttest1: 0.6865717\tbest: 0.6865864 (2259)\ttotal: 27m 55s\tremaining: 8m 28s\n",
      "2400:\ttest: 0.7341844\ttest1: 0.6867096\tbest: 0.6867096 (2400)\ttotal: 29m 9s\tremaining: 7m 16s\n",
      "2500:\ttest: 0.7351412\ttest1: 0.6866519\tbest: 0.6867539 (2407)\ttotal: 30m 24s\tremaining: 6m 3s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6867539347\n",
      "bestIteration = 2407\n",
      "\n",
      "Shrink model to first 2408 iterations.\n",
      "err:  0.6867539347053069\n",
      "0:\ttest: 0.6252571\ttest1: 0.6217601\tbest: 0.6217601 (0)\ttotal: 613ms\tremaining: 30m 37s\n",
      "100:\ttest: 0.6909092\ttest1: 0.6657042\tbest: 0.6657042 (100)\ttotal: 1m 6s\tremaining: 32m 1s\n",
      "200:\ttest: 0.7014081\ttest1: 0.6706778\tbest: 0.6706778 (200)\ttotal: 2m 16s\tremaining: 31m 41s\n",
      "300:\ttest: 0.7062603\ttest1: 0.6726250\tbest: 0.6726250 (300)\ttotal: 3m 25s\tremaining: 30m 44s\n",
      "400:\ttest: 0.7095174\ttest1: 0.6737507\tbest: 0.6737756 (390)\ttotal: 4m 34s\tremaining: 29m 40s\n",
      "500:\ttest: 0.7120212\ttest1: 0.6746288\tbest: 0.6746289 (499)\ttotal: 5m 44s\tremaining: 28m 38s\n",
      "600:\ttest: 0.7143875\ttest1: 0.6750444\tbest: 0.6750502 (599)\ttotal: 6m 55s\tremaining: 27m 40s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "700:\ttest: 0.7160550\ttest1: 0.6754236\tbest: 0.6754236 (700)\ttotal: 8m 7s\tremaining: 26m 37s\n",
      "800:\ttest: 0.7178938\ttest1: 0.6757614\tbest: 0.6757658 (798)\ttotal: 9m 18s\tremaining: 25m 33s\n",
      "900:\ttest: 0.7193325\ttest1: 0.6761872\tbest: 0.6761872 (900)\ttotal: 10m 30s\tremaining: 24m 29s\n",
      "1000:\ttest: 0.7207345\ttest1: 0.6765273\tbest: 0.6765675 (984)\ttotal: 11m 43s\tremaining: 23m 25s\n",
      "1100:\ttest: 0.7221588\ttest1: 0.6768072\tbest: 0.6768363 (1090)\ttotal: 12m 56s\tremaining: 22m 19s\n",
      "1200:\ttest: 0.7233876\ttest1: 0.6771213\tbest: 0.6771301 (1195)\ttotal: 14m 9s\tremaining: 21m 12s\n",
      "1300:\ttest: 0.7245437\ttest1: 0.6772482\tbest: 0.6772512 (1299)\ttotal: 15m 23s\tremaining: 20m 5s\n",
      "1400:\ttest: 0.7256834\ttest1: 0.6773163\tbest: 0.6773412 (1360)\ttotal: 16m 36s\tremaining: 18m 57s\n",
      "1500:\ttest: 0.7267296\ttest1: 0.6774092\tbest: 0.6774163 (1498)\ttotal: 17m 50s\tremaining: 17m 49s\n",
      "1600:\ttest: 0.7276818\ttest1: 0.6776333\tbest: 0.6776333 (1600)\ttotal: 19m 4s\tremaining: 16m 40s\n",
      "1700:\ttest: 0.7286675\ttest1: 0.6777344\tbest: 0.6777901 (1666)\ttotal: 20m 19s\tremaining: 15m 31s\n",
      "1800:\ttest: 0.7297122\ttest1: 0.6777499\tbest: 0.6778452 (1752)\ttotal: 21m 36s\tremaining: 14m 23s\n",
      "1900:\ttest: 0.7305293\ttest1: 0.6778913\tbest: 0.6779490 (1871)\ttotal: 22m 52s\tremaining: 13m 13s\n",
      "2000:\ttest: 0.7315384\ttest1: 0.6780781\tbest: 0.6781310 (1986)\ttotal: 24m 7s\tremaining: 12m 2s\n",
      "2100:\ttest: 0.7324206\ttest1: 0.6782779\tbest: 0.6782905 (2096)\ttotal: 25m 27s\tremaining: 10m 53s\n",
      "2200:\ttest: 0.7332400\ttest1: 0.6782978\tbest: 0.6783161 (2138)\ttotal: 26m 50s\tremaining: 9m 44s\n",
      "2300:\ttest: 0.7341986\ttest1: 0.6785441\tbest: 0.6785479 (2299)\ttotal: 28m 25s\tremaining: 8m 37s\n",
      "2400:\ttest: 0.7350519\ttest1: 0.6785496\tbest: 0.6785907 (2357)\ttotal: 30m 2s\tremaining: 7m 29s\n",
      "2500:\ttest: 0.7358156\ttest1: 0.6784415\tbest: 0.6785946 (2423)\ttotal: 31m 30s\tremaining: 6m 17s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6785945882\n",
      "bestIteration = 2423\n",
      "\n",
      "Shrink model to first 2424 iterations.\n",
      "err:  0.6785945881805754\n",
      "0:\ttest: 0.6259609\ttest1: 0.6198709\tbest: 0.6198709 (0)\ttotal: 631ms\tremaining: 31m 32s\n",
      "100:\ttest: 0.6906976\ttest1: 0.6620759\tbest: 0.6620759 (100)\ttotal: 1m 13s\tremaining: 35m 19s\n",
      "200:\ttest: 0.7011526\ttest1: 0.6686349\tbest: 0.6686349 (200)\ttotal: 2m 23s\tremaining: 33m 14s\n",
      "300:\ttest: 0.7064990\ttest1: 0.6718300\tbest: 0.6718300 (300)\ttotal: 3m 50s\tremaining: 34m 26s\n",
      "400:\ttest: 0.7095533\ttest1: 0.6733505\tbest: 0.6733564 (399)\ttotal: 5m 17s\tremaining: 34m 20s\n",
      "500:\ttest: 0.7123475\ttest1: 0.6742339\tbest: 0.6742339 (500)\ttotal: 6m 50s\tremaining: 34m 8s\n",
      "600:\ttest: 0.7144760\ttest1: 0.6748143\tbest: 0.6748143 (600)\ttotal: 8m 20s\tremaining: 33m 17s\n",
      "700:\ttest: 0.7161403\ttest1: 0.6754028\tbest: 0.6754028 (700)\ttotal: 9m 51s\tremaining: 32m 18s\n",
      "800:\ttest: 0.7178039\ttest1: 0.6758255\tbest: 0.6758613 (796)\ttotal: 11m 22s\tremaining: 31m 12s\n",
      "900:\ttest: 0.7192810\ttest1: 0.6762751\tbest: 0.6762822 (896)\ttotal: 12m 56s\tremaining: 30m 8s\n",
      "1000:\ttest: 0.7204283\ttest1: 0.6765711\tbest: 0.6765711 (1000)\ttotal: 14m 25s\tremaining: 28m 48s\n",
      "1100:\ttest: 0.7219173\ttest1: 0.6767069\tbest: 0.6767131 (1073)\ttotal: 15m 59s\tremaining: 27m 34s\n",
      "1200:\ttest: 0.7231970\ttest1: 0.6768962\tbest: 0.6768962 (1200)\ttotal: 17m 35s\tremaining: 26m 21s\n",
      "1300:\ttest: 0.7246850\ttest1: 0.6772384\tbest: 0.6772384 (1300)\ttotal: 19m 7s\tremaining: 24m 58s\n",
      "1400:\ttest: 0.7264119\ttest1: 0.6772909\tbest: 0.6772937 (1399)\ttotal: 20m 48s\tremaining: 23m 45s\n",
      "1500:\ttest: 0.7275108\ttest1: 0.6774506\tbest: 0.6774506 (1500)\ttotal: 22m 27s\tremaining: 22m 25s\n",
      "1600:\ttest: 0.7288344\ttest1: 0.6776610\tbest: 0.6776610 (1600)\ttotal: 24m 4s\tremaining: 21m 2s\n",
      "1700:\ttest: 0.7299924\ttest1: 0.6778534\tbest: 0.6778534 (1700)\ttotal: 25m 44s\tremaining: 19m 39s\n",
      "1800:\ttest: 0.7311419\ttest1: 0.6779656\tbest: 0.6779844 (1774)\ttotal: 27m 23s\tremaining: 18m 14s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6779843586\n",
      "bestIteration = 1774\n",
      "\n",
      "Shrink model to first 1775 iterations.\n",
      "err:  0.6779843586180255\n",
      "0:\ttest: 0.6248258\ttest1: 0.6228279\tbest: 0.6228279 (0)\ttotal: 630ms\tremaining: 31m 28s\n",
      "100:\ttest: 0.6899329\ttest1: 0.6701601\tbest: 0.6701601 (100)\ttotal: 1m 28s\tremaining: 42m 21s\n",
      "200:\ttest: 0.6995319\ttest1: 0.6756683\tbest: 0.6757024 (199)\ttotal: 2m 55s\tremaining: 40m 42s\n",
      "300:\ttest: 0.7046119\ttest1: 0.6783649\tbest: 0.6783649 (300)\ttotal: 4m 25s\tremaining: 39m 40s\n",
      "400:\ttest: 0.7080227\ttest1: 0.6796551\tbest: 0.6796607 (398)\ttotal: 5m 58s\tremaining: 38m 41s\n",
      "500:\ttest: 0.7107424\ttest1: 0.6807428\tbest: 0.6807551 (499)\ttotal: 7m 27s\tremaining: 37m 10s\n",
      "600:\ttest: 0.7130082\ttest1: 0.6813758\tbest: 0.6813823 (596)\ttotal: 8m 57s\tremaining: 35m 45s\n",
      "700:\ttest: 0.7148086\ttest1: 0.6819565\tbest: 0.6819565 (700)\ttotal: 10m 26s\tremaining: 34m 15s\n",
      "800:\ttest: 0.7163911\ttest1: 0.6823192\tbest: 0.6823192 (800)\ttotal: 12m 2s\tremaining: 33m 3s\n",
      "900:\ttest: 0.7180600\ttest1: 0.6827090\tbest: 0.6827119 (897)\ttotal: 13m 42s\tremaining: 31m 56s\n",
      "1000:\ttest: 0.7194180\ttest1: 0.6830270\tbest: 0.6830270 (1000)\ttotal: 15m 16s\tremaining: 30m 30s\n",
      "1100:\ttest: 0.7205534\ttest1: 0.6831592\tbest: 0.6831592 (1100)\ttotal: 16m 46s\tremaining: 28m 55s\n",
      "1200:\ttest: 0.7217874\ttest1: 0.6834399\tbest: 0.6834756 (1186)\ttotal: 18m 16s\tremaining: 27m 22s\n",
      "1300:\ttest: 0.7229754\ttest1: 0.6835124\tbest: 0.6836166 (1237)\ttotal: 19m 50s\tremaining: 25m 55s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.683616564\n",
      "bestIteration = 1237\n",
      "\n",
      "Shrink model to first 1238 iterations.\n",
      "err:  0.6836165639801294\n",
      "0:\ttest: 0.6248104\ttest1: 0.6292683\tbest: 0.6292683 (0)\ttotal: 617ms\tremaining: 30m 51s\n",
      "100:\ttest: 0.6898125\ttest1: 0.6748798\tbest: 0.6748798 (100)\ttotal: 1m 24s\tremaining: 40m 16s\n",
      "200:\ttest: 0.6994860\ttest1: 0.6808820\tbest: 0.6808820 (200)\ttotal: 2m 49s\tremaining: 39m 15s\n",
      "300:\ttest: 0.7043406\ttest1: 0.6834943\tbest: 0.6834943 (300)\ttotal: 4m 18s\tremaining: 38m 33s\n",
      "400:\ttest: 0.7084075\ttest1: 0.6846496\tbest: 0.6846954 (398)\ttotal: 5m 47s\tremaining: 37m 33s\n",
      "500:\ttest: 0.7116589\ttest1: 0.6852110\tbest: 0.6852110 (500)\ttotal: 7m 19s\tremaining: 36m 32s\n",
      "600:\ttest: 0.7143815\ttest1: 0.6857850\tbest: 0.6858039 (597)\ttotal: 8m 46s\tremaining: 35m\n",
      "700:\ttest: 0.7161694\ttest1: 0.6860436\tbest: 0.6860463 (691)\ttotal: 10m 13s\tremaining: 33m 32s\n",
      "800:\ttest: 0.7179359\ttest1: 0.6862929\tbest: 0.6863101 (794)\ttotal: 11m 47s\tremaining: 32m 21s\n",
      "900:\ttest: 0.7195144\ttest1: 0.6864246\tbest: 0.6864403 (899)\ttotal: 13m 18s\tremaining: 30m 59s\n",
      "1000:\ttest: 0.7209071\ttest1: 0.6866435\tbest: 0.6866490 (998)\ttotal: 14m 50s\tremaining: 29m 37s\n",
      "1100:\ttest: 0.7225568\ttest1: 0.6866326\tbest: 0.6866694 (1062)\ttotal: 16m 23s\tremaining: 28m 15s\n",
      "1200:\ttest: 0.7236615\ttest1: 0.6867712\tbest: 0.6867798 (1199)\ttotal: 17m 58s\tremaining: 26m 55s\n",
      "1300:\ttest: 0.7248648\ttest1: 0.6868461\tbest: 0.6868865 (1277)\ttotal: 19m 32s\tremaining: 25m 31s\n",
      "1400:\ttest: 0.7262052\ttest1: 0.6870214\tbest: 0.6870275 (1399)\ttotal: 21m 3s\tremaining: 24m 1s\n",
      "1500:\ttest: 0.7273854\ttest1: 0.6868933\tbest: 0.6870502 (1408)\ttotal: 22m 37s\tremaining: 22m 35s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6870502366\n",
      "bestIteration = 1408\n",
      "\n",
      "Shrink model to first 1409 iterations.\n",
      "err:  0.6870502366474661\n",
      "0:\ttest: 0.6235737\ttest1: 0.6359907\tbest: 0.6359907 (0)\ttotal: 629ms\tremaining: 31m 27s\n",
      "100:\ttest: 0.6915377\ttest1: 0.6802584\tbest: 0.6802584 (100)\ttotal: 1m 25s\tremaining: 40m 44s\n",
      "200:\ttest: 0.7014228\ttest1: 0.6861090\tbest: 0.6861090 (200)\ttotal: 2m 53s\tremaining: 40m 13s\n",
      "300:\ttest: 0.7059080\ttest1: 0.6887708\tbest: 0.6887708 (300)\ttotal: 4m 20s\tremaining: 38m 56s\n",
      "400:\ttest: 0.7089602\ttest1: 0.6900718\tbest: 0.6900834 (398)\ttotal: 5m 49s\tremaining: 37m 44s\n",
      "500:\ttest: 0.7114341\ttest1: 0.6906471\tbest: 0.6906887 (489)\ttotal: 7m 11s\tremaining: 35m 54s\n",
      "600:\ttest: 0.7137683\ttest1: 0.6910685\tbest: 0.6910696 (597)\ttotal: 8m 38s\tremaining: 34m 29s\n",
      "700:\ttest: 0.7155027\ttest1: 0.6916612\tbest: 0.6916612 (700)\ttotal: 10m 6s\tremaining: 33m 10s\n",
      "800:\ttest: 0.7171802\ttest1: 0.6921072\tbest: 0.6921088 (799)\ttotal: 11m 37s\tremaining: 31m 53s\n",
      "900:\ttest: 0.7187639\ttest1: 0.6923199\tbest: 0.6923199 (900)\ttotal: 13m 4s\tremaining: 30m 28s\n",
      "1000:\ttest: 0.7201671\ttest1: 0.6924518\tbest: 0.6924931 (983)\ttotal: 14m 35s\tremaining: 29m 9s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6924930634\n",
      "bestIteration = 983\n",
      "\n",
      "Shrink model to first 984 iterations.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "err:  0.6924930634078361\n",
      "0:\ttest: 0.6218397\ttest1: 0.6117387\tbest: 0.6117387 (0)\ttotal: 901ms\tremaining: 45m 1s\n",
      "100:\ttest: 0.6896418\ttest1: 0.6629047\tbest: 0.6629047 (100)\ttotal: 1m 25s\tremaining: 40m 50s\n",
      "200:\ttest: 0.6996642\ttest1: 0.6699332\tbest: 0.6699332 (200)\ttotal: 2m 52s\tremaining: 40m\n",
      "300:\ttest: 0.7046280\ttest1: 0.6732518\tbest: 0.6732691 (299)\ttotal: 4m 21s\tremaining: 39m 4s\n",
      "400:\ttest: 0.7081828\ttest1: 0.6753162\tbest: 0.6753162 (400)\ttotal: 5m 48s\tremaining: 37m 40s\n",
      "500:\ttest: 0.7110512\ttest1: 0.6763600\tbest: 0.6763709 (497)\ttotal: 7m 17s\tremaining: 36m 21s\n",
      "600:\ttest: 0.7133232\ttest1: 0.6770069\tbest: 0.6770119 (591)\ttotal: 8m 48s\tremaining: 35m 7s\n",
      "700:\ttest: 0.7150586\ttest1: 0.6775941\tbest: 0.6775941 (700)\ttotal: 10m 15s\tremaining: 33m 37s\n",
      "800:\ttest: 0.7166268\ttest1: 0.6780820\tbest: 0.6780820 (800)\ttotal: 11m 51s\tremaining: 32m 32s\n",
      "900:\ttest: 0.7181712\ttest1: 0.6785842\tbest: 0.6785842 (900)\ttotal: 13m 24s\tremaining: 31m 14s\n",
      "1000:\ttest: 0.7193674\ttest1: 0.6789229\tbest: 0.6789523 (984)\ttotal: 14m 54s\tremaining: 29m 47s\n",
      "1100:\ttest: 0.7205302\ttest1: 0.6792609\tbest: 0.6792609 (1100)\ttotal: 16m 24s\tremaining: 28m 18s\n",
      "1200:\ttest: 0.7219551\ttest1: 0.6793012\tbest: 0.6793198 (1115)\ttotal: 17m 55s\tremaining: 26m 50s\n",
      "1300:\ttest: 0.7229974\ttest1: 0.6796519\tbest: 0.6796844 (1293)\ttotal: 19m 26s\tremaining: 25m 23s\n",
      "1400:\ttest: 0.7241362\ttest1: 0.6798074\tbest: 0.6798393 (1397)\ttotal: 21m 1s\tremaining: 23m 59s\n",
      "1500:\ttest: 0.7251207\ttest1: 0.6800080\tbest: 0.6800287 (1472)\ttotal: 22m 34s\tremaining: 22m 33s\n",
      "1600:\ttest: 0.7262956\ttest1: 0.6802680\tbest: 0.6802680 (1600)\ttotal: 24m 8s\tremaining: 21m 6s\n",
      "1700:\ttest: 0.7273790\ttest1: 0.6803297\tbest: 0.6803960 (1673)\ttotal: 25m 43s\tremaining: 19m 38s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6803960018\n",
      "bestIteration = 1673\n",
      "\n",
      "Shrink model to first 1674 iterations.\n",
      "err:  0.6803960017859532\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score,confusion_matrix,roc_auc_score\n",
    "errcb=[]\n",
    "y_pred_totcb=[]\n",
    "from sklearn.model_selection import KFold,StratifiedKFold\n",
    "fold=StratifiedKFold(n_splits=20,shuffle=True,random_state=1994)\n",
    "i=1\n",
    "for train_index, test_index in fold.split(X,y):\n",
    "    X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n",
    "    y_train, y_test = y[train_index], y[test_index]\n",
    "    m=CatBoostClassifier(n_estimators=3000,random_state=1994,eval_metric='AUC',max_depth=6,learning_rate=0.029,od_wait=50\n",
    "                     ,l2_leaf_reg=10,cat_features=categorical_features_indices,bagging_temperature=0.85,random_strength=100,\n",
    "                     use_best_model=True)\n",
    "    m.fit(X_train,y_train,eval_set=[(X_train,y_train),(X_test, y_test)], early_stopping_rounds=100,verbose=100)\n",
    "    preds=m.predict_proba(X_test)[:,-1]\n",
    "    print(\"err: \",roc_auc_score(y_test,preds))\n",
    "    errcb.append(roc_auc_score(y_test,preds))\n",
    "    p = m.predict_proba(dftest)[:,-1]\n",
    "    y_pred_totcb.append(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\ttest: 0.6213369\ttest1: 0.6195963\tbest: 0.6195963 (0)\ttotal: 1.79s\tremaining: 1h 29m 37s\n",
      "100:\ttest: 0.6981124\ttest1: 0.6719014\tbest: 0.6719014 (100)\ttotal: 2m 11s\tremaining: 1h 2m 52s\n",
      "200:\ttest: 0.7055669\ttest1: 0.6762466\tbest: 0.6762638 (198)\ttotal: 4m 10s\tremaining: 58m 3s\n",
      "300:\ttest: 0.7104634\ttest1: 0.6781424\tbest: 0.6781508 (299)\ttotal: 6m 22s\tremaining: 57m 9s\n",
      "400:\ttest: 0.7142091\ttest1: 0.6793192\tbest: 0.6793330 (394)\ttotal: 8m 12s\tremaining: 53m 8s\n",
      "500:\ttest: 0.7174067\ttest1: 0.6801475\tbest: 0.6801475 (500)\ttotal: 10m\tremaining: 49m 57s\n",
      "600:\ttest: 0.7201182\ttest1: 0.6805944\tbest: 0.6806643 (590)\ttotal: 11m 48s\tremaining: 47m 8s\n",
      "700:\ttest: 0.7220414\ttest1: 0.6809836\tbest: 0.6809836 (700)\ttotal: 13m 35s\tremaining: 44m 33s\n",
      "800:\ttest: 0.7239852\ttest1: 0.6812761\tbest: 0.6812779 (799)\ttotal: 15m 23s\tremaining: 42m 14s\n",
      "900:\ttest: 0.7259074\ttest1: 0.6816682\tbest: 0.6816854 (885)\ttotal: 17m 10s\tremaining: 40m 1s\n",
      "1000:\ttest: 0.7275491\ttest1: 0.6819562\tbest: 0.6819562 (1000)\ttotal: 18m 58s\tremaining: 37m 53s\n",
      "1100:\ttest: 0.7291136\ttest1: 0.6820895\tbest: 0.6820895 (1100)\ttotal: 20m 47s\tremaining: 35m 51s\n",
      "1200:\ttest: 0.7305760\ttest1: 0.6821406\tbest: 0.6822131 (1134)\ttotal: 22m 35s\tremaining: 33m 49s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6822130858\n",
      "bestIteration = 1134\n",
      "\n",
      "Shrink model to first 1135 iterations.\n",
      "err:  0.6822130858104505\n",
      "0:\ttest: 0.6235820\ttest1: 0.6147475\tbest: 0.6147475 (0)\ttotal: 1.11s\tremaining: 55m 42s\n",
      "100:\ttest: 0.6975692\ttest1: 0.6708222\tbest: 0.6708222 (100)\ttotal: 1m 45s\tremaining: 50m 22s\n",
      "200:\ttest: 0.7058673\ttest1: 0.6755687\tbest: 0.6755687 (200)\ttotal: 3m 30s\tremaining: 48m 50s\n",
      "300:\ttest: 0.7100458\ttest1: 0.6774657\tbest: 0.6774657 (300)\ttotal: 5m 16s\tremaining: 47m 18s\n",
      "400:\ttest: 0.7137707\ttest1: 0.6787591\tbest: 0.6787591 (400)\ttotal: 7m 3s\tremaining: 45m 46s\n",
      "500:\ttest: 0.7163285\ttest1: 0.6796033\tbest: 0.6796033 (500)\ttotal: 8m 52s\tremaining: 44m 17s\n",
      "600:\ttest: 0.7184287\ttest1: 0.6800921\tbest: 0.6800946 (599)\ttotal: 10m 40s\tremaining: 42m 36s\n",
      "700:\ttest: 0.7204730\ttest1: 0.6805689\tbest: 0.6805689 (700)\ttotal: 12m 27s\tremaining: 40m 52s\n",
      "800:\ttest: 0.7223588\ttest1: 0.6808530\tbest: 0.6808530 (800)\ttotal: 14m 15s\tremaining: 39m 7s\n",
      "900:\ttest: 0.7242657\ttest1: 0.6811782\tbest: 0.6811889 (899)\ttotal: 16m 5s\tremaining: 37m 29s\n",
      "1000:\ttest: 0.7260298\ttest1: 0.6812892\tbest: 0.6813286 (962)\ttotal: 17m 54s\tremaining: 35m 45s\n",
      "1100:\ttest: 0.7278347\ttest1: 0.6814963\tbest: 0.6814963 (1100)\ttotal: 19m 48s\tremaining: 34m 10s\n",
      "1200:\ttest: 0.7297271\ttest1: 0.6817708\tbest: 0.6817914 (1196)\ttotal: 21m 41s\tremaining: 32m 29s\n",
      "1300:\ttest: 0.7312970\ttest1: 0.6820149\tbest: 0.6820149 (1300)\ttotal: 23m 25s\tremaining: 30m 35s\n",
      "1400:\ttest: 0.7328784\ttest1: 0.6818453\tbest: 0.6820201 (1310)\ttotal: 25m 17s\tremaining: 28m 52s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6820201485\n",
      "bestIteration = 1310\n",
      "\n",
      "Shrink model to first 1311 iterations.\n",
      "err:  0.6820201484913364\n",
      "0:\ttest: 0.6223588\ttest1: 0.6295011\tbest: 0.6295011 (0)\ttotal: 1.39s\tremaining: 1h 9m 37s\n",
      "100:\ttest: 0.6991930\ttest1: 0.6803456\tbest: 0.6803456 (100)\ttotal: 2m 17s\tremaining: 1h 5m 52s\n",
      "200:\ttest: 0.7068944\ttest1: 0.6849174\tbest: 0.6849174 (200)\ttotal: 4m 36s\tremaining: 1h 4m 14s\n",
      "300:\ttest: 0.7116002\ttest1: 0.6867384\tbest: 0.6867465 (298)\ttotal: 6m 56s\tremaining: 1h 2m 10s\n",
      "400:\ttest: 0.7146307\ttest1: 0.6876667\tbest: 0.6876667 (400)\ttotal: 9m 15s\tremaining: 59m 58s\n",
      "500:\ttest: 0.7170828\ttest1: 0.6883404\tbest: 0.6883502 (498)\ttotal: 11m 36s\tremaining: 57m 52s\n",
      "600:\ttest: 0.7193291\ttest1: 0.6887698\tbest: 0.6887701 (599)\ttotal: 13m 56s\tremaining: 55m 40s\n",
      "700:\ttest: 0.7210806\ttest1: 0.6891830\tbest: 0.6891921 (698)\ttotal: 16m 18s\tremaining: 53m 28s\n",
      "800:\ttest: 0.7228434\ttest1: 0.6896411\tbest: 0.6896494 (782)\ttotal: 18m 39s\tremaining: 51m 14s\n",
      "900:\ttest: 0.7245136\ttest1: 0.6900649\tbest: 0.6900799 (894)\ttotal: 20m 54s\tremaining: 48m 42s\n",
      "1000:\ttest: 0.7261771\ttest1: 0.6902520\tbest: 0.6902538 (998)\ttotal: 23m 11s\tremaining: 46m 17s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6902537766\n",
      "bestIteration = 998\n",
      "\n",
      "Shrink model to first 999 iterations.\n",
      "err:  0.6902537765635581\n",
      "0:\ttest: 0.6229849\ttest1: 0.6148701\tbest: 0.6148701 (0)\ttotal: 1.35s\tremaining: 1h 7m 25s\n",
      "100:\ttest: 0.6968604\ttest1: 0.6746094\tbest: 0.6746094 (100)\ttotal: 2m 13s\tremaining: 1h 3m 54s\n",
      "200:\ttest: 0.7045175\ttest1: 0.6789394\tbest: 0.6789394 (200)\ttotal: 4m 26s\tremaining: 1h 1m 48s\n",
      "300:\ttest: 0.7098741\ttest1: 0.6808855\tbest: 0.6808888 (295)\ttotal: 6m 41s\tremaining: 1h\n",
      "400:\ttest: 0.7133048\ttest1: 0.6816008\tbest: 0.6816268 (399)\ttotal: 8m 57s\tremaining: 58m 4s\n",
      "500:\ttest: 0.7163110\ttest1: 0.6823078\tbest: 0.6823078 (500)\ttotal: 11m 14s\tremaining: 56m 6s\n",
      "600:\ttest: 0.7186025\ttest1: 0.6825933\tbest: 0.6825933 (600)\ttotal: 13m 32s\tremaining: 54m 4s\n",
      "700:\ttest: 0.7207478\ttest1: 0.6829172\tbest: 0.6829573 (684)\ttotal: 15m 52s\tremaining: 52m 5s\n",
      "800:\ttest: 0.7226949\ttest1: 0.6831042\tbest: 0.6831042 (800)\ttotal: 18m 12s\tremaining: 49m 58s\n",
      "900:\ttest: 0.7242299\ttest1: 0.6831822\tbest: 0.6832065 (827)\ttotal: 20m 31s\tremaining: 47m 49s\n",
      "1000:\ttest: 0.7260949\ttest1: 0.6835772\tbest: 0.6835772 (1000)\ttotal: 22m 49s\tremaining: 45m 34s\n",
      "1100:\ttest: 0.7279073\ttest1: 0.6836958\tbest: 0.6836965 (1092)\ttotal: 25m 11s\tremaining: 43m 27s\n",
      "1200:\ttest: 0.7293679\ttest1: 0.6838999\tbest: 0.6838999 (1200)\ttotal: 27m 28s\tremaining: 41m 9s\n",
      "1300:\ttest: 0.7307119\ttest1: 0.6840152\tbest: 0.6840256 (1289)\ttotal: 29m 48s\tremaining: 38m 55s\n",
      "1400:\ttest: 0.7323591\ttest1: 0.6840144\tbest: 0.6840463 (1305)\ttotal: 32m 7s\tremaining: 36m 39s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6840462756\n",
      "bestIteration = 1305\n",
      "\n",
      "Shrink model to first 1306 iterations.\n",
      "err:  0.6840462756258168\n",
      "0:\ttest: 0.6237374\ttest1: 0.6133749\tbest: 0.6133749 (0)\ttotal: 1.44s\tremaining: 1h 11m 46s\n",
      "100:\ttest: 0.7010739\ttest1: 0.6646566\tbest: 0.6646566 (100)\ttotal: 2m 12s\tremaining: 1h 3m 35s\n",
      "200:\ttest: 0.7079725\ttest1: 0.6687937\tbest: 0.6687937 (200)\ttotal: 4m 26s\tremaining: 1h 1m 46s\n",
      "300:\ttest: 0.7121802\ttest1: 0.6703933\tbest: 0.6704244 (299)\ttotal: 6m 39s\tremaining: 59m 46s\n",
      "400:\ttest: 0.7153444\ttest1: 0.6714090\tbest: 0.6714090 (400)\ttotal: 8m 54s\tremaining: 57m 47s\n",
      "500:\ttest: 0.7178556\ttest1: 0.6720347\tbest: 0.6720909 (495)\ttotal: 11m 10s\tremaining: 55m 44s\n",
      "600:\ttest: 0.7198472\ttest1: 0.6723604\tbest: 0.6723616 (599)\ttotal: 13m 27s\tremaining: 53m 42s\n",
      "700:\ttest: 0.7220309\ttest1: 0.6726632\tbest: 0.6726632 (700)\ttotal: 15m 44s\tremaining: 51m 39s\n",
      "800:\ttest: 0.7239384\ttest1: 0.6729579\tbest: 0.6729579 (800)\ttotal: 18m 4s\tremaining: 49m 37s\n",
      "900:\ttest: 0.7258757\ttest1: 0.6733501\tbest: 0.6733501 (900)\ttotal: 20m 13s\tremaining: 47m 6s\n",
      "1000:\ttest: 0.7275239\ttest1: 0.6734353\tbest: 0.6734885 (987)\ttotal: 22m\tremaining: 43m 57s\n",
      "1100:\ttest: 0.7292246\ttest1: 0.6736884\tbest: 0.6736884 (1100)\ttotal: 23m 51s\tremaining: 41m 8s\n",
      "1200:\ttest: 0.7306503\ttest1: 0.6738423\tbest: 0.6738423 (1200)\ttotal: 25m 42s\tremaining: 38m 30s\n",
      "1300:\ttest: 0.7321156\ttest1: 0.6741690\tbest: 0.6741690 (1300)\ttotal: 27m 33s\tremaining: 35m 58s\n",
      "1400:\ttest: 0.7336338\ttest1: 0.6743094\tbest: 0.6743099 (1398)\ttotal: 29m 23s\tremaining: 33m 32s\n",
      "1500:\ttest: 0.7350451\ttest1: 0.6746020\tbest: 0.6746193 (1476)\ttotal: 31m 14s\tremaining: 31m 11s\n",
      "1600:\ttest: 0.7365105\ttest1: 0.6747373\tbest: 0.6747438 (1595)\ttotal: 33m 3s\tremaining: 28m 53s\n",
      "1700:\ttest: 0.7376889\ttest1: 0.6748919\tbest: 0.6749123 (1696)\ttotal: 34m 53s\tremaining: 26m 39s\n",
      "1800:\ttest: 0.7392242\ttest1: 0.6749712\tbest: 0.6749864 (1768)\ttotal: 36m 44s\tremaining: 24m 27s\n",
      "1900:\ttest: 0.7404274\ttest1: 0.6750273\tbest: 0.6750283 (1896)\ttotal: 38m 36s\tremaining: 22m 19s\n",
      "2000:\ttest: 0.7421673\ttest1: 0.6750407\tbest: 0.6750528 (1990)\ttotal: 40m 27s\tremaining: 20m 11s\n",
      "2100:\ttest: 0.7436542\ttest1: 0.6750050\tbest: 0.6750859 (2051)\ttotal: 42m 18s\tremaining: 18m 6s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6750858979\n",
      "bestIteration = 2051\n",
      "\n",
      "Shrink model to first 2052 iterations.\n",
      "err:  0.6750858979202498\n",
      "0:\ttest: 0.6214847\ttest1: 0.6261067\tbest: 0.6261067 (0)\ttotal: 1.15s\tremaining: 57m 33s\n",
      "100:\ttest: 0.6978871\ttest1: 0.6792061\tbest: 0.6792061 (100)\ttotal: 1m 45s\tremaining: 50m 16s\n",
      "200:\ttest: 0.7048481\ttest1: 0.6826030\tbest: 0.6826030 (200)\ttotal: 3m 32s\tremaining: 49m 17s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "300:\ttest: 0.7096651\ttest1: 0.6838672\tbest: 0.6838672 (300)\ttotal: 5m 20s\tremaining: 47m 50s\n",
      "400:\ttest: 0.7128202\ttest1: 0.6844667\tbest: 0.6844667 (400)\ttotal: 7m 7s\tremaining: 46m 12s\n",
      "500:\ttest: 0.7156778\ttest1: 0.6851212\tbest: 0.6851212 (500)\ttotal: 8m 55s\tremaining: 44m 30s\n",
      "600:\ttest: 0.7176453\ttest1: 0.6854359\tbest: 0.6854797 (596)\ttotal: 10m 43s\tremaining: 42m 50s\n",
      "700:\ttest: 0.7200977\ttest1: 0.6855448\tbest: 0.6855644 (672)\ttotal: 12m 36s\tremaining: 41m 19s\n",
      "800:\ttest: 0.7221989\ttest1: 0.6859183\tbest: 0.6859318 (794)\ttotal: 14m 26s\tremaining: 39m 39s\n",
      "900:\ttest: 0.7240577\ttest1: 0.6859862\tbest: 0.6860232 (875)\ttotal: 16m 17s\tremaining: 37m 58s\n",
      "1000:\ttest: 0.7261459\ttest1: 0.6862396\tbest: 0.6862396 (1000)\ttotal: 18m 8s\tremaining: 36m 14s\n",
      "1100:\ttest: 0.7278783\ttest1: 0.6863492\tbest: 0.6863698 (1077)\ttotal: 19m 59s\tremaining: 34m 29s\n",
      "1200:\ttest: 0.7297762\ttest1: 0.6864110\tbest: 0.6864677 (1154)\ttotal: 21m 51s\tremaining: 32m 45s\n",
      "1300:\ttest: 0.7315974\ttest1: 0.6865857\tbest: 0.6866102 (1294)\ttotal: 23m 47s\tremaining: 31m 3s\n",
      "1400:\ttest: 0.7335005\ttest1: 0.6865766\tbest: 0.6866648 (1382)\ttotal: 25m 45s\tremaining: 29m 23s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6866647629\n",
      "bestIteration = 1382\n",
      "\n",
      "Shrink model to first 1383 iterations.\n",
      "err:  0.6866647628633747\n",
      "0:\ttest: 0.6258923\ttest1: 0.6164015\tbest: 0.6164015 (0)\ttotal: 1.14s\tremaining: 57m 13s\n",
      "100:\ttest: 0.6985872\ttest1: 0.6660104\tbest: 0.6660104 (100)\ttotal: 1m 45s\tremaining: 50m 28s\n",
      "200:\ttest: 0.7061503\ttest1: 0.6706253\tbest: 0.6706253 (200)\ttotal: 3m 30s\tremaining: 48m 56s\n",
      "300:\ttest: 0.7102362\ttest1: 0.6728095\tbest: 0.6728095 (300)\ttotal: 5m 16s\tremaining: 47m 13s\n",
      "400:\ttest: 0.7134880\ttest1: 0.6738862\tbest: 0.6738994 (399)\ttotal: 7m 2s\tremaining: 45m 38s\n",
      "500:\ttest: 0.7159279\ttest1: 0.6746015\tbest: 0.6746015 (500)\ttotal: 8m 48s\tremaining: 43m 57s\n",
      "600:\ttest: 0.7183480\ttest1: 0.6751896\tbest: 0.6751896 (600)\ttotal: 10m 37s\tremaining: 42m 25s\n",
      "700:\ttest: 0.7203023\ttest1: 0.6755656\tbest: 0.6755656 (700)\ttotal: 12m 25s\tremaining: 40m 46s\n",
      "800:\ttest: 0.7222796\ttest1: 0.6760413\tbest: 0.6760669 (797)\ttotal: 14m 13s\tremaining: 39m 3s\n",
      "900:\ttest: 0.7241779\ttest1: 0.6764454\tbest: 0.6764454 (900)\ttotal: 16m 2s\tremaining: 37m 22s\n",
      "1000:\ttest: 0.7259434\ttest1: 0.6769371\tbest: 0.6769589 (999)\ttotal: 17m 51s\tremaining: 35m 40s\n",
      "1100:\ttest: 0.7275920\ttest1: 0.6770566\tbest: 0.6771160 (1062)\ttotal: 19m 42s\tremaining: 33m 58s\n",
      "1200:\ttest: 0.7297618\ttest1: 0.6773628\tbest: 0.6773882 (1194)\ttotal: 21m 32s\tremaining: 32m 16s\n",
      "1300:\ttest: 0.7313473\ttest1: 0.6775823\tbest: 0.6776361 (1284)\ttotal: 23m 22s\tremaining: 30m 31s\n",
      "1400:\ttest: 0.7329699\ttest1: 0.6776241\tbest: 0.6776660 (1380)\ttotal: 25m 12s\tremaining: 28m 46s\n",
      "1500:\ttest: 0.7346618\ttest1: 0.6779127\tbest: 0.6779550 (1491)\ttotal: 26m 56s\tremaining: 26m 54s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.677955036\n",
      "bestIteration = 1491\n",
      "\n",
      "Shrink model to first 1492 iterations.\n",
      "err:  0.6779550359937674\n",
      "0:\ttest: 0.6254269\ttest1: 0.6228617\tbest: 0.6228617 (0)\ttotal: 1.33s\tremaining: 1h 6m 32s\n",
      "100:\ttest: 0.7010396\ttest1: 0.6683166\tbest: 0.6683166 (100)\ttotal: 2m 9s\tremaining: 1h 2m 9s\n",
      "200:\ttest: 0.7094409\ttest1: 0.6725966\tbest: 0.6725966 (200)\ttotal: 4m 21s\tremaining: 1h 35s\n",
      "300:\ttest: 0.7138341\ttest1: 0.6743565\tbest: 0.6743640 (299)\ttotal: 6m 35s\tremaining: 59m 4s\n",
      "400:\ttest: 0.7170714\ttest1: 0.6752083\tbest: 0.6752375 (397)\ttotal: 8m 49s\tremaining: 57m 11s\n",
      "500:\ttest: 0.7201725\ttest1: 0.6755948\tbest: 0.6757139 (475)\ttotal: 11m 5s\tremaining: 55m 21s\n",
      "600:\ttest: 0.7223597\ttest1: 0.6760532\tbest: 0.6760566 (599)\ttotal: 13m 21s\tremaining: 53m 20s\n",
      "700:\ttest: 0.7242405\ttest1: 0.6763845\tbest: 0.6764618 (685)\ttotal: 15m 38s\tremaining: 51m 16s\n",
      "800:\ttest: 0.7257639\ttest1: 0.6765539\tbest: 0.6765695 (785)\ttotal: 17m 55s\tremaining: 49m 12s\n",
      "900:\ttest: 0.7275955\ttest1: 0.6766865\tbest: 0.6767122 (886)\ttotal: 20m 14s\tremaining: 47m 8s\n",
      "1000:\ttest: 0.7290209\ttest1: 0.6770832\tbest: 0.6770884 (988)\ttotal: 22m 32s\tremaining: 45m\n",
      "1100:\ttest: 0.7305162\ttest1: 0.6771972\tbest: 0.6772021 (1095)\ttotal: 24m 50s\tremaining: 42m 50s\n",
      "1200:\ttest: 0.7319378\ttest1: 0.6772497\tbest: 0.6772615 (1142)\ttotal: 27m 8s\tremaining: 40m 39s\n",
      "1300:\ttest: 0.7332955\ttest1: 0.6775251\tbest: 0.6775397 (1296)\ttotal: 29m 26s\tremaining: 38m 26s\n",
      "1400:\ttest: 0.7347619\ttest1: 0.6777119\tbest: 0.6777240 (1398)\ttotal: 31m 44s\tremaining: 36m 13s\n",
      "1500:\ttest: 0.7364311\ttest1: 0.6776848\tbest: 0.6777816 (1439)\ttotal: 34m 4s\tremaining: 34m 1s\n",
      "1600:\ttest: 0.7378411\ttest1: 0.6778371\tbest: 0.6778408 (1599)\ttotal: 36m 22s\tremaining: 31m 47s\n",
      "1700:\ttest: 0.7392896\ttest1: 0.6779210\tbest: 0.6779239 (1671)\ttotal: 38m 42s\tremaining: 29m 33s\n",
      "1800:\ttest: 0.7408369\ttest1: 0.6780369\tbest: 0.6780370 (1799)\ttotal: 41m 1s\tremaining: 27m 18s\n",
      "1900:\ttest: 0.7421337\ttest1: 0.6780188\tbest: 0.6780529 (1802)\ttotal: 43m 20s\tremaining: 25m 3s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6780529323\n",
      "bestIteration = 1802\n",
      "\n",
      "Shrink model to first 1803 iterations.\n",
      "err:  0.6780529322694809\n",
      "0:\ttest: 0.6241174\ttest1: 0.6234224\tbest: 0.6234224 (0)\ttotal: 1.49s\tremaining: 1h 14m 37s\n",
      "100:\ttest: 0.6972827\ttest1: 0.6769849\tbest: 0.6769849 (100)\ttotal: 2m 11s\tremaining: 1h 3m 7s\n",
      "200:\ttest: 0.7055825\ttest1: 0.6806246\tbest: 0.6806333 (199)\ttotal: 4m 25s\tremaining: 1h 1m 36s\n",
      "300:\ttest: 0.7102137\ttest1: 0.6817881\tbest: 0.6817881 (300)\ttotal: 6m 38s\tremaining: 59m 36s\n",
      "400:\ttest: 0.7141252\ttest1: 0.6828531\tbest: 0.6828705 (397)\ttotal: 8m 54s\tremaining: 57m 46s\n",
      "500:\ttest: 0.7170751\ttest1: 0.6833519\tbest: 0.6833643 (494)\ttotal: 11m 9s\tremaining: 55m 40s\n",
      "600:\ttest: 0.7194634\ttest1: 0.6836285\tbest: 0.6836677 (584)\ttotal: 13m 26s\tremaining: 53m 39s\n",
      "700:\ttest: 0.7214723\ttest1: 0.6840231\tbest: 0.6840231 (700)\ttotal: 15m 44s\tremaining: 51m 36s\n",
      "800:\ttest: 0.7232192\ttest1: 0.6844747\tbest: 0.6844762 (799)\ttotal: 17m 52s\tremaining: 49m 4s\n",
      "900:\ttest: 0.7254302\ttest1: 0.6845082\tbest: 0.6845082 (900)\ttotal: 19m 53s\tremaining: 46m 21s\n",
      "1000:\ttest: 0.7279339\ttest1: 0.6846107\tbest: 0.6847435 (967)\ttotal: 22m 3s\tremaining: 44m 3s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6847434673\n",
      "bestIteration = 967\n",
      "\n",
      "Shrink model to first 968 iterations.\n",
      "err:  0.6847434672691639\n",
      "0:\ttest: 0.6229028\ttest1: 0.6266405\tbest: 0.6266405 (0)\ttotal: 1.36s\tremaining: 1h 7m 52s\n",
      "100:\ttest: 0.6986143\ttest1: 0.6750348\tbest: 0.6750348 (100)\ttotal: 2m 7s\tremaining: 1h 51s\n",
      "200:\ttest: 0.7068503\ttest1: 0.6802927\tbest: 0.6802927 (200)\ttotal: 4m 17s\tremaining: 59m 51s\n",
      "300:\ttest: 0.7116617\ttest1: 0.6821052\tbest: 0.6821052 (300)\ttotal: 6m 25s\tremaining: 57m 39s\n",
      "400:\ttest: 0.7148564\ttest1: 0.6830851\tbest: 0.6830851 (400)\ttotal: 8m 36s\tremaining: 55m 48s\n",
      "500:\ttest: 0.7173919\ttest1: 0.6838827\tbest: 0.6838827 (500)\ttotal: 10m 46s\tremaining: 53m 45s\n",
      "600:\ttest: 0.7197508\ttest1: 0.6844301\tbest: 0.6844301 (600)\ttotal: 12m 58s\tremaining: 51m 49s\n",
      "700:\ttest: 0.7215551\ttest1: 0.6847946\tbest: 0.6847946 (700)\ttotal: 15m 9s\tremaining: 49m 43s\n",
      "800:\ttest: 0.7235252\ttest1: 0.6850756\tbest: 0.6850792 (791)\ttotal: 17m 21s\tremaining: 47m 39s\n",
      "900:\ttest: 0.7258008\ttest1: 0.6853299\tbest: 0.6853372 (868)\ttotal: 19m 34s\tremaining: 45m 35s\n",
      "1000:\ttest: 0.7275961\ttest1: 0.6853636\tbest: 0.6853741 (999)\ttotal: 21m 45s\tremaining: 43m 27s\n",
      "1100:\ttest: 0.7295753\ttest1: 0.6855847\tbest: 0.6856005 (1099)\ttotal: 23m 59s\tremaining: 41m 23s\n",
      "1200:\ttest: 0.7310136\ttest1: 0.6855416\tbest: 0.6856285 (1119)\ttotal: 26m 13s\tremaining: 39m 16s\n",
      "Stopped by overfitting detector  (100 iterations wait)\n",
      "\n",
      "bestTest = 0.6856284522\n",
      "bestIteration = 1119\n",
      "\n",
      "Shrink model to first 1120 iterations.\n",
      "err:  0.6856284521994805\n"
     ]
    }
   ],
   "source": [
    "# from sklearn.metrics import accuracy_score,confusion_matrix,roc_auc_score\n",
    "# errcb=[]\n",
    "# y_pred_totcb=[]\n",
    "# from sklearn.model_selection import KFold,StratifiedKFold\n",
    "# fold=StratifiedKFold(n_splits=10,shuffle=True,random_state=1994)\n",
    "# i=1\n",
    "# for train_index, test_index in fold.split(X,y):\n",
    "#     X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n",
    "#     y_train, y_test = y[train_index], y[test_index]\n",
    "#     m=CatBoostClassifier(n_estimators=3000,random_state=1994,eval_metric='AUC',max_depth=6,learning_rate=0.05,od_wait=50\n",
    "#                      ,l2_leaf_reg=10,cat_features=categorical_features_indices,bagging_temperature=1,random_strength=100,\n",
    "#                      use_best_model=True)\n",
    "#     m.fit(X_train,y_train,eval_set=[(X_train,y_train),(X_test, y_test)], early_stopping_rounds=100,verbose=100)\n",
    "#     preds=m.predict_proba(X_test)[:,-1]\n",
    "#     print(\"err: \",roc_auc_score(y_test,preds))\n",
    "#     errcb.append(roc_auc_score(y_test,preds))\n",
    "#     p = m.predict_proba(dftest)[:,-1]\n",
    "#     y_pred_totcb.append(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.682666383500668"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# normal 2 splits\n",
    "np.mean(errcb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.682823114704624"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 10 -folds cross validated with tuning\n",
    "np.mean(errcb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6834699720831957"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 10 folds CV with more tuning\n",
    "np.mean(errcb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6836636808938612"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# final 20 folds CV with tuning\n",
    "np.mean(errcb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "s=pd.DataFrame({'UniqueID':test['UniqueID'],'loan_default':np.mean(y_pred_totcb,0)})\n",
    "s.head()\n",
    "\n",
    "\n",
    "s.to_csv('cb20foldscatMostfeat22Opt_U2.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "* Catboost is best when you have Categorical data as well as numerical, also many hyperparameters to play with. \n",
    "* To remove Overfitting, on must check their local score, difference of score in validation set.\n",
    "* K-Folds cross validation helps curbs overfitting\n",
    "\n",
    "\n",
    "## Thanks AV for the Hackathon\n",
    "\n",
    "# Public LB - 17th Rank - 0.6662550192\n",
    "# Private LB - 13th Rank - 0.6703492666"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}