Kaggle Days Paris - Skopt + CatBoost solution.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://storage.googleapis.com/kaggle-competitions/kaggle/3338/media/gate.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Amazon.com - Employee Access Challenge\n",
    "\n",
    "When an employee at any company starts work, they first need to obtain the computer access necessary to fulfill their role. This access may allow an employee to read/manipulate resources through various applications or web portals. It is assumed that employees fulfilling the functions of a given role will access the same or similar resources. It is often the case that employees figure out the access they need as they encounter roadblocks during their daily work (e.g. not able to log into a reporting portal). A knowledgeable supervisor then takes time to manually grant the needed access in order to overcome access obstacles. As employees move throughout a company, this access discovery/recovery cycle wastes a nontrivial amount of time and money.\n",
    "\n",
    "There is a considerable amount of data regarding an employee’s role within an organization and the resources to which they have access. Given the data related to current employees and their provisioned access, models can be built that automatically determine access privileges as employees enter and leave roles within a company. These auto-access models seek to minimize the human involvement required to grant or revoke employee access."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting git+https://github.com/scikit-optimize/scikit-optimize.git\n",
      "  Cloning https://github.com/scikit-optimize/scikit-optimize.git to c:\\users\\utente\\appdata\\local\\temp\\pip-req-build-7l750zmj\n",
      "Requirement already satisfied (use --upgrade to upgrade): scikit-optimize==0.5.2+48.g692c21d from git+https://github.com/scikit-optimize/scikit-optimize.git in c:\\anaconda3\\lib\\site-packages\n",
      "Requirement already satisfied: pyaml in c:\\anaconda3\\lib\\site-packages (from scikit-optimize==0.5.2+48.g692c21d) (18.11.0)\n",
      "Requirement already satisfied: numpy in c:\\anaconda3\\lib\\site-packages (from scikit-optimize==0.5.2+48.g692c21d) (1.15.4)\n",
      "Requirement already satisfied: scipy>=0.14.0 in c:\\anaconda3\\lib\\site-packages (from scikit-optimize==0.5.2+48.g692c21d) (1.1.0)\n",
      "Requirement already satisfied: scikit-learn>=0.19.1 in c:\\anaconda3\\lib\\site-packages (from scikit-optimize==0.5.2+48.g692c21d) (0.20.2)\n",
      "Requirement already satisfied: PyYAML in c:\\anaconda3\\lib\\site-packages (from pyaml->scikit-optimize==0.5.2+48.g692c21d) (3.12)\n",
      "Building wheels for collected packages: scikit-optimize\n",
      "  Running setup.py bdist_wheel for scikit-optimize: started\n",
      "  Running setup.py bdist_wheel for scikit-optimize: finished with status 'done'\n",
      "  Stored in directory: C:\\Users\\utente\\AppData\\Local\\Temp\\pip-ephem-wheel-cache-65vkmfmz\\wheels\\11\\6f\\86\\2b772172db85ad0b4487d67e325e535ee8e7782b2a1dfcadf5\n",
      "Successfully built scikit-optimize\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using pip version 18.0, however version 19.0.1 is available.\n",
      "You should consider upgrading via the 'python -m pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "# Installing the most recent version of skopt directly from Github\n",
    "!pip install git+https://github.com/scikit-optimize/scikit-optimize.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already up-to-date: catboost in c:\\anaconda3\\lib\\site-packages (0.12.2)\n",
      "Requirement already satisfied, skipping upgrade: pandas>=0.19.1 in c:\\anaconda3\\lib\\site-packages (from catboost) (0.20.3)\n",
      "Requirement already satisfied, skipping upgrade: numpy>=1.11.1 in c:\\anaconda3\\lib\\site-packages (from catboost) (1.15.4)\n",
      "Requirement already satisfied, skipping upgrade: six in c:\\anaconda3\\lib\\site-packages (from catboost) (1.11.0)\n",
      "Requirement already satisfied, skipping upgrade: enum34 in c:\\anaconda3\\lib\\site-packages (from catboost) (1.1.6)\n",
      "Requirement already satisfied, skipping upgrade: python-dateutil>=2 in c:\\anaconda3\\lib\\site-packages (from pandas>=0.19.1->catboost) (2.6.1)\n",
      "Requirement already satisfied, skipping upgrade: pytz>=2011k in c:\\anaconda3\\lib\\site-packages (from pandas>=0.19.1->catboost) (2017.2)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using pip version 18.0, however version 19.0.1 is available.\n",
      "You should consider upgrading via the 'python -m pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "# Assuring you have the most recent CatBoost release\n",
    "!pip install catboost -U"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "_uuid": "e66da8605d755b1e2ade0f6484938125f91f9687",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Importing core libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from time import time\n",
    "import pprint\n",
    "import joblib\n",
    "\n",
    "# Suppressing warnings because of skopt verbosity\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "# Classifiers\n",
    "from catboost import CatBoostClassifier\n",
    "\n",
    "# Model selection\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "# Metrics\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import make_scorer\n",
    "\n",
    "# Skopt functions\n",
    "from skopt import BayesSearchCV\n",
    "from skopt.callbacks import DeadlineStopper, VerboseCallback, DeltaXStopper\n",
    "from skopt.space import Real, Categorical, Integer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Loading data directly from CatBoost\n",
    "from catboost.datasets import amazon\n",
    "\n",
    "X, Xt = amazon()\n",
    "\n",
    "y = X[\"ACTION\"].apply(lambda x: 1 if x == 1 else 0)\n",
    "X.drop([\"ACTION\"], axis=1, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "_uuid": "bf2c611ba1115560a5961c03269b5c354740a6b9"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>RESOURCE</th>\n",
       "      <th>MGR_ID</th>\n",
       "      <th>ROLE_ROLLUP_1</th>\n",
       "      <th>ROLE_ROLLUP_2</th>\n",
       "      <th>ROLE_DEPTNAME</th>\n",
       "      <th>ROLE_TITLE</th>\n",
       "      <th>ROLE_FAMILY_DESC</th>\n",
       "      <th>ROLE_FAMILY</th>\n",
       "      <th>ROLE_CODE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>39353</td>\n",
       "      <td>85475</td>\n",
       "      <td>117961</td>\n",
       "      <td>118300</td>\n",
       "      <td>123472</td>\n",
       "      <td>117905</td>\n",
       "      <td>117906</td>\n",
       "      <td>290919</td>\n",
       "      <td>117908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>17183</td>\n",
       "      <td>1540</td>\n",
       "      <td>117961</td>\n",
       "      <td>118343</td>\n",
       "      <td>123125</td>\n",
       "      <td>118536</td>\n",
       "      <td>118536</td>\n",
       "      <td>308574</td>\n",
       "      <td>118539</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>36724</td>\n",
       "      <td>14457</td>\n",
       "      <td>118219</td>\n",
       "      <td>118220</td>\n",
       "      <td>117884</td>\n",
       "      <td>117879</td>\n",
       "      <td>267952</td>\n",
       "      <td>19721</td>\n",
       "      <td>117880</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>36135</td>\n",
       "      <td>5396</td>\n",
       "      <td>117961</td>\n",
       "      <td>118343</td>\n",
       "      <td>119993</td>\n",
       "      <td>118321</td>\n",
       "      <td>240983</td>\n",
       "      <td>290919</td>\n",
       "      <td>118322</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>42680</td>\n",
       "      <td>5905</td>\n",
       "      <td>117929</td>\n",
       "      <td>117930</td>\n",
       "      <td>119569</td>\n",
       "      <td>119323</td>\n",
       "      <td>123932</td>\n",
       "      <td>19793</td>\n",
       "      <td>119325</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   RESOURCE  MGR_ID  ROLE_ROLLUP_1  ROLE_ROLLUP_2  ROLE_DEPTNAME  ROLE_TITLE  \\\n",
       "0     39353   85475         117961         118300         123472      117905   \n",
       "1     17183    1540         117961         118343         123125      118536   \n",
       "2     36724   14457         118219         118220         117884      117879   \n",
       "3     36135    5396         117961         118343         119993      118321   \n",
       "4     42680    5905         117929         117930         119569      119323   \n",
       "\n",
       "   ROLE_FAMILY_DESC  ROLE_FAMILY  ROLE_CODE  \n",
       "0            117906       290919     117908  \n",
       "1            118536       308574     118539  \n",
       "2            267952        19721     117880  \n",
       "3            240983       290919     118322  \n",
       "4            123932        19793     119325  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "_uuid": "4fc60f85cd4004cd96556a44f91e0abc8fbba69b"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>RESOURCE</th>\n",
       "      <th>MGR_ID</th>\n",
       "      <th>ROLE_ROLLUP_1</th>\n",
       "      <th>ROLE_ROLLUP_2</th>\n",
       "      <th>ROLE_DEPTNAME</th>\n",
       "      <th>ROLE_TITLE</th>\n",
       "      <th>ROLE_FAMILY_DESC</th>\n",
       "      <th>ROLE_FAMILY</th>\n",
       "      <th>ROLE_CODE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>78766</td>\n",
       "      <td>72734</td>\n",
       "      <td>118079</td>\n",
       "      <td>118080</td>\n",
       "      <td>117878</td>\n",
       "      <td>117879</td>\n",
       "      <td>118177</td>\n",
       "      <td>19721</td>\n",
       "      <td>117880</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>40644</td>\n",
       "      <td>4378</td>\n",
       "      <td>117961</td>\n",
       "      <td>118327</td>\n",
       "      <td>118507</td>\n",
       "      <td>118863</td>\n",
       "      <td>122008</td>\n",
       "      <td>118398</td>\n",
       "      <td>118865</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>75443</td>\n",
       "      <td>2395</td>\n",
       "      <td>117961</td>\n",
       "      <td>118300</td>\n",
       "      <td>119488</td>\n",
       "      <td>118172</td>\n",
       "      <td>301534</td>\n",
       "      <td>249618</td>\n",
       "      <td>118175</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>43219</td>\n",
       "      <td>19986</td>\n",
       "      <td>117961</td>\n",
       "      <td>118225</td>\n",
       "      <td>118403</td>\n",
       "      <td>120773</td>\n",
       "      <td>136187</td>\n",
       "      <td>118960</td>\n",
       "      <td>120774</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>42093</td>\n",
       "      <td>50015</td>\n",
       "      <td>117961</td>\n",
       "      <td>118343</td>\n",
       "      <td>119598</td>\n",
       "      <td>118422</td>\n",
       "      <td>300136</td>\n",
       "      <td>118424</td>\n",
       "      <td>118425</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  RESOURCE  MGR_ID  ROLE_ROLLUP_1  ROLE_ROLLUP_2  ROLE_DEPTNAME  \\\n",
       "0   1     78766   72734         118079         118080         117878   \n",
       "1   2     40644    4378         117961         118327         118507   \n",
       "2   3     75443    2395         117961         118300         119488   \n",
       "3   4     43219   19986         117961         118225         118403   \n",
       "4   5     42093   50015         117961         118343         119598   \n",
       "\n",
       "   ROLE_TITLE  ROLE_FAMILY_DESC  ROLE_FAMILY  ROLE_CODE  \n",
       "0      117879            118177        19721     117880  \n",
       "1      118863            122008       118398     118865  \n",
       "2      118172            301534       249618     118175  \n",
       "3      120773            136187       118960     120774  \n",
       "4      118422            300136       118424     118425  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Xt.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "_uuid": "ed51e2ec1496e715163aaef18f62eee327156368",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Reporting util for different optimizers\n",
    "def report_perf(optimizer, X, y, title, callbacks=None):\n",
    "    \"\"\"\n",
    "    A wrapper for measuring time and performances of different optmizers\n",
    "    \n",
    "    optimizer = a sklearn or a skopt optimizer\n",
    "    X = the training set \n",
    "    y = our target\n",
    "    title = a string label for the experiment\n",
    "    \"\"\"\n",
    "    start = time()\n",
    "    if callbacks:\n",
    "        optimizer.fit(X, y, callback=callbacks)\n",
    "    else:\n",
    "        optimizer.fit(X, y)\n",
    "    d=pd.DataFrame(optimizer.cv_results_)\n",
    "    best_score = optimizer.best_score_\n",
    "    best_score_std = d.iloc[optimizer.best_index_].std_test_score\n",
    "    best_params = optimizer.best_params_\n",
    "    print((title + \" took %.2f seconds,  candidates checked: %d, best CV score: %.3f \"\n",
    "           +u\"\\u00B1\"+\" %.3f\") % (time() - start, \n",
    "                                  len(optimizer.cv_results_['params']),\n",
    "                                  best_score,\n",
    "                                  best_score_std))    \n",
    "    print('Best parameters:')\n",
    "    pprint.pprint(best_params)\n",
    "    print()\n",
    "    return best_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Converting average precision score into a scorer suitable for model selection\n",
    "roc_auc = make_scorer(roc_auc_score, greater_is_better=True, needs_threshold=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Setting a 5-fold stratified cross-validation (note: shuffle=True)\n",
    "skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "_uuid": "df96bd6f47aa2ca5fa31f9a6a3a223b5ea6ab4b2",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Initializing a CatBoostClassifier\n",
    "clf = CatBoostClassifier(thread_count=2,\n",
    "                         loss_function='Logloss',\n",
    "                         verbose = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "_uuid": "5b09cf2accbc06a1be031393fde0e683a52e56d3",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Defining your search space\n",
    "search_spaces = {'iterations': Integer(10, 300),\n",
    "                 'depth': Integer(1, 8),\n",
    "                 'learning_rate': Real(0.01, 1.0, 'log-uniform'),\n",
    "                 'random_strength': Real(1e-9, 10, 'log-uniform'),\n",
    "                 'bagging_temperature': Real(0.0, 1.0),\n",
    "                 'border_count': Integer(1, 255),\n",
    "                 'l2_leaf_reg': Integer(2, 30),\n",
    "                 'scale_pos_weight':Real(0.01, 1.0, 'uniform')}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Setting up BayesSearchCV\n",
    "opt = BayesSearchCV(clf,\n",
    "                    search_spaces,\n",
    "                    scoring=roc_auc,\n",
    "                    cv=skf,\n",
    "                    n_iter=100,\n",
    "                    n_jobs=1,  # use just 1 job with CatBoost in order to avoid segmentation fault\n",
    "                    return_train_score=False,\n",
    "                    refit=True,\n",
    "                    optimizer_kwargs={'base_estimator': 'GP'},\n",
    "                    random_state=42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "_uuid": "e964985474c381e059050564f81ae3f675ba28aa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration No: 1 started. Searching for the next optimal point.\n",
      "Iteration No: 1 ended. Search finished for the next optimal point.\n",
      "Time taken: 146.4917\n",
      "Function value obtained: -0.7276\n",
      "Current minimum: -0.7276\n",
      "Iteration No: 2 started. Searching for the next optimal point.\n",
      "Iteration No: 2 ended. Search finished for the next optimal point.\n",
      "Time taken: 684.3241\n",
      "Function value obtained: -0.7884\n",
      "Current minimum: -0.7884\n",
      "Iteration No: 3 started. Searching for the next optimal point.\n",
      "CatBoost took 967.68 seconds,  candidates checked: 2, best CV score: 0.788 ± 0.018\n",
      "Best parameters:\n",
      "{'bagging_temperature': 0.8373883555532844,\n",
      " 'border_count': 225,\n",
      " 'ctr_border_count': 78,\n",
      " 'depth': 8,\n",
      " 'iterations': 261,\n",
      " 'l2_leaf_reg': 4,\n",
      " 'learning_rate': 0.018906758484967926,\n",
      " 'random_strength': 3.43458268604567e-06,\n",
      " 'scale_pos_weight': 0.6393718108603786}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Running the optimization\n",
    "best_params = report_perf(opt, X, y,'CatBoost', \n",
    "                          callbacks=[VerboseCallback(100), \n",
    "                                     DeadlineStopper(60*10)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "_uuid": "f7809e4678d1932f1da4f51766145c7ceb8d4acc",
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Using optimized BayesSearchCV for predictions\n",
    "submission = pd.DataFrame(Xt.id)\n",
    "submission['Action'] = opt.predict_proba(Xt)[:,1]\n",
    "submission.to_csv(\"amazon_submission.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# {'bagging_temperature': 1.0,\n",
    "#  'border_count': 109,\n",
    "#  'depth': 8,\n",
    "#  'iterations': 239,\n",
    "#  'l2_leaf_reg': 30,\n",
    "#  'learning_rate': 0.10388349218286635,\n",
    "#  'random_strength': 1e-09,\n",
    "#  'scale_pos_weight': 0.07517702003645463}\n",
    "\n",
    "# --> https://www.kaggle.com/c/amazon-employee-access-challenge/submit\n",
    "# Public LB : 0.83130\n",
    "# Private LB : 0.84270"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}