3. Logistic Regression/4. Logistic Regression with Statsmodels.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Logistic Regression**: HR Dataset 🕵️‍♀️"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the dataset of a large company, which has around 4000 employees. Every year, around 15% of its employees leave the company and need to be replaced with the talent pool available in the job market. This level of attrition (employees leaving, either on their own or because they got fired) is bad for the company, because of the following reasons:\n",
    "* the former employees’ projects get delayed, which makes it difficult to meet timelines, resulting in a reputation loss among consumers and partners\n",
    "* a sizeable department has to be maintained, for the purposes of recruiting new talent\n",
    "* more often than not, the new employees have to be trained for the job and/or given time to acclimatise themselves to the company\n",
    "\n",
    "Hence, it is important to understand what factors cause attrition. In other words, it is important to undersand what needs to be changed in order to get most of the employees to stay. Therefore, the goal of the study is to model the probability of attrition based on certain features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Logistic Regression with Statsmodels**\n",
    "\n",
    "$\\qquad$ <span style=\"color:gray\"><b>0.</b> Settings </span><br>\n",
    "$\\qquad$ <span style=\"color:gray\"><b>1.</b> Dataset </span><br>\n",
    "$\\qquad$ <span style=\"color:gray\"><b>2.</b> Data Preprocessing </span><br>\n",
    "$\\qquad$ <span style=\"color:gray\"><b>3.</b> Data Preparation </span><br>\n",
    "$\\qquad$ <span style=\"color:gray\"><b>4.</b> Logistic Regression with Statsmodels </span><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **0.** Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Libraries\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "import statsmodels.api as sm\n",
    "import pandas as pd\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **1.** Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "    DATASET INFORMATIONS\n",
    "\n",
    "    |--------------------------|------------|-------------------------------------------------------------------------------|\n",
    "    | Name                     | Data Type  | Description                                                                   |\n",
    "    |--------------------------|------------|-------------------------------------------------------------------------------|\n",
    "    | Age                      | continuous | Age of the employee                                                           |\n",
    "    | Attrition                | nominal    | Whether the employee left in the previous year or not                         |\n",
    "    | BusinessTravel           | nominal    | How frequently the employees travelled for business purposes in the last year |\n",
    "    | Department               | nominal    | Department in company                                                         |\n",
    "    | DistanceFromHome         | continuous | Distance from home in kms                                                     |\n",
    "    | Education                | continuous | Education Level                                                               |\n",
    "    | EducationField           | nominal    | Field of education                                                            |\n",
    "    | EmployeeCount            | continuous | Employee count                                                                |\n",
    "    | EmployeeID               | nominal    | Employee number/id                                                            |\n",
    "    | Gender                   | nominal    | Gender of employee                                                            |\n",
    "    | JobLevel                 | continuous | Job level at company on a scale of 1 to 5                                     |\n",
    "    | JobRole                  | nominal    | Name of job role in company                                                   |\n",
    "    | MaritalStatus            | nominal    | Marital status of the employee                                                |\n",
    "    | MonthlyIncome            | continuous | Monthly income in rupees per month                                            |\n",
    "    | NumCompaniesWorked       | continuous | Total number of companies the employee has worked for                         |\n",
    "    | Over18                   | nominal    | Whether the employee is above 18 years of age or not                          |\n",
    "    | PercentSalaryHike        | continuous | Percent salary hike for last year                                             |\n",
    "    | PerformanceRating\t       | continuous | Performance rating for last year                                              |\n",
    "    | RelationshipSatisfaction | continuous | Relationship satisfaction level                                               |\n",
    "    | StandardHours            | continuous | Standard hours of work for the employee                                       |\n",
    "    | StockOptionLevel         | continuous | Stock option level of the employee                                            |\n",
    "    | TotalWorkingYears        | continuous | Total number of years the employee has worked so far                          |\n",
    "    | TrainingTimesLastYear    | continuous | Number of times training was conducted for this employee last year            |\n",
    "    | WorkLifeBalance          | continuous | Work life balance level                                                       |\n",
    "    | YearsAtCompany           | continuous | Total number of years spent at the company by the employee                    |\n",
    "    | YearsSinceLastPromotion  | continuous | Number of years since last promotion                                          |\n",
    "    | YearsWithCurrManager     | continuous | Number of years under current manager                                         |\n",
    "    |--------------------------|------------|-------------------------------------------------------------------------------|\n",
    "\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Age</th>\n",
       "      <td>51</td>\n",
       "      <td>31</td>\n",
       "      <td>32</td>\n",
       "      <td>38</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Attrition</th>\n",
       "      <td>No</td>\n",
       "      <td>Yes</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "      <td>No</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BusinessTravel</th>\n",
       "      <td>Travel_Rarely</td>\n",
       "      <td>Travel_Frequently</td>\n",
       "      <td>Travel_Frequently</td>\n",
       "      <td>Non-Travel</td>\n",
       "      <td>Travel_Rarely</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Department</th>\n",
       "      <td>Sales</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "      <td>Research &amp; Development</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DistanceFromHome</th>\n",
       "      <td>6</td>\n",
       "      <td>10</td>\n",
       "      <td>17</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Education</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EducationField</th>\n",
       "      <td>Life Sciences</td>\n",
       "      <td>Life Sciences</td>\n",
       "      <td>Other</td>\n",
       "      <td>Life Sciences</td>\n",
       "      <td>Medical</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EmployeeCount</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EmployeeID</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Gender</th>\n",
       "      <td>Female</td>\n",
       "      <td>Female</td>\n",
       "      <td>Male</td>\n",
       "      <td>Male</td>\n",
       "      <td>Male</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>JobLevel</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>JobRole</th>\n",
       "      <td>Healthcare Representative</td>\n",
       "      <td>Research Scientist</td>\n",
       "      <td>Sales Executive</td>\n",
       "      <td>Human Resources</td>\n",
       "      <td>Sales Executive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MaritalStatus</th>\n",
       "      <td>Married</td>\n",
       "      <td>Single</td>\n",
       "      <td>Married</td>\n",
       "      <td>Married</td>\n",
       "      <td>Single</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MonthlyIncome</th>\n",
       "      <td>131160</td>\n",
       "      <td>41890</td>\n",
       "      <td>193280</td>\n",
       "      <td>83210</td>\n",
       "      <td>23420</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NumCompaniesWorked</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Over18</th>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "      <td>Y</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PercentSalaryHike</th>\n",
       "      <td>11</td>\n",
       "      <td>23</td>\n",
       "      <td>15</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>StandardHours</th>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>StockOptionLevel</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TotalWorkingYears</th>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TrainingTimesLastYear</th>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>YearsAtCompany</th>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>5</td>\n",
       "      <td>8</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>YearsSinceLastPromotion</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>YearsWithCurrManager</th>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 0                       1  \\\n",
       "Age                                             51                      31   \n",
       "Attrition                                       No                     Yes   \n",
       "BusinessTravel                       Travel_Rarely       Travel_Frequently   \n",
       "Department                                   Sales  Research & Development   \n",
       "DistanceFromHome                                 6                      10   \n",
       "Education                                        2                       1   \n",
       "EducationField                       Life Sciences           Life Sciences   \n",
       "EmployeeCount                                    1                       1   \n",
       "EmployeeID                                       1                       2   \n",
       "Gender                                      Female                  Female   \n",
       "JobLevel                                         1                       1   \n",
       "JobRole                  Healthcare Representative      Research Scientist   \n",
       "MaritalStatus                              Married                  Single   \n",
       "MonthlyIncome                               131160                   41890   \n",
       "NumCompaniesWorked                             1.0                     0.0   \n",
       "Over18                                           Y                       Y   \n",
       "PercentSalaryHike                               11                      23   \n",
       "StandardHours                                    8                       8   \n",
       "StockOptionLevel                                 0                       1   \n",
       "TotalWorkingYears                              1.0                     6.0   \n",
       "TrainingTimesLastYear                            6                       3   \n",
       "YearsAtCompany                                   1                       5   \n",
       "YearsSinceLastPromotion                          0                       1   \n",
       "YearsWithCurrManager                             0                       4   \n",
       "\n",
       "                                              2                       3  \\\n",
       "Age                                          32                      38   \n",
       "Attrition                                    No                      No   \n",
       "BusinessTravel                Travel_Frequently              Non-Travel   \n",
       "Department               Research & Development  Research & Development   \n",
       "DistanceFromHome                             17                       2   \n",
       "Education                                     4                       5   \n",
       "EducationField                            Other           Life Sciences   \n",
       "EmployeeCount                                 1                       1   \n",
       "EmployeeID                                    3                       4   \n",
       "Gender                                     Male                    Male   \n",
       "JobLevel                                      4                       3   \n",
       "JobRole                         Sales Executive         Human Resources   \n",
       "MaritalStatus                           Married                 Married   \n",
       "MonthlyIncome                            193280                   83210   \n",
       "NumCompaniesWorked                          1.0                     3.0   \n",
       "Over18                                        Y                       Y   \n",
       "PercentSalaryHike                            15                      11   \n",
       "StandardHours                                 8                       8   \n",
       "StockOptionLevel                              3                       3   \n",
       "TotalWorkingYears                           5.0                    13.0   \n",
       "TrainingTimesLastYear                         2                       5   \n",
       "YearsAtCompany                                5                       8   \n",
       "YearsSinceLastPromotion                       0                       7   \n",
       "YearsWithCurrManager                          3                       5   \n",
       "\n",
       "                                              4  \n",
       "Age                                          32  \n",
       "Attrition                                    No  \n",
       "BusinessTravel                    Travel_Rarely  \n",
       "Department               Research & Development  \n",
       "DistanceFromHome                             10  \n",
       "Education                                     1  \n",
       "EducationField                          Medical  \n",
       "EmployeeCount                                 1  \n",
       "EmployeeID                                    5  \n",
       "Gender                                     Male  \n",
       "JobLevel                                      1  \n",
       "JobRole                         Sales Executive  \n",
       "MaritalStatus                            Single  \n",
       "MonthlyIncome                             23420  \n",
       "NumCompaniesWorked                          4.0  \n",
       "Over18                                        Y  \n",
       "PercentSalaryHike                            12  \n",
       "StandardHours                                 8  \n",
       "StockOptionLevel                              2  \n",
       "TotalWorkingYears                           9.0  \n",
       "TrainingTimesLastYear                         2  \n",
       "YearsAtCompany                                6  \n",
       "YearsSinceLastPromotion                       0  \n",
       "YearsWithCurrManager                          4  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Import the dataset\n",
    "data = pd.read_csv('./dataset.csv')\n",
    "data.head().T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Age', 'Attrition', 'BusinessTravel', 'Department', 'DistanceFromHome',\n",
       "       'Education', 'EducationField', 'EmployeeCount', 'EmployeeID', 'Gender',\n",
       "       'JobLevel', 'JobRole', 'MaritalStatus', 'MonthlyIncome',\n",
       "       'NumCompaniesWorked', 'Over18', 'PercentSalaryHike', 'StandardHours',\n",
       "       'StockOptionLevel', 'TotalWorkingYears', 'TrainingTimesLastYear',\n",
       "       'YearsAtCompany', 'YearsSinceLastPromotion', 'YearsWithCurrManager'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 4410 entries, 0 to 4409\n",
      "Data columns (total 24 columns):\n",
      " #   Column                   Non-Null Count  Dtype  \n",
      "---  ------                   --------------  -----  \n",
      " 0   Age                      4410 non-null   int64  \n",
      " 1   Attrition                4410 non-null   object \n",
      " 2   BusinessTravel           4410 non-null   object \n",
      " 3   Department               4410 non-null   object \n",
      " 4   DistanceFromHome         4410 non-null   int64  \n",
      " 5   Education                4410 non-null   int64  \n",
      " 6   EducationField           4410 non-null   object \n",
      " 7   EmployeeCount            4410 non-null   int64  \n",
      " 8   EmployeeID               4410 non-null   int64  \n",
      " 9   Gender                   4410 non-null   object \n",
      " 10  JobLevel                 4410 non-null   int64  \n",
      " 11  JobRole                  4410 non-null   object \n",
      " 12  MaritalStatus            4410 non-null   object \n",
      " 13  MonthlyIncome            4410 non-null   int64  \n",
      " 14  NumCompaniesWorked       4391 non-null   float64\n",
      " 15  Over18                   4410 non-null   object \n",
      " 16  PercentSalaryHike        4410 non-null   int64  \n",
      " 17  StandardHours            4410 non-null   int64  \n",
      " 18  StockOptionLevel         4410 non-null   int64  \n",
      " 19  TotalWorkingYears        4401 non-null   float64\n",
      " 20  TrainingTimesLastYear    4410 non-null   int64  \n",
      " 21  YearsAtCompany           4410 non-null   int64  \n",
      " 22  YearsSinceLastPromotion  4410 non-null   int64  \n",
      " 23  YearsWithCurrManager     4410 non-null   int64  \n",
      "dtypes: float64(2), int64(14), object(8)\n",
      "memory usage: 827.0+ KB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **2.** Data Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Age                         0\n",
       "Attrition                   0\n",
       "BusinessTravel              0\n",
       "Department                  0\n",
       "DistanceFromHome            0\n",
       "Education                   0\n",
       "EducationField              0\n",
       "EmployeeCount               0\n",
       "EmployeeID                  0\n",
       "Gender                      0\n",
       "JobLevel                    0\n",
       "JobRole                     0\n",
       "MaritalStatus               0\n",
       "MonthlyIncome               0\n",
       "NumCompaniesWorked         19\n",
       "Over18                      0\n",
       "PercentSalaryHike           0\n",
       "StandardHours               0\n",
       "StockOptionLevel            0\n",
       "TotalWorkingYears           9\n",
       "TrainingTimesLastYear       0\n",
       "YearsAtCompany              0\n",
       "YearsSinceLastPromotion     0\n",
       "YearsWithCurrManager        0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Null elements\n",
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Age                        False\n",
       "Attrition                  False\n",
       "BusinessTravel             False\n",
       "Department                 False\n",
       "DistanceFromHome           False\n",
       "Education                  False\n",
       "EducationField             False\n",
       "EmployeeCount              False\n",
       "EmployeeID                 False\n",
       "Gender                     False\n",
       "JobLevel                   False\n",
       "JobRole                    False\n",
       "MaritalStatus              False\n",
       "MonthlyIncome              False\n",
       "NumCompaniesWorked          True\n",
       "Over18                     False\n",
       "PercentSalaryHike          False\n",
       "StandardHours              False\n",
       "StockOptionLevel           False\n",
       "TotalWorkingYears           True\n",
       "TrainingTimesLastYear      False\n",
       "YearsAtCompany             False\n",
       "YearsSinceLastPromotion    False\n",
       "YearsWithCurrManager       False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull().any()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4410, 24)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4382, 24)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can either fill the nulls with 0 or delete the rows. \n",
    "# In this case we can try to delete the rows with missing \n",
    "# elements since we have quite a lot of data.\n",
    "\n",
    "# Fill\n",
    "# data.fillna(0, inplace =True)\n",
    "\n",
    "# Drop\n",
    "data = data.dropna(how='any', axis=0)\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4382, 20)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We remove unnecessary information\n",
    "data.drop(['EmployeeCount','EmployeeID','StandardHours', 'Over18'], axis=1, inplace=True)\n",
    "\n",
    "#   * 'EmployeeCount' is always equal to 1 (as employees are interviewed one at a time)\n",
    "#   * 'EmployeeID' are unique values \n",
    "#   * 'StandardHours' is (basically) always 8\n",
    "#   * 'Over18' because all the employees are >18\n",
    "\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **3.** Data Preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From previous analyses (EDA) we certainly want to take into account (since they were visually influential):\n",
    "\n",
    "* Business Travel\n",
    "* Department\n",
    "* Education Field\n",
    "* Marital Status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Age', 'Attrition', 'BusinessTravel', 'Department', 'DistanceFromHome',\n",
       "       'Education', 'EducationField', 'Gender', 'JobLevel', 'JobRole',\n",
       "       'MaritalStatus', 'MonthlyIncome', 'NumCompaniesWorked',\n",
       "       'PercentSalaryHike', 'StockOptionLevel', 'TotalWorkingYears',\n",
       "       'TrainingTimesLastYear', 'YearsAtCompany', 'YearsSinceLastPromotion',\n",
       "       'YearsWithCurrManager'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Age                                               51\n",
       "Attrition                                         No\n",
       "BusinessTravel                         Travel_Rarely\n",
       "Department                                     Sales\n",
       "DistanceFromHome                                   6\n",
       "Education                                          2\n",
       "EducationField                         Life Sciences\n",
       "Gender                                        Female\n",
       "JobLevel                                           1\n",
       "JobRole                    Healthcare Representative\n",
       "MaritalStatus                                Married\n",
       "MonthlyIncome                                 131160\n",
       "NumCompaniesWorked                               1.0\n",
       "PercentSalaryHike                                 11\n",
       "StockOptionLevel                                   0\n",
       "TotalWorkingYears                                1.0\n",
       "TrainingTimesLastYear                              6\n",
       "YearsAtCompany                                     1\n",
       "YearsSinceLastPromotion                            0\n",
       "YearsWithCurrManager                               0\n",
       "Name: 0, dtype: object"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Travel_Rarely' 'Travel_Frequently' 'Non-Travel']\n",
      "['Sales' 'Research & Development' 'Human Resources']\n",
      "['Life Sciences' 'Other' 'Medical' 'Marketing' 'Technical Degree'\n",
      " 'Human Resources']\n",
      "['Female' 'Male']\n",
      "['Healthcare Representative' 'Research Scientist' 'Sales Executive'\n",
      " 'Human Resources' 'Research Director' 'Laboratory Technician'\n",
      " 'Manufacturing Director' 'Sales Representative' 'Manager']\n",
      "['Married' 'Single' 'Divorced']\n"
     ]
    }
   ],
   "source": [
    "# Convert all the Categorical data into numerical data\n",
    "print(data['BusinessTravel'].unique())\n",
    "print(data['Department'].unique())\n",
    "print(data['EducationField'].unique())\n",
    "print(data['Gender'].unique())\n",
    "print(data['JobRole'].unique())\n",
    "print(data['MaritalStatus'].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encode categorical features\n",
    "labelEncoder_X = LabelEncoder()\n",
    "\n",
    "data['BusinessTravel'] = labelEncoder_X.fit_transform(data['BusinessTravel'])\n",
    "data['Department']     = labelEncoder_X.fit_transform(data['Department'])\n",
    "data['EducationField'] = labelEncoder_X.fit_transform(data['EducationField'])\n",
    "data['Gender']         = labelEncoder_X.fit_transform(data['Gender'])\n",
    "data['JobRole']        = labelEncoder_X.fit_transform(data['JobRole'])\n",
    "data['MaritalStatus']  = labelEncoder_X.fit_transform(data['MaritalStatus'])\n",
    "\n",
    "# Encode label\n",
    "label_encoder_y = LabelEncoder()\n",
    "\n",
    "data['Attrition'] = label_encoder_y.fit_transform(data['Attrition'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This encoding transforms categorical variables into numerical values, it is not one-hot-encoding.<br>\n",
    "We proceed in this way in order to visualize the correlation of the categorical variables with the target (also categorical)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 4382 entries, 0 to 4408\n",
      "Data columns (total 20 columns):\n",
      " #   Column                   Non-Null Count  Dtype  \n",
      "---  ------                   --------------  -----  \n",
      " 0   Age                      4382 non-null   int64  \n",
      " 1   Attrition                4382 non-null   int32  \n",
      " 2   BusinessTravel           4382 non-null   int32  \n",
      " 3   Department               4382 non-null   int32  \n",
      " 4   DistanceFromHome         4382 non-null   int64  \n",
      " 5   Education                4382 non-null   int64  \n",
      " 6   EducationField           4382 non-null   int32  \n",
      " 7   Gender                   4382 non-null   int32  \n",
      " 8   JobLevel                 4382 non-null   int64  \n",
      " 9   JobRole                  4382 non-null   int32  \n",
      " 10  MaritalStatus            4382 non-null   int32  \n",
      " 11  MonthlyIncome            4382 non-null   int64  \n",
      " 12  NumCompaniesWorked       4382 non-null   float64\n",
      " 13  PercentSalaryHike        4382 non-null   int64  \n",
      " 14  StockOptionLevel         4382 non-null   int64  \n",
      " 15  TotalWorkingYears        4382 non-null   float64\n",
      " 16  TrainingTimesLastYear    4382 non-null   int64  \n",
      " 17  YearsAtCompany           4382 non-null   int64  \n",
      " 18  YearsSinceLastPromotion  4382 non-null   int64  \n",
      " 19  YearsWithCurrManager     4382 non-null   int64  \n",
      "dtypes: float64(2), int32(7), int64(11)\n",
      "memory usage: 599.1 KB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Age</th>\n",
       "      <td>51.0</td>\n",
       "      <td>31.0</td>\n",
       "      <td>32.0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>32.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Attrition</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BusinessTravel</th>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Department</th>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DistanceFromHome</th>\n",
       "      <td>6.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Education</th>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>EducationField</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Gender</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>JobLevel</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>JobRole</th>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>7.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MaritalStatus</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MonthlyIncome</th>\n",
       "      <td>131160.0</td>\n",
       "      <td>41890.0</td>\n",
       "      <td>193280.0</td>\n",
       "      <td>83210.0</td>\n",
       "      <td>23420.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NumCompaniesWorked</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PercentSalaryHike</th>\n",
       "      <td>11.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>StockOptionLevel</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TotalWorkingYears</th>\n",
       "      <td>1.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>9.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TrainingTimesLastYear</th>\n",
       "      <td>6.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>YearsAtCompany</th>\n",
       "      <td>1.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>YearsSinceLastPromotion</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>YearsWithCurrManager</th>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                0        1         2        3        4\n",
       "Age                          51.0     31.0      32.0     38.0     32.0\n",
       "Attrition                     0.0      1.0       0.0      0.0      0.0\n",
       "BusinessTravel                2.0      1.0       1.0      0.0      2.0\n",
       "Department                    2.0      1.0       1.0      1.0      1.0\n",
       "DistanceFromHome              6.0     10.0      17.0      2.0     10.0\n",
       "Education                     2.0      1.0       4.0      5.0      1.0\n",
       "EducationField                1.0      1.0       4.0      1.0      3.0\n",
       "Gender                        0.0      0.0       1.0      1.0      1.0\n",
       "JobLevel                      1.0      1.0       4.0      3.0      1.0\n",
       "JobRole                       0.0      6.0       7.0      1.0      7.0\n",
       "MaritalStatus                 1.0      2.0       1.0      1.0      2.0\n",
       "MonthlyIncome            131160.0  41890.0  193280.0  83210.0  23420.0\n",
       "NumCompaniesWorked            1.0      0.0       1.0      3.0      4.0\n",
       "PercentSalaryHike            11.0     23.0      15.0     11.0     12.0\n",
       "StockOptionLevel              0.0      1.0       3.0      3.0      2.0\n",
       "TotalWorkingYears             1.0      6.0       5.0     13.0      9.0\n",
       "TrainingTimesLastYear         6.0      3.0       2.0      5.0      2.0\n",
       "YearsAtCompany                1.0      5.0       5.0      8.0      6.0\n",
       "YearsSinceLastPromotion       0.0      1.0       0.0      7.0      0.0\n",
       "YearsWithCurrManager          0.0      4.0       3.0      5.0      4.0"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head().T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3505, 19)\n",
      "(877, 19)\n",
      "(3505,)\n",
      "(877,)\n"
     ]
    }
   ],
   "source": [
    "X = data.drop('Attrition', axis=1)\n",
    "Y = data['Attrition']\n",
    "\n",
    "# Train and validation split\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2, random_state=5)\n",
    "print(X_train.shape)\n",
    "print(X_test.shape)\n",
    "print(Y_train.shape)\n",
    "print(Y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **4.** Logistic Regression with **Statsmodels**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimization terminated successfully.\n",
      "         Current function value: 0.399419\n",
      "         Iterations 7\n",
      "                           Logit Regression Results                           \n",
      "==============================================================================\n",
      "Dep. Variable:              Attrition   No. Observations:                 3505\n",
      "Model:                          Logit   Df Residuals:                     3485\n",
      "Method:                           MLE   Df Model:                           19\n",
      "Date:                Fri, 17 Jun 2022   Pseudo R-squ.:                  0.1051\n",
      "Time:                        17:51:07   Log-Likelihood:                -1400.0\n",
      "converged:                       True   LL-Null:                       -1564.4\n",
      "Covariance Type:            nonrobust   LLR p-value:                 2.388e-58\n",
      "==============================================================================\n",
      "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
      "------------------------------------------------------------------------------\n",
      "const         -1.8948      0.057    -33.497      0.000      -2.006      -1.784\n",
      "x1            -0.2976      0.069     -4.305      0.000      -0.433      -0.162\n",
      "x2            -0.0088      0.049     -0.182      0.856      -0.104       0.086\n",
      "x3            -0.1451      0.048     -3.024      0.002      -0.239      -0.051\n",
      "x4             0.0180      0.048      0.373      0.709      -0.077       0.113\n",
      "x5            -0.0573      0.048     -1.186      0.236      -0.152       0.037\n",
      "x6            -0.0933      0.049     -1.907      0.057      -0.189       0.003\n",
      "x7             0.0254      0.049      0.521      0.602      -0.070       0.121\n",
      "x8            -0.0308      0.049     -0.630      0.529      -0.127       0.065\n",
      "x9             0.0769      0.049      1.570      0.116      -0.019       0.173\n",
      "x10            0.4372      0.051      8.491      0.000       0.336       0.538\n",
      "x11           -0.0780      0.050     -1.566      0.117      -0.176       0.020\n",
      "x12            0.2805      0.051      5.490      0.000       0.180       0.381\n",
      "x13            0.0620      0.048      1.303      0.192      -0.031       0.155\n",
      "x14           -0.0635      0.049     -1.297      0.195      -0.159       0.032\n",
      "x15           -0.3980      0.101     -3.930      0.000      -0.596      -0.199\n",
      "x16           -0.1480      0.050     -2.960      0.003      -0.246      -0.050\n",
      "x17            0.0751      0.125      0.603      0.547      -0.169       0.319\n",
      "x18            0.3965      0.073      5.414      0.000       0.253       0.540\n",
      "x19           -0.4898      0.089     -5.524      0.000      -0.664      -0.316\n",
      "==============================================================================\n"
     ]
    }
   ],
   "source": [
    "# Normalization of the features\n",
    "Scaler_X = StandardScaler()\n",
    "X_train  = Scaler_X.fit_transform(X_train)\n",
    "X_test   = Scaler_X.transform(X_test)\n",
    "\n",
    "# To have the intercept in the model\n",
    "# (in Statsmodels the intercept has to be added manually)\n",
    "X_train = sm.add_constant(X_train)\n",
    "X_test  = sm.add_constant(X_test)\n",
    "\n",
    "# Logistic Regression\n",
    "model = sm.Logit(Y_train, X_train)\n",
    "model = model.fit()\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "FULL OUTPUT\n",
    "\n",
    "Optimization terminated successfully.\n",
    "         Current function value: 0.399419\n",
    "         Iterations 7\n",
    "                           Logit Regression Results                           \n",
    "==============================================================================\n",
    "Dep. Variable:              Attrition   No. Observations:                 3505\n",
    "Model:                          Logit   Df Residuals:                     3485\n",
    "Method:                           MLE   Df Model:                           19\n",
    "Date:                Fri, 17 Jun 2022   Pseudo R-squ.:                  0.1051\n",
    "Time:                        17:51:07   Log-Likelihood:                -1400.0\n",
    "converged:                       True   LL-Null:                       -1564.4\n",
    "Covariance Type:            nonrobust   LLR p-value:                 2.388e-58\n",
    "==============================================================================\n",
    "                 coef    std err          z      P>|z|      [0.025      0.975]\n",
    "------------------------------------------------------------------------------\n",
    "const         -1.8948      0.057    -33.497      0.000      -2.006      -1.784\n",
    "x1            -0.2976      0.069     -4.305      0.000      -0.433      -0.162\n",
    "x2            -0.0088      0.049     -0.182      0.856      -0.104       0.086\n",
    "x3            -0.1451      0.048     -3.024      0.002      -0.239      -0.051\n",
    "x4             0.0180      0.048      0.373      0.709      -0.077       0.113\n",
    "x5            -0.0573      0.048     -1.186      0.236      -0.152       0.037\n",
    "x6            -0.0933      0.049     -1.907      0.057      -0.189       0.003\n",
    "x7             0.0254      0.049      0.521      0.602      -0.070       0.121\n",
    "x8            -0.0308      0.049     -0.630      0.529      -0.127       0.065\n",
    "x9             0.0769      0.049      1.570      0.116      -0.019       0.173\n",
    "x10            0.4372      0.051      8.491      0.000       0.336       0.538\n",
    "x11           -0.0780      0.050     -1.566      0.117      -0.176       0.020\n",
    "x12            0.2805      0.051      5.490      0.000       0.180       0.381\n",
    "x13            0.0620      0.048      1.303      0.192      -0.031       0.155\n",
    "x14           -0.0635      0.049     -1.297      0.195      -0.159       0.032\n",
    "x15           -0.3980      0.101     -3.930      0.000      -0.596      -0.199\n",
    "x16           -0.1480      0.050     -2.960      0.003      -0.246      -0.050\n",
    "x17            0.0751      0.125      0.603      0.547      -0.169       0.319\n",
    "x18            0.3965      0.073      5.414      0.000       0.253       0.540\n",
    "x19           -0.4898      0.089     -5.524      0.000      -0.664      -0.316\n",
    "==============================================================================\n",
    "\n",
    "'''"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.10 64-bit (system)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "9805bccb659ffcb6952b8b29440036fac091f8170d71a5a402ad9a8f2988209f"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}