Firstproject.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyzing borrowers’ risk of defaulting\n",
    "\n",
    "Your project is to prepare a report for a bank’s loan division. You’ll need to find out if a customer’s marital status and number of children has an impact on whether they will default on a loan. The bank already has some data on customers’ credit worthiness.\n",
    "\n",
    "Your report will be considered when building a **credit scoring** of a potential customer. A ** credit scoring ** is used to evaluate the ability of a potential borrower to repay their loan."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1. Open the data file and have a look at the general information. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "data = pd.read_csv('/datasets/credit_scoring_eng.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I opened the file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 21525 entries, 0 to 21524\n",
      "Data columns (total 12 columns):\n",
      "children            21525 non-null int64\n",
      "days_employed       19351 non-null float64\n",
      "dob_years           21525 non-null int64\n",
      "education           21525 non-null object\n",
      "education_id        21525 non-null int64\n",
      "family_status       21525 non-null object\n",
      "family_status_id    21525 non-null int64\n",
      "gender              21525 non-null object\n",
      "income_type         21525 non-null object\n",
      "debt                21525 non-null int64\n",
      "total_income        19351 non-null float64\n",
      "purpose             21525 non-null object\n",
      "dtypes: float64(2), int64(5), object(5)\n",
      "memory usage: 2.0+ MB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(21525, 12)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking how many rows and columns there are in the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checked the information for the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>children</th>\n",
       "      <th>days_employed</th>\n",
       "      <th>dob_years</th>\n",
       "      <th>education</th>\n",
       "      <th>education_id</th>\n",
       "      <th>family_status</th>\n",
       "      <th>family_status_id</th>\n",
       "      <th>gender</th>\n",
       "      <th>income_type</th>\n",
       "      <th>debt</th>\n",
       "      <th>total_income</th>\n",
       "      <th>purpose</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>-8437.673028</td>\n",
       "      <td>42</td>\n",
       "      <td>bachelor's degree</td>\n",
       "      <td>0</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>F</td>\n",
       "      <td>employee</td>\n",
       "      <td>0</td>\n",
       "      <td>40620.102</td>\n",
       "      <td>purchase of the house</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-4024.803754</td>\n",
       "      <td>36</td>\n",
       "      <td>secondary education</td>\n",
       "      <td>1</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>F</td>\n",
       "      <td>employee</td>\n",
       "      <td>0</td>\n",
       "      <td>17932.802</td>\n",
       "      <td>car purchase</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>-5623.422610</td>\n",
       "      <td>33</td>\n",
       "      <td>Secondary Education</td>\n",
       "      <td>1</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>employee</td>\n",
       "      <td>0</td>\n",
       "      <td>23341.752</td>\n",
       "      <td>purchase of the house</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>-4124.747207</td>\n",
       "      <td>32</td>\n",
       "      <td>secondary education</td>\n",
       "      <td>1</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>employee</td>\n",
       "      <td>0</td>\n",
       "      <td>42820.568</td>\n",
       "      <td>supplementary education</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>340266.072047</td>\n",
       "      <td>53</td>\n",
       "      <td>secondary education</td>\n",
       "      <td>1</td>\n",
       "      <td>civil partnership</td>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>retiree</td>\n",
       "      <td>0</td>\n",
       "      <td>25378.572</td>\n",
       "      <td>to have a wedding</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>-926.185831</td>\n",
       "      <td>27</td>\n",
       "      <td>bachelor's degree</td>\n",
       "      <td>0</td>\n",
       "      <td>civil partnership</td>\n",
       "      <td>1</td>\n",
       "      <td>M</td>\n",
       "      <td>business</td>\n",
       "      <td>0</td>\n",
       "      <td>40922.170</td>\n",
       "      <td>purchase of the house</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>-2879.202052</td>\n",
       "      <td>43</td>\n",
       "      <td>bachelor's degree</td>\n",
       "      <td>0</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>F</td>\n",
       "      <td>business</td>\n",
       "      <td>0</td>\n",
       "      <td>38484.156</td>\n",
       "      <td>housing transactions</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>-152.779569</td>\n",
       "      <td>50</td>\n",
       "      <td>SECONDARY EDUCATION</td>\n",
       "      <td>1</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>employee</td>\n",
       "      <td>0</td>\n",
       "      <td>21731.829</td>\n",
       "      <td>education</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2</td>\n",
       "      <td>-6929.865299</td>\n",
       "      <td>35</td>\n",
       "      <td>BACHELOR'S DEGREE</td>\n",
       "      <td>0</td>\n",
       "      <td>civil partnership</td>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>employee</td>\n",
       "      <td>0</td>\n",
       "      <td>15337.093</td>\n",
       "      <td>having a wedding</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>-2188.756445</td>\n",
       "      <td>41</td>\n",
       "      <td>secondary education</td>\n",
       "      <td>1</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>employee</td>\n",
       "      <td>0</td>\n",
       "      <td>23108.150</td>\n",
       "      <td>purchase of the house for my family</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   children  days_employed  dob_years            education  education_id  \\\n",
       "0         1   -8437.673028         42    bachelor's degree             0   \n",
       "1         1   -4024.803754         36  secondary education             1   \n",
       "2         0   -5623.422610         33  Secondary Education             1   \n",
       "3         3   -4124.747207         32  secondary education             1   \n",
       "4         0  340266.072047         53  secondary education             1   \n",
       "5         0    -926.185831         27    bachelor's degree             0   \n",
       "6         0   -2879.202052         43    bachelor's degree             0   \n",
       "7         0    -152.779569         50  SECONDARY EDUCATION             1   \n",
       "8         2   -6929.865299         35    BACHELOR'S DEGREE             0   \n",
       "9         0   -2188.756445         41  secondary education             1   \n",
       "\n",
       "       family_status  family_status_id gender income_type  debt  total_income  \\\n",
       "0            married                 0      F    employee     0     40620.102   \n",
       "1            married                 0      F    employee     0     17932.802   \n",
       "2            married                 0      M    employee     0     23341.752   \n",
       "3            married                 0      M    employee     0     42820.568   \n",
       "4  civil partnership                 1      F     retiree     0     25378.572   \n",
       "5  civil partnership                 1      M    business     0     40922.170   \n",
       "6            married                 0      F    business     0     38484.156   \n",
       "7            married                 0      M    employee     0     21731.829   \n",
       "8  civil partnership                 1      F    employee     0     15337.093   \n",
       "9            married                 0      M    employee     0     23108.150   \n",
       "\n",
       "                               purpose  \n",
       "0                purchase of the house  \n",
       "1                         car purchase  \n",
       "2                purchase of the house  \n",
       "3              supplementary education  \n",
       "4                    to have a wedding  \n",
       "5                purchase of the house  \n",
       "6                 housing transactions  \n",
       "7                            education  \n",
       "8                     having a wedding  \n",
       "9  purchase of the house for my family  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checked ten rows of the data to see what I am dealing with. Noticed issues of uppercase lowercase as well as purpose having different wording in the data. I will only really need children and marital status for the questions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['children', 'days_employed', 'dob_years', 'education', 'education_id',\n",
       "       'family_status', 'family_status_id', 'gender', 'income_type', 'debt',\n",
       "       'total_income', 'purpose'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checked which columns I have"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "35    617\n",
       "40    609\n",
       "41    607\n",
       "34    603\n",
       "38    598\n",
       "42    597\n",
       "33    581\n",
       "39    573\n",
       "31    560\n",
       "36    555\n",
       "44    547\n",
       "29    545\n",
       "30    540\n",
       "48    538\n",
       "37    537\n",
       "50    514\n",
       "43    513\n",
       "32    510\n",
       "49    508\n",
       "28    503\n",
       "45    497\n",
       "27    493\n",
       "56    487\n",
       "52    484\n",
       "47    480\n",
       "54    479\n",
       "46    475\n",
       "58    461\n",
       "57    460\n",
       "53    459\n",
       "51    448\n",
       "59    444\n",
       "55    443\n",
       "26    408\n",
       "60    377\n",
       "25    357\n",
       "61    355\n",
       "62    352\n",
       "63    269\n",
       "64    265\n",
       "24    264\n",
       "23    254\n",
       "65    194\n",
       "66    183\n",
       "22    183\n",
       "67    167\n",
       "21    111\n",
       "0     101\n",
       "68     99\n",
       "69     85\n",
       "70     65\n",
       "71     58\n",
       "20     51\n",
       "72     33\n",
       "19     14\n",
       "73      8\n",
       "74      6\n",
       "75      1\n",
       "Name: dob_years, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['dob_years'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checked data of dob_years. Found 101 people had 0 as an answer. That is not a significant number in the amount that we have of data. But also I would say its the 101 people who did not want to tell their age."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    19784\n",
       "1     1741\n",
       "Name: debt, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['debt'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checked how many people defaulted on debt. Only 1741 have out of 19784. Good amount for customer credit scoring"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "employee                       11119\n",
       "business                        5085\n",
       "retiree                         3856\n",
       "civil servant                   1459\n",
       "unemployed                         2\n",
       "entrepreneur                       2\n",
       "paternity / maternity leave        1\n",
       "student                            1\n",
       "Name: income_type, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['income_type'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checked data on how many people in data have a job. Over 90% have an income source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "children              int64\n",
       "days_employed       float64\n",
       "dob_years             int64\n",
       "education            object\n",
       "education_id          int64\n",
       "family_status        object\n",
       "family_status_id      int64\n",
       "gender               object\n",
       "income_type          object\n",
       "debt                  int64\n",
       "total_income        float64\n",
       "purpose              object\n",
       "dtype: object"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checked the data types and days_employed and total_income are floating data types and I will change them to integer types. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>children</th>\n",
       "      <th>days_employed</th>\n",
       "      <th>dob_years</th>\n",
       "      <th>education_id</th>\n",
       "      <th>family_status_id</th>\n",
       "      <th>debt</th>\n",
       "      <th>total_income</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>count</td>\n",
       "      <td>21525.000000</td>\n",
       "      <td>19351.000000</td>\n",
       "      <td>21525.000000</td>\n",
       "      <td>21525.000000</td>\n",
       "      <td>21525.000000</td>\n",
       "      <td>21525.000000</td>\n",
       "      <td>19351.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>mean</td>\n",
       "      <td>0.538908</td>\n",
       "      <td>63046.497661</td>\n",
       "      <td>43.293380</td>\n",
       "      <td>0.817236</td>\n",
       "      <td>0.972544</td>\n",
       "      <td>0.080883</td>\n",
       "      <td>26787.568355</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>std</td>\n",
       "      <td>1.381587</td>\n",
       "      <td>140827.311974</td>\n",
       "      <td>12.574584</td>\n",
       "      <td>0.548138</td>\n",
       "      <td>1.420324</td>\n",
       "      <td>0.272661</td>\n",
       "      <td>16475.450632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>min</td>\n",
       "      <td>-1.000000</td>\n",
       "      <td>-18388.949901</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3306.762000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25%</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-2747.423625</td>\n",
       "      <td>33.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>16488.504500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>50%</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>-1203.369529</td>\n",
       "      <td>42.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>23202.870000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>75%</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>-291.095954</td>\n",
       "      <td>53.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>32549.611000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>max</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>401755.400475</td>\n",
       "      <td>75.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>362496.645000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           children  days_employed     dob_years  education_id  \\\n",
       "count  21525.000000   19351.000000  21525.000000  21525.000000   \n",
       "mean       0.538908   63046.497661     43.293380      0.817236   \n",
       "std        1.381587  140827.311974     12.574584      0.548138   \n",
       "min       -1.000000  -18388.949901      0.000000      0.000000   \n",
       "25%        0.000000   -2747.423625     33.000000      1.000000   \n",
       "50%        0.000000   -1203.369529     42.000000      1.000000   \n",
       "75%        1.000000    -291.095954     53.000000      1.000000   \n",
       "max       20.000000  401755.400475     75.000000      4.000000   \n",
       "\n",
       "       family_status_id          debt   total_income  \n",
       "count      21525.000000  21525.000000   19351.000000  \n",
       "mean           0.972544      0.080883   26787.568355  \n",
       "std            1.420324      0.272661   16475.450632  \n",
       "min            0.000000      0.000000    3306.762000  \n",
       "25%            0.000000      0.000000   16488.504500  \n",
       "50%            0.000000      0.000000   23202.870000  \n",
       "75%            1.000000      0.000000   32549.611000  \n",
       "max            4.000000      1.000000  362496.645000  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checked data more clearly and most people have an education of some sort. I found negative numbers in children and days-employed and I will change that. Also there seems to be families with 20 children. I see that as a typo and will change it to 2 children"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "children               0\n",
       "days_employed       2174\n",
       "dob_years              0\n",
       "education              0\n",
       "education_id           0\n",
       "family_status          0\n",
       "family_status_id       0\n",
       "gender                 0\n",
       "income_type            0\n",
       "debt                   0\n",
       "total_income        2174\n",
       "purpose                0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I found there to be missing values in the column days_employed and total_income"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I examined the data and noticed missing values in 2174 rows of the columns days_employed and total_income. Also the children column and days_employed column have negative numbers. There are families with 20 children but this does not make sense and I will count this as a typo. In the education column there are issues of lowercase and uppercase. I will change everything to lowercase. Also discovered that 101 people have 0 in age. This is probably because they declined to write there age for the analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2. Data preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['children'] = data['children'].abs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Changed the Children column to positive numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['children'] = data['children'].replace(20,2) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Changed the 20 in 'Children' column to 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    21525.000000\n",
       "mean         0.479721\n",
       "std          0.755528\n",
       "min          0.000000\n",
       "25%          0.000000\n",
       "50%          0.000000\n",
       "75%          1.000000\n",
       "max          5.000000\n",
       "Name: children, dtype: float64"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['children'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "checking if the code worked for the Children column and it looks like it did. Instead of 20 max I have 5 max and min is not negative 1 anymore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['days_employed'] = data['days_employed'].abs()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "changed the Days_employed column to positive numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count     19351.000000\n",
       "mean      66914.728907\n",
       "std      139030.880527\n",
       "min          24.141633\n",
       "25%         927.009265\n",
       "50%        2194.220567\n",
       "75%        5537.882441\n",
       "max      401755.400475\n",
       "Name: days_employed, dtype: float64"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['days_employed'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "check to see if days_employed is positive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "secondary education    13750\n",
       "bachelor's degree       4718\n",
       "SECONDARY EDUCATION      772\n",
       "Secondary Education      711\n",
       "some college             668\n",
       "BACHELOR'S DEGREE        274\n",
       "Bachelor's Degree        268\n",
       "primary education        250\n",
       "Some College              47\n",
       "SOME COLLEGE              29\n",
       "PRIMARY EDUCATION         17\n",
       "Primary Education         15\n",
       "graduate degree            4\n",
       "Graduate Degree            1\n",
       "GRADUATE DEGREE            1\n",
       "Name: education, dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['education'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "looking into education column to see the issue of lowercase uppercase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['education'] = data['education'].str.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Made all the education column lowercase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "secondary education    15233\n",
       "bachelor's degree       5260\n",
       "some college             744\n",
       "primary education        282\n",
       "graduate degree            6\n",
       "Name: education, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['education'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "checked to see if lowercase worked."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['purpose'] = data['purpose'].str.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even though did not see any issue of lowercase in purpose it did not hurt to do the same for purpose in case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "wedding ceremony                            797\n",
       "having a wedding                            777\n",
       "to have a wedding                           774\n",
       "real estate transactions                    676\n",
       "buy commercial real estate                  664\n",
       "buying property for renting out             653\n",
       "housing transactions                        653\n",
       "transactions with commercial real estate    651\n",
       "housing                                     647\n",
       "purchase of the house                       647\n",
       "purchase of the house for my family         641\n",
       "construction of own property                635\n",
       "property                                    634\n",
       "transactions with my real estate            630\n",
       "building a real estate                      626\n",
       "buy real estate                             624\n",
       "building a property                         620\n",
       "purchase of my own house                    620\n",
       "housing renovation                          612\n",
       "buy residential real estate                 607\n",
       "buying my own car                           505\n",
       "going to university                         496\n",
       "car                                         495\n",
       "second-hand car purchase                    489\n",
       "to own a car                                480\n",
       "buying a second-hand car                    479\n",
       "cars                                        478\n",
       "to buy a car                                472\n",
       "car purchase                                462\n",
       "supplementary education                     462\n",
       "purchase of a car                           455\n",
       "university education                        453\n",
       "to get a supplementary education            447\n",
       "education                                   447\n",
       "getting an education                        443\n",
       "profile education                           436\n",
       "getting higher education                    426\n",
       "to become educated                          412\n",
       "Name: purpose, dtype: int64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['purpose'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checked to see all the different categories in purpose. Will need to categorize this better in order to analyze the data better"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Processing missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "median_employed = data['days_employed'].median()\n",
    "data['days_employed'].fillna(median_employed, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Filled in the missing data for days employed with the median of days_employed so there will be some value there. It may mess up the data later but I do not believe it will be significant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count     21525.000000\n",
       "mean      60378.032733\n",
       "std      133257.558514\n",
       "min          24.141633\n",
       "25%        1025.608174\n",
       "50%        2194.220567\n",
       "75%        4779.587738\n",
       "max      401755.400475\n",
       "Name: days_employed, dtype: float64"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['days_employed'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This column will not be significant in calculating the conclusion because we need to look at marital status and number of children more closely. It also is I believe MCAR based on the fact that I can still figure out analysis from other columns. I decided to use this method because it was simplest for me and also because I do not believe it will be significant in my investigation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "median_income = data['total_income'].median()\n",
    "data['total_income'].fillna(median_employed, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did the same with total income in which I did with days employed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count     21525.000000\n",
       "mean      24303.668792\n",
       "std       17290.003687\n",
       "min        2194.220567\n",
       "25%       14178.053000\n",
       "50%       21682.354000\n",
       "75%       31286.979000\n",
       "max      362496.645000\n",
       "Name: total_income, dtype: float64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['total_income'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I don't think this will make a significant change in the data and it was the easiest to use the median here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "children            0\n",
       "days_employed       0\n",
       "dob_years           0\n",
       "education           0\n",
       "education_id        0\n",
       "family_status       0\n",
       "family_status_id    0\n",
       "gender              0\n",
       "income_type         0\n",
       "debt                0\n",
       "total_income        0\n",
       "purpose             0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checked to see if there were still any missing values but there are none"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I replaced all the missing values in 'days_employed' and 'total_income' with median value. This changed the data a little bit but not significantly enough for what we are investigating."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data type replacement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['days_employed'] = data['days_employed'].astype('int')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['total_income'] = data['total_income'].astype('int')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "changed days employed and total income to integer type to better work with the data and have all of the number columns be in integer. Also to change them with astype is better because it won't change the other way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 21525 entries, 0 to 21524\n",
      "Data columns (total 12 columns):\n",
      "children            21525 non-null int64\n",
      "days_employed       21525 non-null int64\n",
      "dob_years           21525 non-null int64\n",
      "education           21525 non-null object\n",
      "education_id        21525 non-null int64\n",
      "family_status       21525 non-null object\n",
      "family_status_id    21525 non-null int64\n",
      "gender              21525 non-null object\n",
      "income_type         21525 non-null object\n",
      "debt                21525 non-null int64\n",
      "total_income        21525 non-null int64\n",
      "purpose             21525 non-null object\n",
      "dtypes: int64(7), object(5)\n",
      "memory usage: 2.0+ MB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I was able to change the days_employed column and total_income column to integer instead of float and should be able to work better with the data now."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Processing duplicates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "71"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.duplicated().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking how many duplicated data I have"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>children</th>\n",
       "      <th>days_employed</th>\n",
       "      <th>dob_years</th>\n",
       "      <th>education</th>\n",
       "      <th>education_id</th>\n",
       "      <th>family_status</th>\n",
       "      <th>family_status_id</th>\n",
       "      <th>gender</th>\n",
       "      <th>income_type</th>\n",
       "      <th>debt</th>\n",
       "      <th>total_income</th>\n",
       "      <th>purpose</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>2849</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>41</td>\n",
       "      <td>secondary education</td>\n",
       "      <td>1</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>F</td>\n",
       "      <td>employee</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>purchase of the house for my family</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3290</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>58</td>\n",
       "      <td>secondary education</td>\n",
       "      <td>1</td>\n",
       "      <td>civil partnership</td>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>retiree</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>to have a wedding</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4182</td>\n",
       "      <td>1</td>\n",
       "      <td>2194</td>\n",
       "      <td>34</td>\n",
       "      <td>bachelor's degree</td>\n",
       "      <td>0</td>\n",
       "      <td>civil partnership</td>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>employee</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>wedding ceremony</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4851</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>60</td>\n",
       "      <td>secondary education</td>\n",
       "      <td>1</td>\n",
       "      <td>civil partnership</td>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>retiree</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>wedding ceremony</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5557</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>58</td>\n",
       "      <td>secondary education</td>\n",
       "      <td>1</td>\n",
       "      <td>civil partnership</td>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>retiree</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>to have a wedding</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20702</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>64</td>\n",
       "      <td>secondary education</td>\n",
       "      <td>1</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>F</td>\n",
       "      <td>retiree</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>supplementary education</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21032</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>60</td>\n",
       "      <td>secondary education</td>\n",
       "      <td>1</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>F</td>\n",
       "      <td>retiree</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>to become educated</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21132</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>47</td>\n",
       "      <td>secondary education</td>\n",
       "      <td>1</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>F</td>\n",
       "      <td>employee</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>housing renovation</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21281</td>\n",
       "      <td>1</td>\n",
       "      <td>2194</td>\n",
       "      <td>30</td>\n",
       "      <td>bachelor's degree</td>\n",
       "      <td>0</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>F</td>\n",
       "      <td>employee</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>buy commercial real estate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21415</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>54</td>\n",
       "      <td>secondary education</td>\n",
       "      <td>1</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>F</td>\n",
       "      <td>retiree</td>\n",
       "      <td>0</td>\n",
       "      <td>2194</td>\n",
       "      <td>housing transactions</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>71 rows × 12 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       children  days_employed  dob_years            education  education_id  \\\n",
       "2849          0           2194         41  secondary education             1   \n",
       "3290          0           2194         58  secondary education             1   \n",
       "4182          1           2194         34    bachelor's degree             0   \n",
       "4851          0           2194         60  secondary education             1   \n",
       "5557          0           2194         58  secondary education             1   \n",
       "...         ...            ...        ...                  ...           ...   \n",
       "20702         0           2194         64  secondary education             1   \n",
       "21032         0           2194         60  secondary education             1   \n",
       "21132         0           2194         47  secondary education             1   \n",
       "21281         1           2194         30    bachelor's degree             0   \n",
       "21415         0           2194         54  secondary education             1   \n",
       "\n",
       "           family_status  family_status_id gender income_type  debt  \\\n",
       "2849             married                 0      F    employee     0   \n",
       "3290   civil partnership                 1      F     retiree     0   \n",
       "4182   civil partnership                 1      F    employee     0   \n",
       "4851   civil partnership                 1      F     retiree     0   \n",
       "5557   civil partnership                 1      F     retiree     0   \n",
       "...                  ...               ...    ...         ...   ...   \n",
       "20702            married                 0      F     retiree     0   \n",
       "21032            married                 0      F     retiree     0   \n",
       "21132            married                 0      F    employee     0   \n",
       "21281            married                 0      F    employee     0   \n",
       "21415            married                 0      F     retiree     0   \n",
       "\n",
       "       total_income                              purpose  \n",
       "2849           2194  purchase of the house for my family  \n",
       "3290           2194                    to have a wedding  \n",
       "4182           2194                     wedding ceremony  \n",
       "4851           2194                     wedding ceremony  \n",
       "5557           2194                    to have a wedding  \n",
       "...             ...                                  ...  \n",
       "20702          2194              supplementary education  \n",
       "21032          2194                   to become educated  \n",
       "21132          2194                   housing renovation  \n",
       "21281          2194           buy commercial real estate  \n",
       "21415          2194                 housing transactions  \n",
       "\n",
       "[71 rows x 12 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.loc[data.duplicated(), :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are duplications in days employed and total income column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I dropped the duplicates from the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = data.drop_duplicates(keep = 'first')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.duplicated().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checked to see if any duplicates remained"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking the 71 duplications and in Days_employed and total_income columns they are the same numbers. This happened because I added the median to fill the missing data. This won't affect the data that I am looking at and 71 duplicates is only .32% of the data so should not affect my conclusion. I dropped the duplicated data from the dataframe and have no more duplicates. Days_employed is not a necessary column for my analysis because I am not assessing how long the person has worked but I am looking at their total_income. This second column may affect the analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Categorizing Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.stem import WordNetLemmatizer\n",
    " \n",
    "def lemmatizing(text):\n",
    "    wordnet_lemma = WordNetLemmatizer()\n",
    "    words = nltk.word_tokenize(text)\n",
    "    return [wordnet_lemma.lemmatize(w, pos = 'n') for w in words]\n",
    " \n",
    "data['lemmas'] = data['purpose'].apply(lemmatizing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Importing the lemmatizer to be able to split the purpose column into different categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        [purchase, of, the, house]\n",
       "1                   [car, purchase]\n",
       "2        [purchase, of, the, house]\n",
       "3        [supplementary, education]\n",
       "4            [to, have, a, wedding]\n",
       "                    ...            \n",
       "21520        [housing, transaction]\n",
       "21521        [purchase, of, a, car]\n",
       "21522                    [property]\n",
       "21523        [buying, my, own, car]\n",
       "21524             [to, buy, a, car]\n",
       "Name: lemmas, Length: 21454, dtype: object"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['lemmas']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "checking the lemmatization worked"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[car]                                            972\n",
       "[wedding, ceremony]                              791\n",
       "[having, a, wedding]                             768\n",
       "[to, have, a, wedding]                           765\n",
       "[real, estate, transaction]                      675\n",
       "[buy, commercial, real, estate]                  661\n",
       "[housing, transaction]                           652\n",
       "[buying, property, for, renting, out]            651\n",
       "[transaction, with, commercial, real, estate]    650\n",
       "[purchase, of, the, house]                       646\n",
       "[housing]                                        646\n",
       "[purchase, of, the, house, for, my, family]      638\n",
       "[construction, of, own, property]                635\n",
       "[property]                                       633\n",
       "[transaction, with, my, real, estate]            627\n",
       "[building, a, real, estate]                      624\n",
       "[buy, real, estate]                              621\n",
       "[purchase, of, my, own, house]                   620\n",
       "[building, a, property]                          619\n",
       "[housing, renovation]                            607\n",
       "[buy, residential, real, estate]                 606\n",
       "[buying, my, own, car]                           505\n",
       "[going, to, university]                          496\n",
       "[second-hand, car, purchase]                     486\n",
       "[to, own, a, car]                                478\n",
       "[buying, a, second-hand, car]                    478\n",
       "[to, buy, a, car]                                471\n",
       "[car, purchase]                                  461\n",
       "[supplementary, education]                       460\n",
       "[purchase, of, a, car]                           455\n",
       "[university, education]                          452\n",
       "[education]                                      447\n",
       "[to, get, a, supplementary, education]           446\n",
       "[getting, an, education]                         442\n",
       "[profile, education]                             436\n",
       "[getting, higher, education]                     426\n",
       "[to, become, educated]                           408\n",
       "Name: lemmas, dtype: int64"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['lemmas'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checking the different categories I would like to make based on the lemmatization. I will make a category of wedding, car, real estate, personal property, and education"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import SnowballStemmer\n",
    "english_stemmer = SnowballStemmer('english')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "importing stemmer to see if any words need to be added to my function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['purchas', 'of', 'the', 'hous']"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stemmed_list_0=[english_stemmer.stem(word)for word in data['purpose'][0].split(\" \")]\n",
    "stemmed_list_0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['car', 'purchas']"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stemmed_list_1=[english_stemmer.stem(word)for word in data['purpose'][1].split(\" \")]\n",
    "stemmed_list_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['supplementari', 'educ']"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stemmed_list_3=[english_stemmer.stem(word)for word in data['purpose'][3].split(\" \")]\n",
    "stemmed_list_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['to', 'have', 'a', 'wed']"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stemmed_list_4=[english_stemmer.stem(word)for word in data['purpose'][4].split(\" \")]\n",
    "stemmed_list_4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def purpose_category(element):\n",
    "    if 'estate' in element or 'housing' in element or 'house' in element or 'construction' in element or 'property' in element or'renting' in element:\n",
    "        element = 'property'\n",
    "    elif 'university' in element or 'education' in element or 'educated' in element:\n",
    "        element = 'education'\n",
    "    elif 'wedding' in element:\n",
    "        element = 'wedding'\n",
    "    elif 'car' in element:\n",
    "        element = 'car'\n",
    "    return element\n",
    "   \n",
    "data['purpose_category'] = data['purpose'].apply(purpose_category)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Created purpose category column and applied it to the data. I took four separate categories to make the data clearer and get rid of too many variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "property     10811\n",
       "car           4306\n",
       "education     4013\n",
       "wedding       2324\n",
       "Name: purpose_category, dtype: int64"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['purpose_category'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "now I have four categories and much easier to work with the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count     21454.000000\n",
       "mean      24376.367950\n",
       "std       17271.632609\n",
       "min        2194.000000\n",
       "25%       14254.000000\n",
       "50%       21724.500000\n",
       "75%       31330.000000\n",
       "max      362496.000000\n",
       "Name: total_income, dtype: float64"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['total_income'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "rechecking total income to pull numbers for my next categorization which would be of income"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def income_bracket(income):\n",
    "    if income < 14178:\n",
    "        return 'low income'\n",
    "    elif income >= 14178 and income <= 21682:\n",
    "        return 'mid income'\n",
    "    elif income > 21682:\n",
    "        return 'high income'\n",
    "    return income\n",
    "data['income_bracket'] = data['total_income'].apply(income_bracket)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Making a new column income_bracket to categorize income into low income bracket, mid income bracket, and high income bracket and applied to to the table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "high income    10762\n",
       "mid income      5382\n",
       "low income      5310\n",
       "Name: income_bracket, dtype: int64"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data['income_bracket'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Found that many people have a high income and then there is a split between mid income and low income."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>children</th>\n",
       "      <th>days_employed</th>\n",
       "      <th>dob_years</th>\n",
       "      <th>education</th>\n",
       "      <th>education_id</th>\n",
       "      <th>family_status</th>\n",
       "      <th>family_status_id</th>\n",
       "      <th>gender</th>\n",
       "      <th>income_type</th>\n",
       "      <th>debt</th>\n",
       "      <th>total_income</th>\n",
       "      <th>purpose</th>\n",
       "      <th>lemmas</th>\n",
       "      <th>purpose_category</th>\n",
       "      <th>income_bracket</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>8437</td>\n",
       "      <td>42</td>\n",
       "      <td>bachelor's degree</td>\n",
       "      <td>0</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>F</td>\n",
       "      <td>employee</td>\n",
       "      <td>0</td>\n",
       "      <td>40620</td>\n",
       "      <td>purchase of the house</td>\n",
       "      <td>[purchase, of, the, house]</td>\n",
       "      <td>property</td>\n",
       "      <td>high income</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4024</td>\n",
       "      <td>36</td>\n",
       "      <td>secondary education</td>\n",
       "      <td>1</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>F</td>\n",
       "      <td>employee</td>\n",
       "      <td>0</td>\n",
       "      <td>17932</td>\n",
       "      <td>car purchase</td>\n",
       "      <td>[car, purchase]</td>\n",
       "      <td>car</td>\n",
       "      <td>mid income</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>5623</td>\n",
       "      <td>33</td>\n",
       "      <td>secondary education</td>\n",
       "      <td>1</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>employee</td>\n",
       "      <td>0</td>\n",
       "      <td>23341</td>\n",
       "      <td>purchase of the house</td>\n",
       "      <td>[purchase, of, the, house]</td>\n",
       "      <td>property</td>\n",
       "      <td>high income</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>4124</td>\n",
       "      <td>32</td>\n",
       "      <td>secondary education</td>\n",
       "      <td>1</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>M</td>\n",
       "      <td>employee</td>\n",
       "      <td>0</td>\n",
       "      <td>42820</td>\n",
       "      <td>supplementary education</td>\n",
       "      <td>[supplementary, education]</td>\n",
       "      <td>education</td>\n",
       "      <td>high income</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>340266</td>\n",
       "      <td>53</td>\n",
       "      <td>secondary education</td>\n",
       "      <td>1</td>\n",
       "      <td>civil partnership</td>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>retiree</td>\n",
       "      <td>0</td>\n",
       "      <td>25378</td>\n",
       "      <td>to have a wedding</td>\n",
       "      <td>[to, have, a, wedding]</td>\n",
       "      <td>wedding</td>\n",
       "      <td>high income</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   children  days_employed  dob_years            education  education_id  \\\n",
       "0         1           8437         42    bachelor's degree             0   \n",
       "1         1           4024         36  secondary education             1   \n",
       "2         0           5623         33  secondary education             1   \n",
       "3         3           4124         32  secondary education             1   \n",
       "4         0         340266         53  secondary education             1   \n",
       "\n",
       "       family_status  family_status_id gender income_type  debt  total_income  \\\n",
       "0            married                 0      F    employee     0         40620   \n",
       "1            married                 0      F    employee     0         17932   \n",
       "2            married                 0      M    employee     0         23341   \n",
       "3            married                 0      M    employee     0         42820   \n",
       "4  civil partnership                 1      F     retiree     0         25378   \n",
       "\n",
       "                   purpose                      lemmas purpose_category  \\\n",
       "0    purchase of the house  [purchase, of, the, house]         property   \n",
       "1             car purchase             [car, purchase]              car   \n",
       "2    purchase of the house  [purchase, of, the, house]         property   \n",
       "3  supplementary education  [supplementary, education]        education   \n",
       "4        to have a wedding      [to, have, a, wedding]          wedding   \n",
       "\n",
       "  income_bracket  \n",
       "0    high income  \n",
       "1     mid income  \n",
       "2    high income  \n",
       "3    high income  \n",
       "4    high income  "
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Checked my table to see everything was there including the new columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "credit_scoring = data.drop(['days_employed', 'dob_years', 'education', 'education_id', 'family_status_id', 'gender', 'income_type', 'purpose', 'lemmas'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "dropped the columns I felt was not needed to contribute to the questions that I would be analyzing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>children</th>\n",
       "      <th>family_status</th>\n",
       "      <th>debt</th>\n",
       "      <th>total_income</th>\n",
       "      <th>purpose_category</th>\n",
       "      <th>income_bracket</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>40620</td>\n",
       "      <td>property</td>\n",
       "      <td>high income</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>17932</td>\n",
       "      <td>car</td>\n",
       "      <td>mid income</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>23341</td>\n",
       "      <td>property</td>\n",
       "      <td>high income</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>42820</td>\n",
       "      <td>education</td>\n",
       "      <td>high income</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>civil partnership</td>\n",
       "      <td>0</td>\n",
       "      <td>25378</td>\n",
       "      <td>wedding</td>\n",
       "      <td>high income</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21520</td>\n",
       "      <td>1</td>\n",
       "      <td>civil partnership</td>\n",
       "      <td>0</td>\n",
       "      <td>35966</td>\n",
       "      <td>property</td>\n",
       "      <td>high income</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21521</td>\n",
       "      <td>0</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>24959</td>\n",
       "      <td>car</td>\n",
       "      <td>high income</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21522</td>\n",
       "      <td>1</td>\n",
       "      <td>civil partnership</td>\n",
       "      <td>1</td>\n",
       "      <td>14347</td>\n",
       "      <td>property</td>\n",
       "      <td>mid income</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21523</td>\n",
       "      <td>3</td>\n",
       "      <td>married</td>\n",
       "      <td>1</td>\n",
       "      <td>39054</td>\n",
       "      <td>car</td>\n",
       "      <td>high income</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21524</td>\n",
       "      <td>2</td>\n",
       "      <td>married</td>\n",
       "      <td>0</td>\n",
       "      <td>13127</td>\n",
       "      <td>car</td>\n",
       "      <td>low income</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>21454 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       children      family_status  debt  total_income purpose_category  \\\n",
       "0             1            married     0         40620         property   \n",
       "1             1            married     0         17932              car   \n",
       "2             0            married     0         23341         property   \n",
       "3             3            married     0         42820        education   \n",
       "4             0  civil partnership     0         25378          wedding   \n",
       "...         ...                ...   ...           ...              ...   \n",
       "21520         1  civil partnership     0         35966         property   \n",
       "21521         0            married     0         24959              car   \n",
       "21522         1  civil partnership     1         14347         property   \n",
       "21523         3            married     1         39054              car   \n",
       "21524         2            married     0         13127              car   \n",
       "\n",
       "      income_bracket  \n",
       "0        high income  \n",
       "1         mid income  \n",
       "2        high income  \n",
       "3        high income  \n",
       "4        high income  \n",
       "...              ...  \n",
       "21520    high income  \n",
       "21521    high income  \n",
       "21522     mid income  \n",
       "21523    high income  \n",
       "21524     low income  \n",
       "\n",
       "[21454 rows x 6 columns]"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credit_scoring"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Printed out the new table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I was able to categorize the purpose column which showed that most loans were taken for housing projects and the least were for weddings. Also categorized the total_income and found that most of the families had a middle income. This will help me determine better what I am analyzing when it comes to income because I will be able to see which family income types default on a loan as well as where the loan is being used for in a much more concise way. I also created data_credit_scoring in order to clean the data for the questions I will be asking. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3. Answer these questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Is there a relation between having kids and repaying a loan on time?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">family_status</th>\n",
       "      <th colspan=\"2\" halign=\"left\">income_bracket</th>\n",
       "      <th colspan=\"2\" halign=\"left\">purpose_category</th>\n",
       "      <th colspan=\"2\" halign=\"left\">total_income</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>debt</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>children</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>13028.0</td>\n",
       "      <td>1063.0</td>\n",
       "      <td>13028.0</td>\n",
       "      <td>1063.0</td>\n",
       "      <td>13028.0</td>\n",
       "      <td>1063.0</td>\n",
       "      <td>13028.0</td>\n",
       "      <td>1063.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4410.0</td>\n",
       "      <td>445.0</td>\n",
       "      <td>4410.0</td>\n",
       "      <td>445.0</td>\n",
       "      <td>4410.0</td>\n",
       "      <td>445.0</td>\n",
       "      <td>4410.0</td>\n",
       "      <td>445.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1926.0</td>\n",
       "      <td>202.0</td>\n",
       "      <td>1926.0</td>\n",
       "      <td>202.0</td>\n",
       "      <td>1926.0</td>\n",
       "      <td>202.0</td>\n",
       "      <td>1926.0</td>\n",
       "      <td>202.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>303.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>303.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>303.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>303.0</td>\n",
       "      <td>27.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>37.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>37.0</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>9.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>9.0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         family_status         income_bracket         purpose_category  \\\n",
       "debt                 0       1              0       1                0   \n",
       "children                                                                 \n",
       "0              13028.0  1063.0        13028.0  1063.0          13028.0   \n",
       "1               4410.0   445.0         4410.0   445.0           4410.0   \n",
       "2               1926.0   202.0         1926.0   202.0           1926.0   \n",
       "3                303.0    27.0          303.0    27.0            303.0   \n",
       "4                 37.0     4.0           37.0     4.0             37.0   \n",
       "5                  9.0     NaN            9.0     NaN              9.0   \n",
       "\n",
       "                 total_income          \n",
       "debt           1            0       1  \n",
       "children                               \n",
       "0         1063.0      13028.0  1063.0  \n",
       "1          445.0       4410.0   445.0  \n",
       "2          202.0       1926.0   202.0  \n",
       "3           27.0        303.0    27.0  \n",
       "4            4.0         37.0     4.0  \n",
       "5            NaN          9.0     NaN  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credit_scoring_pivot_children = credit_scoring.pivot_table(index = [\"children\"], columns= [\"debt\"], aggfunc = \"count\")\n",
    "credit_scoring_pivot_children"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The numbers show that the more kids the family has the less likely they will default on their debt. I do have NaN for families with 5 children but I can tell already from the rest of the data in the area of 1 to 3 children that the numbers of default on debt go lower and lower. The average of people who end up defaulting on the loan is around the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "children\n",
       "0    14091\n",
       "1     4855\n",
       "2     2128\n",
       "3      330\n",
       "4       41\n",
       "5        9\n",
       "Name: debt, dtype: int64"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# credit_scoring.groupby('children')['debt'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "children\n",
       "0    0.075438\n",
       "1    0.091658\n",
       "2    0.094925\n",
       "3    0.081818\n",
       "4    0.097561\n",
       "5    0.000000\n",
       "Name: debt, dtype: float64"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credit_scoring.groupby('children')['debt'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Is there a relation between marital status and repaying a loan on time?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">children</th>\n",
       "      <th colspan=\"2\" halign=\"left\">income_bracket</th>\n",
       "      <th colspan=\"2\" halign=\"left\">purpose_category</th>\n",
       "      <th colspan=\"2\" halign=\"left\">total_income</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>debt</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>family_status</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>civil partnership</td>\n",
       "      <td>3763</td>\n",
       "      <td>388</td>\n",
       "      <td>3763</td>\n",
       "      <td>388</td>\n",
       "      <td>3763</td>\n",
       "      <td>388</td>\n",
       "      <td>3763</td>\n",
       "      <td>388</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>divorced</td>\n",
       "      <td>1110</td>\n",
       "      <td>85</td>\n",
       "      <td>1110</td>\n",
       "      <td>85</td>\n",
       "      <td>1110</td>\n",
       "      <td>85</td>\n",
       "      <td>1110</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>married</td>\n",
       "      <td>11408</td>\n",
       "      <td>931</td>\n",
       "      <td>11408</td>\n",
       "      <td>931</td>\n",
       "      <td>11408</td>\n",
       "      <td>931</td>\n",
       "      <td>11408</td>\n",
       "      <td>931</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>unmarried</td>\n",
       "      <td>2536</td>\n",
       "      <td>274</td>\n",
       "      <td>2536</td>\n",
       "      <td>274</td>\n",
       "      <td>2536</td>\n",
       "      <td>274</td>\n",
       "      <td>2536</td>\n",
       "      <td>274</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>widow / widower</td>\n",
       "      <td>896</td>\n",
       "      <td>63</td>\n",
       "      <td>896</td>\n",
       "      <td>63</td>\n",
       "      <td>896</td>\n",
       "      <td>63</td>\n",
       "      <td>896</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  children      income_bracket      purpose_category       \\\n",
       "debt                     0    1              0    1                0    1   \n",
       "family_status                                                               \n",
       "civil partnership     3763  388           3763  388             3763  388   \n",
       "divorced              1110   85           1110   85             1110   85   \n",
       "married              11408  931          11408  931            11408  931   \n",
       "unmarried             2536  274           2536  274             2536  274   \n",
       "widow / widower        896   63            896   63              896   63   \n",
       "\n",
       "                  total_income       \n",
       "debt                         0    1  \n",
       "family_status                        \n",
       "civil partnership         3763  388  \n",
       "divorced                  1110   85  \n",
       "married                  11408  931  \n",
       "unmarried                 2536  274  \n",
       "widow / widower            896   63  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credit_scoring_pivot_family = credit_scoring.pivot_table(index = [\"family_status\"], columns = [\"debt\"], aggfunc = \"count\")\n",
    "credit_scoring_pivot_family"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "family_status\n",
       "civil partnership    0.093471\n",
       "divorced             0.071130\n",
       "married              0.075452\n",
       "unmarried            0.097509\n",
       "widow / widower      0.065693\n",
       "Name: debt, dtype: float64"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credit_scoring.groupby('family_status')['debt'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to the table most of the people are married and they have the higher number of defaults on loans. The average is around the same for all types of family status. Most people are married and have the most defaults on loans but it is not so different from the rate of people defaulting if not married."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Is there a relation between income level and repaying a loan on time?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">children</th>\n",
       "      <th colspan=\"2\" halign=\"left\">family_status</th>\n",
       "      <th colspan=\"2\" halign=\"left\">purpose_category</th>\n",
       "      <th colspan=\"2\" halign=\"left\">total_income</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>debt</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>income_bracket</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>high income</td>\n",
       "      <td>9890</td>\n",
       "      <td>872</td>\n",
       "      <td>9890</td>\n",
       "      <td>872</td>\n",
       "      <td>9890</td>\n",
       "      <td>872</td>\n",
       "      <td>9890</td>\n",
       "      <td>872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>low income</td>\n",
       "      <td>4893</td>\n",
       "      <td>417</td>\n",
       "      <td>4893</td>\n",
       "      <td>417</td>\n",
       "      <td>4893</td>\n",
       "      <td>417</td>\n",
       "      <td>4893</td>\n",
       "      <td>417</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>mid income</td>\n",
       "      <td>4930</td>\n",
       "      <td>452</td>\n",
       "      <td>4930</td>\n",
       "      <td>452</td>\n",
       "      <td>4930</td>\n",
       "      <td>452</td>\n",
       "      <td>4930</td>\n",
       "      <td>452</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               children      family_status      purpose_category       \\\n",
       "debt                  0    1             0    1                0    1   \n",
       "income_bracket                                                          \n",
       "high income        9890  872          9890  872             9890  872   \n",
       "low income         4893  417          4893  417             4893  417   \n",
       "mid income         4930  452          4930  452             4930  452   \n",
       "\n",
       "               total_income       \n",
       "debt                      0    1  \n",
       "income_bracket                    \n",
       "high income            9890  872  \n",
       "low income             4893  417  \n",
       "mid income             4930  452  "
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credit_scoring_pivot_income = credit_scoring.pivot_table(index = [\"income_bracket\"], columns = [\"debt\"], aggfunc = \"count\")\n",
    "credit_scoring_pivot_income"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "income_bracket\n",
       "high income    0.081026\n",
       "low income     0.078531\n",
       "mid income     0.083984\n",
       "Name: debt, dtype: float64"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credit_scoring.groupby('income_bracket')['debt'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is no relation. The numbers show that there will almost always be the same number of people defaulting on their loan no matter the bracket of income they are in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- How do different loan purposes affect on-time repayment of the loan?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">children</th>\n",
       "      <th colspan=\"2\" halign=\"left\">family_status</th>\n",
       "      <th colspan=\"2\" halign=\"left\">income_bracket</th>\n",
       "      <th colspan=\"2\" halign=\"left\">total_income</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>debt</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>purpose_category</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>car</td>\n",
       "      <td>3903</td>\n",
       "      <td>403</td>\n",
       "      <td>3903</td>\n",
       "      <td>403</td>\n",
       "      <td>3903</td>\n",
       "      <td>403</td>\n",
       "      <td>3903</td>\n",
       "      <td>403</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>education</td>\n",
       "      <td>3643</td>\n",
       "      <td>370</td>\n",
       "      <td>3643</td>\n",
       "      <td>370</td>\n",
       "      <td>3643</td>\n",
       "      <td>370</td>\n",
       "      <td>3643</td>\n",
       "      <td>370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>property</td>\n",
       "      <td>10029</td>\n",
       "      <td>782</td>\n",
       "      <td>10029</td>\n",
       "      <td>782</td>\n",
       "      <td>10029</td>\n",
       "      <td>782</td>\n",
       "      <td>10029</td>\n",
       "      <td>782</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>wedding</td>\n",
       "      <td>2138</td>\n",
       "      <td>186</td>\n",
       "      <td>2138</td>\n",
       "      <td>186</td>\n",
       "      <td>2138</td>\n",
       "      <td>186</td>\n",
       "      <td>2138</td>\n",
       "      <td>186</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 children      family_status      income_bracket       \\\n",
       "debt                    0    1             0    1              0    1   \n",
       "purpose_category                                                        \n",
       "car                  3903  403          3903  403           3903  403   \n",
       "education            3643  370          3643  370           3643  370   \n",
       "property            10029  782         10029  782          10029  782   \n",
       "wedding              2138  186          2138  186           2138  186   \n",
       "\n",
       "                 total_income       \n",
       "debt                        0    1  \n",
       "purpose_category                    \n",
       "car                      3903  403  \n",
       "education                3643  370  \n",
       "property                10029  782  \n",
       "wedding                  2138  186  "
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credit_scoring_pivot_purpose = credit_scoring.pivot_table(index = [\"purpose_category\"], columns = [\"debt\"], aggfunc = \"count\")\n",
    "credit_scoring_pivot_purpose"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "purpose_category\n",
       "property     10811\n",
       "car           4306\n",
       "education     4013\n",
       "wedding       2324\n",
       "Name: debt, dtype: int64"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# credit_scoring.groupby('purpose_category')['debt'].count().sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "purpose_category\n",
       "car          0.093590\n",
       "education    0.092200\n",
       "wedding      0.080034\n",
       "property     0.072334\n",
       "Name: debt, dtype: float64"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credit_scoring.groupby('purpose_category')['debt'].mean().sort_values(ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again it seems to be that the numbers are higher for not paying back debt but the numbers of people with that purpose are higher as well. I do not see a correlation between different loan purposes and on-time repayments. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4. General conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After going through the data the credit scoring of individuals is not adversely affected by having more children. On the contrary there are less loans given the more children you have. And the purpose for the loan or your marital status does not necessarily affect your payment back for the loan either. The numbers of defaulting on loans are similar with each category. I would think that checking the age differences and life experience will be more relevant to see if their is a credit worthiness for the individual. Some of the data needed to be manipulated to fill missing values and this may have affected the total_income of individuals but I deemed it not significant enough."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Project Readiness Checklist\n",
    "\n",
    "Put 'x' in the completed points. Then press Shift + Enter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [x]  file open;\n",
    "- [x]  file examined;\n",
    "- [x]  missing values defined;\n",
    "- [x]  missing values are filled;\n",
    "- [x]  an explanation of which missing value types were detected;\n",
    "- [ ]  explanation for the possible causes of missing values;\n",
    "- [x]  an explanation of how the blanks are filled;\n",
    "- [x]  replaced the real data type with an integer;\n",
    "- [x]  an explanation of which method is used to change the data type and why;\n",
    "- [x]  duplicates deleted;\n",
    "- [x]  an explanation of which method is used to find and remove duplicates;\n",
    "- [x]  description of the possible reasons for the appearance of duplicates in the data;\n",
    "- [x]  data is categorized;\n",
    "- [x]  an explanation of the principle of data categorization;\n",
    "- [x]  an answer to the question \"Is there a relation between having kids and repaying a loan on time?\";\n",
    "- [x]  an answer to the question \" Is there a relation between marital status and repaying a loan on time?\";\n",
    "- [x]   an answer to the question \" Is there a relation between income level and repaying a loan on time?\";\n",
    "- [x]  an answer to the question \" How do different loan purposes affect on-time repayment of the loan?\"\n",
    "- [x]  conclusions are present on each stage;\n",
    "- [x]  a general conclusion is made."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}