03-Data-Preparation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing and Preparing Source Data\n",
    "- Download data from the Internet\n",
    "- Profile the data for initial understanding and to identify flaws\n",
    "- Address any issues and prepare for Exploratory Data Analysis (EDA)\n",
    "- Prepare data for Machine Learning model development"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Required Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from sklearn.preprocessing import LabelEncoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "source_url = 'https://community.watsonanalytics.com/wp-content/uploads/2015/03/WA_Fn-UseC_-Telco-Customer-Churn.csv';\n",
    "data_dir = os.path.join(os.getcwd(), 'Data')\n",
    "\n",
    "# Read the source from the Web into a Pandas DataFrame.\n",
    "df = pd.read_csv(source_url, header=0)\n",
    "\n",
    "# Save a local copy of the raw source data file.\n",
    "raw_file = os.path.join(data_dir, 'WA-Telco-Customer-Churn-Raw.csv')\n",
    "df.to_csv(raw_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Profile the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Display the first 5 observations\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ensure we've identified all the features.\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inspect the unique values for each feature: looking for Nulls, NaNs, etc.\n",
    "print(\"------------------------------------------------------------\")\n",
    "print(\"Number of Observations:\", len(df))\n",
    "print(\"Number of Unique Customers:\", len(df.customerID.unique()))\n",
    "print(\"------------------------------------------------------------\")\n",
    "print(\"Gender:\", df.gender.unique())\n",
    "print(\"SeniorCitizen:\", df.SeniorCitizen.unique())\n",
    "print(\"Partner:\", df.Partner.unique())\n",
    "print(\"Dependents:\", df.Dependents.unique())\n",
    "print(\"Tenure:\", df.tenure.unique())\n",
    "print(\"PhoneService:\", df.PhoneService.unique())\n",
    "print(\"MultipleLines:\", df.MultipleLines.unique())\n",
    "print(\"InternetService:\", df.InternetService.unique())\n",
    "print(\"OnlineSecurity:\", df.OnlineSecurity.unique())\n",
    "print(\"OnlineBackup:\", df.OnlineBackup.unique())\n",
    "print(\"DeviceProtection:\", df.DeviceProtection.unique())\n",
    "print(\"TechSupport\", df.TechSupport.unique())\n",
    "print(\"StreamingTV\", df.StreamingTV.unique())\n",
    "print(\"StreamingMovies\", df.StreamingMovies.unique())\n",
    "print(\"Contract\", df.Contract.unique())\n",
    "print(\"PaperlessBilling\", df.PaperlessBilling.unique())\n",
    "print(\"PaymentMethod\", df.PaymentMethod.unique())\n",
    "print(\"MonthlyCharges\", df.MonthlyCharges.unique())\n",
    "print(\"TotalCharges\", df.TotalCharges.unique())\n",
    "print(\"Churn\", df.Churn.unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Check for Any Null or NaN Observations in the 2 Continuous Numeric Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.MonthlyCharges.isnull().any())\n",
    "print(df.TotalCharges.isnull().any())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare for Exploratory Data Analysis\n",
    "- Textual Labels Produce More Readable Data Visualizations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Convert SeniorCitizen to a textual feature.\n",
    "df.loc[df.SeniorCitizen==1, ('SeniorCitizen')] = 'Yes'\n",
    "df.loc[df.SeniorCitizen==0, ('SeniorCitizen')] = 'No'\n",
    "\n",
    "# Replace NaN with zero for those observations\n",
    "df.TotalCharges.fillna(0, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save the Readable Feature-Set for EDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dest_file = os.path.join(data_dir, 'WA-Telco-Customer-Churn-EDA.xlsx')\n",
    "df.to_excel(dest_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Prepare for Machine Learning\n",
    "- Machine Learning Algorithms Cannot Operate on Textual Data \n",
    "- Use LabelEncoding to Convert Textual Features into Numeric Values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "le = LabelEncoder()\n",
    "df.gender = le.fit_transform(df.gender)\n",
    "df.SeniorCitizen = le.fit_transform(df.SeniorCitizen)\n",
    "df.Partner = le.fit_transform(df.Partner)\n",
    "df.Dependents = le.fit_transform(df.Dependents)\n",
    "df.PhoneService = le.fit_transform(df.PhoneService)\n",
    "df.MultipleLines = le.fit_transform(df.MultipleLines)\n",
    "df.InternetService = le.fit_transform(df.InternetService)\n",
    "df.OnlineSecurity = le.fit_transform(df.OnlineSecurity)\n",
    "df.OnlineBackup = le.fit_transform(df.OnlineBackup)\n",
    "df.DeviceProtection = le.fit_transform(df.DeviceProtection)\n",
    "df.TechSupport = le.fit_transform(df.TechSupport)\n",
    "df.StreamingTV = le.fit_transform(df.StreamingTV)\n",
    "df.StreamingMovies = le.fit_transform(df.StreamingMovies)\n",
    "df.Contract = le.fit_transform(df.Contract)\n",
    "df.PaperlessBilling = le.fit_transform(df.PaperlessBilling)\n",
    "df.PaymentMethod = le.fit_transform(df.PaymentMethod)\n",
    "df.Churn = le.fit_transform(df.Churn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Make Appropriate Data Type Assignments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.gender = df.gender.astype('category')\n",
    "df.SeniorCitizen = df.SeniorCitizen.astype('category')\n",
    "df.Partner = df.Partner.astype('category')\n",
    "df.Dependents = df.Dependents.astype('category')\n",
    "df.tenure = pd.to_numeric(df.tenure, downcast='integer', errors='coerce')\n",
    "df.PhoneService = df.PhoneService.astype('category')\n",
    "df.MultipleLines = df.MultipleLines.astype('category')\n",
    "df.InternetService = df.InternetService.astype('category')\n",
    "df.OnlineSecurity = df.OnlineSecurity.astype('category')\n",
    "df.OnlineBackup = df.OnlineBackup.astype('category')\n",
    "df.DeviceProtection = df.DeviceProtection.astype('category')\n",
    "df.TechSupport = df.TechSupport.astype('category')\n",
    "df.StreamingTV = df.StreamingTV.astype('category')\n",
    "df.StreamingMovies = df.StreamingMovies.astype('category')\n",
    "df.Contract = df.Contract.astype('category')\n",
    "df.PaperlessBilling = df.PaperlessBilling.astype('category')\n",
    "df.PaymentMethod = df.PaymentMethod.astype('category')\n",
    "df.MonthlyCharges = pd.to_numeric(df.MonthlyCharges, downcast='float', errors='coerce')\n",
    "df.TotalCharges = pd.to_numeric(df.TotalCharges, downcast='float', errors='coerce')\n",
    "df.Churn = df.Churn.astype('category')\n",
    "\n",
    "# Replace NaN with zero for those observations\n",
    "df.TotalCharges.fillna(0, inplace=True)\n",
    "\n",
    "# Validate new data type assignments.\n",
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Inspect Converted Feature-Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#### Save Converted Features for Machine Learning Modeling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dest_file = os.path.join(data_dir, 'WA-Telco-Customer-Churn-ML.xlsx')\n",
    "df.to_excel(dest_file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}