tutorials/attrition/airbender_attrition_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Airbender Tutorial 2: Employee Attrition Analysis\n",
    "\n",
    "This overview will guide you through how to orchestrate your own machine learning experiments with the airbender tool using a more sophisticated experiment with employee attrition.\n",
    "\n",
    "Author: Sam Showalter\n",
    "\n",
    "First, let's bring in general imports"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes\n",
    "\n",
    "A few quick notes on the format of the configuration\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/sshowalter/repos/airbender/tutorials/attrition\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "import sys\n",
    "print(os.getcwd())\n",
    "sys.path.append(\"../../\")\n",
    "\n",
    "#Airflow DAG specific imports\n",
    "from airbender.dag.layers import DagLayer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Sources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data_sources = [DagLayer({'./airbender_attrition_demo.csv': \n",
    "               {pd.read_csv:None}})]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## EDA\n",
    "\n",
    "EDA has already been done for this dataset, but feel free to analyze it yourself by reading it in using the link above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "#EDA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import *\n",
    "from airbender.static.preprocessing import *\n",
    "\n",
    "preprocessing = { \n",
    "                    'cleansing': DagLayer({'median_impute': {impute : {'method': \"median\"}}})        \n",
    "                }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Engineering\n",
    "\n",
    "If you want to preprocess several columns in the same way but in separate tasks, you may do so by adding all of the names as a `tuple`. **Do not** use a list here as lists are unhashable and will through an error in the configuration. Functionality allowing for bulk data operations across multiple columns is coming shortly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import *\n",
    "from airbender.static.feature_engineering import *\n",
    "from scipy.stats import *\n",
    "\n",
    "feature_engineering = {\n",
    "                            'transformations': DagLayer({\n",
    "                                    ('Department', 'EducationField', 'JobRole', \"MaritalStatus\"): {pd.get_dummies: None},\n",
    "                                    'BusinessTravel': {create_ordinal_df: {'ordinal_dict':{\"Non-Travel\" :0, \n",
    "                                                                                            \"Travel_Rarely\":1, \n",
    "                                                                                            \"Travel_Frequently\": 2}}},\n",
    "                                    ('Attrition', 'Gender', 'OverTime') : {create_boolean_df: \n",
    "                                                                           {'boolean_names_and_values':\n",
    "                                                                            {\"Attrition\": [\"Yes\", \"No\"],\n",
    "                                                                            \"Gender\": [\"Male\", \"Female\"],\n",
    "                                                                            \"OverTime\": [\"Yes\", \"No\"]}}},\n",
    "                                    ('Age', 'DistanceFromHome', \"EnvironmentSatisfaction\",\n",
    "                                    'JobInvolvement', 'JobSatisfaction', \"MonthlyIncome\", \n",
    "                                     \"PerformanceRating\",\n",
    "                                    'RelationshipSatisfaction', 'TotalWorkingYears',\n",
    "                                    'TrainingTimesLastYear', 'WorkLifeBalance', 'YearsAtCompany',\n",
    "                                    'YearsInCurrentRole', 'YearsSinceLastPromotion',\n",
    "                                    'YearsWithCurrManager') : {normalize_values: None},\n",
    "                                \n",
    "                                    ('NumCompaniesWorked', 'PercentSalaryHike'): \n",
    "                                                    {winsorize: {'limits': [0.05, 0.05]},\n",
    "                                                    normalize_values: None},\n",
    "                                    'MonthlyRate': {winsorize: {'limits': [0.05, 0.05]},\n",
    "                                                    linear_transformation: {'method': boxcox},\n",
    "                                                    normalize_values: None},\n",
    "                               })\n",
    "                            \n",
    "                           }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from airbender.static.splitting import *\n",
    "\n",
    "splitting = DagLayer({'split': {train_test_split  : {'target': \"Attrition\",\n",
    "                                                                        'test_ratio': 0.2,\n",
    "                                                                         'random_state': 42}}})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modeling\n",
    "\n",
    "Here, we analyze the effectiveness of five models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.neural_network import MLPClassifier\n",
    "\n",
    "modeling = {'modeling': DagLayer({'LOG': {LogisticRegression: None}, \n",
    "                               'GNB': {GaussianNB: None},\n",
    "                               'RF':{RandomForestClassifier: {'n_estimators': 10}},\n",
    "                               'MLP': {MLPClassifier: {'activation': 'tanh'}},\n",
    "                               'SVM': {SVC: {'kernel': 'linear'}},\n",
    "                               'KNN': {KNeighborsClassifier: {'n_neighbors':3 }}})}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Expects predictions\n",
    "evaluation = {'metrics' : DagLayer({'acc': {accuracy_score: None}, \n",
    "                                    'f1': {f1_score: None}, \n",
    "                                    'precision': {precision_score: None},\n",
    "                                    'conf_matrix': {confusion_matrix: None}, \n",
    "                                    'class_report': {classification_report: None},\n",
    "                                    'recall': {recall_score :None}})}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Consolidate all Configurations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "attrition_config = \\\n",
    "{   \n",
    "    'modeling': modeling,\n",
    " \n",
    "    'data_sources': data_sources,\n",
    "\n",
    "    'splitting': splitting,\n",
    " \n",
    "    'preprocessing': preprocessing,\n",
    "    \n",
    "    'feature_engineering': feature_engineering,\n",
    " \n",
    "    'evaluation' : evaluation\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate as Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Import DAG generator\n",
    "from airbender.dag.generator import DagGenerator\n",
    "\n",
    "airbender_config = { \n",
    "                        'dag_name': \"Airbender_Attrition_Tutorial\",\n",
    "                        'dag':      {\n",
    "                                        'owner': 'airbender',\n",
    "                                        # 'email': [<EMAIL>, <EMAIL>, ...],\n",
    "                                        # 'op_args':{},\n",
    "                                        # 'op_kwargs': {},\n",
    "                                    },\n",
    "                        'config' : attrition_config\n",
    "                   }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the DAG Generator with your arguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Displaying Ordered Dag Layers with Tags:\n",
      "\n",
      "0 ['data_sources']\n",
      "1 ['splitting']\n",
      "2 ['preprocessing', 'cleansing']\n",
      "3 ['feature_engineering', 'transformations']\n",
      "4 ['modeling', 'modeling']\n",
      "5 ['evaluation', 'metrics']\n",
      "\n",
      "Generated airflow file with name: Airbender_Attrition_Tutorial_airbender_10-28-2019--13.38.03.py\n"
     ]
    }
   ],
   "source": [
    "dg = DagGenerator(airbender_config)\n",
    "dg.generate_file()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}