featuretools_v1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fun with Featuretools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example [Featuretools](https://www.featuretools.com/) a python frame for automated feature engineering is used to generate a wide range of features for machine learning. The aim of the Notebook is to show how the framework is used and how it could support ML workflow, but should not be considered a tutorial on Machine Learning or model fine tuning.\n",
    "\n",
    "One of the main benefits of the framework is the so-called deep feature synthesis (dfs), which used relations in the data for automatically generating features, this could be, obviously depending on your data, features like mean session length, average purchasing price, or standard deviation of time since the last session. The term deep comes refers to stacking generated features on top of each other. \n",
    "\n",
    "The main building block of Featuretools is feature primitives and can be represented in two main groups: aggregation and transformation primitive. As the name indicates, aggregation primitives take related inputs and return a single output, similar to group by aggregations known from sql. Transformation primitives, on the other hand, are converting, eg. timestamps into data, hours, minutes, seconds, etc, or calculating differences in certain events. The full list of possible primitives can be found [here](https://primitives.featurelabs.com/).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import featuretools as ft\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "from featuretools.selection import (\n",
    "    remove_highly_null_features,\n",
    "    remove_single_value_features,\n",
    ")\n",
    "\n",
    "from woodwork.logical_types import Categorical, Boolean, Datetime, Double\n",
    "from datetime import datetime\n",
    "from dateutil.relativedelta import relativedelta\n",
    "from uuid import uuid4\n",
    "from functools import reduce\n",
    "from dataclasses import dataclass\n",
    "\n",
    "from sklearn.feature_selection import SelectFromModel, VarianceThreshold\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "from sklearn import svm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an example we are using the classic Telco Customer Churn dataset which could be found on [Kaggle](https://www.kaggle.com/datasets/blastchar/telco-customer-churn?select=WA_Fn-UseC_-Telco-Customer-Churn.csv), also the description and column definition could be found there, IBM provide a csv in their Githup repo which could be directly sourced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "URL_TO_DATA = \"https://raw.githubusercontent.com/IBM/telco-customer-churn-on-icp4d/master/data/Telco-Customer-Churn.csv\"\n",
    "BACK_COUNT_DATE = datetime.fromisoformat(\"2022-01-01\")\n",
    "TEST_SIZE = 0.2\n",
    "VALID_SIZE = 0.25\n",
    "RANDOM_STATE = 42\n",
    "NUMERIC_TRANSFORMER_REPLACEMENT = \"median\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cols_by_type(\n",
    "    in_df: pd.DataFrame, types_to_search: list, exclude_ids: Boolean = True\n",
    ") -> list:\n",
    "    \"\"\"Small helper function selecting columns according to their type.\"\"\"\n",
    "    cols = in_df.select_dtypes(include=types_to_search).columns.to_list()\n",
    "\n",
    "    if exclude_ids:\n",
    "        return list(filter(lambda x: not x.endswith(\"ID\"), cols))\n",
    "\n",
    "    return cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# in case of CERTIFICATE_VERIFY_FAILED run Install Certificates.command\n",
    "# see also https://stackoverflow.com/questions/50236117/scraping-ssl-certificate-verify-failed-error-for-http-en-wikipedia-org\n",
    "df = pd.read_csv(filepath_or_buffer=URL_TO_DATA, index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since Featuretools heavily rely on relations within the data, we split up the data frame into three entities: Customers, Subscriptions, and Billing, and introduce IDs to represent the relationship between them. \n",
    "\n",
    "- customer_df: customer_id (PK), subscription_id (FK), billing_id (FK), gender, SeniorCitizen, Partner, Dependents, tenure, Churn\n",
    "- services_df: service_id (PK), service_name\n",
    "- subscription_df: subscription_id (PK), service_id, customer_id\n",
    "- billing_df: billing_id (PK), Contract, PaperlessBilling, PaymentMethod, MonthlyCharges, TotalCharges\n",
    "\n",
    "PK indicates primary keys and FK foreign keys.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# mapping between column names and Featuretools data types.\n",
    "CUSTOMERS = {\n",
    "    \"gender\": Boolean,\n",
    "    \"SeniorCitizen\": Boolean,\n",
    "    \"Partner\": Boolean,\n",
    "    \"Dependents\": Boolean,\n",
    "}\n",
    "\n",
    "\n",
    "SUBSCRIPTIONS = {\n",
    "    \"PhoneService\": Categorical,\n",
    "    \"MultipleLines\": Boolean,\n",
    "    \"InternetService\": Categorical,\n",
    "    \"OnlineSecurity\": Categorical,\n",
    "    \"OnlineBackup\": Categorical,\n",
    "    \"DeviceProtection\": Categorical,\n",
    "    \"TechSupport\": Categorical,\n",
    "    \"StreamingTV\": Categorical,\n",
    "    \"StreamingMovies\": Categorical,\n",
    "}\n",
    "\n",
    "\n",
    "BILLINGS = {\n",
    "    \"tenure\": Double,\n",
    "    \"Contract\": Categorical,\n",
    "    \"PaperlessBilling\": Boolean,\n",
    "    \"PaymentMethod\": Categorical,\n",
    "    \"MonthlyCharges\": Double,\n",
    "    \"TotalCharges\": Double,\n",
    "}\n",
    "\n",
    "\n",
    "@dataclass\n",
    "class EntitySetColumns:\n",
    "    \"\"\"Small helper class for organizing data frames and columns.\"\"\"\n",
    "\n",
    "    new_dataframe_name: str\n",
    "    index: str\n",
    "    additional_columns: list\n",
    "\n",
    "\n",
    "entity_set_columns = {\n",
    "    \"customers\": EntitySetColumns(\n",
    "        index=\"customerID\",\n",
    "        new_dataframe_name=None,\n",
    "        additional_columns=list(CUSTOMERS.keys()),\n",
    "    ),\n",
    "    \"subscriptions\": EntitySetColumns(\n",
    "        index=\"subscriptionID\",\n",
    "        new_dataframe_name=\"subscriptions\",\n",
    "        additional_columns=list(SUBSCRIPTIONS.keys()),\n",
    "    ),\n",
    "    \"billings\": EntitySetColumns(\n",
    "        index=\"billingID\",\n",
    "        new_dataframe_name=\"billings\",\n",
    "        additional_columns=list(BILLINGS.keys()),\n",
    "    ),\n",
    "}\n",
    "\n",
    "# adding PK's to the data\n",
    "df[\"customerID\"] = df.index\n",
    "df[\"billingID\"] = [str(uuid4()) for _ in range(df.shape[0])]\n",
    "df[\"subscriptionID\"] = [str(uuid4()) for _ in range(df.shape[0])]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Further we need to clean the data a bit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert total charges to numeric\n",
    "df[\"TotalCharges\"] = pd.to_numeric(df[\"TotalCharges\"], errors=\"coerce\")\n",
    "\n",
    "# generate syntetic time index\n",
    "df[\"ContractStartDate\"] = list(\n",
    "    map(\n",
    "        lambda tenure, dat=BACK_COUNT_DATE: dat - relativedelta(months=-tenure),\n",
    "        df[\"tenure\"],\n",
    "    )\n",
    ")\n",
    "\n",
    "# convert to 1/0\n",
    "df[\"Churn\"] = np.where(df[\"Churn\"] == \"Yes\", 1, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since not some features require domain knowledge, not all features could be generated automatically, for example, categorize the length of a contract or how many products the user subscribed to, which could not be determined automatically in our example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "# adding more features\n",
    "df[\"ShortContract\"] = df[\"Contract\"] == \"Month-to-month\"\n",
    "BILLINGS[\"ShortContract\"] = Boolean\n",
    "\n",
    "# remove because it depends on Phone service\n",
    "products = [n for n in list(SUBSCRIPTIONS.keys()) if n != \"MultipleLines\"]\n",
    "df[\"TotalProductCount\"] = (df[products] != \"No\").sum(axis=1)\n",
    "SUBSCRIPTIONS[\"TotalProductCount\"] = Double\n",
    "\n",
    "df[\"UserCategory\"] = pd.cut(\n",
    "    x=df[\"TotalProductCount\"], bins=[0, 3, 6, np.Inf], labels=[\"light\", \"mid\", \"heavy\"]\n",
    ")\n",
    "CUSTOMERS[\"UserCategory\"] = Categorical"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = df[\"Churn\"]\n",
    "X = df.drop(columns=[\"Churn\"])\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X, y, test_size=TEST_SIZE, random_state=RANDOM_STATE\n",
    ")\n",
    "X_train, X_val, y_train, y_val = train_test_split(\n",
    "    X_train, y_train, test_size=VALID_SIZE, random_state=RANDOM_STATE\n",
    ")  # 0.25 x 0.8 = 0.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For representing data frames and their relations, Featuretools uses entity sets. In our case, we represent the data in a parent-child relation, where the customer dataframe is the parent and billings and subscriptions are the children. By providing our primary keys as an index and having foreign keys present in the parent dataframe, the framework could set up the relations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def entityset_wrapper(\n",
    "    id: str,\n",
    "    data_frame_name: str,\n",
    "    feature_dataset: pd.DataFrame,\n",
    "    variable_types: dict,\n",
    "    entity_set_columns: dict,\n",
    ") -> ft.EntitySet:\n",
    "    \"\"\"Non generic wrapper for making ft.EntitySet\"\"\"\n",
    "    es = ft.EntitySet(id=id)\n",
    "\n",
    "    es.add_dataframe(\n",
    "        dataframe_name=data_frame_name,\n",
    "        index=entity_set_columns.get(\"customers\").index,\n",
    "        logical_types=variable_types,\n",
    "        dataframe=feature_dataset,\n",
    "    )\n",
    "\n",
    "    es.normalize_dataframe(\n",
    "        base_dataframe_name=data_frame_name,\n",
    "        new_dataframe_name=entity_set_columns.get(\"billings\").new_dataframe_name,\n",
    "        index=entity_set_columns.get(\"billings\").index,\n",
    "        additional_columns=entity_set_columns.get(\"billings\").additional_columns,\n",
    "    )\n",
    "\n",
    "    es.normalize_dataframe(\n",
    "        base_dataframe_name=data_frame_name,\n",
    "        new_dataframe_name=entity_set_columns.get(\"subscriptions\").new_dataframe_name,\n",
    "        index=entity_set_columns.get(\"subscriptions\").index,\n",
    "        additional_columns=entity_set_columns.get(\"subscriptions\").additional_columns,\n",
    "    )\n",
    "\n",
    "    return es"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<!-- Generated by graphviz version 5.0.1 (20220820.1526)\n -->\n<!-- Title: customers_train Pages: 1 -->\n<svg width=\"406pt\" height=\"423pt\"\n viewBox=\"0.00 0.00 406.00 423.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 419)\">\n<title>customers_train</title>\n<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-419 402,-419 402,4 -4,4\"/>\n<!-- customers -->\n<g id=\"node1\" class=\"node\">\n<title>customers</title>\n<polygon fill=\"none\" stroke=\"black\" points=\"77.5,-218.5 77.5,-414.5 311.5,-414.5 311.5,-218.5 77.5,-218.5\"/>\n<text text-anchor=\"middle\" x=\"194.5\" y=\"-399.3\" font-family=\"Times,serif\" font-size=\"14.00\">customers (4225 rows)</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"77.5,-391.5 311.5,-391.5\"/>\n<text text-anchor=\"start\" x=\"85.5\" y=\"-376.3\" font-family=\"Times,serif\" font-size=\"14.00\">gender : Boolean</text>\n<text text-anchor=\"start\" x=\"85.5\" y=\"-361.3\" font-family=\"Times,serif\" font-size=\"14.00\">SeniorCitizen : Boolean</text>\n<text text-anchor=\"start\" x=\"85.5\" y=\"-346.3\" font-family=\"Times,serif\" font-size=\"14.00\">Partner : Boolean</text>\n<text text-anchor=\"start\" x=\"85.5\" y=\"-331.3\" font-family=\"Times,serif\" font-size=\"14.00\">Dependents : Boolean</text>\n<text text-anchor=\"start\" x=\"85.5\" y=\"-316.3\" font-family=\"Times,serif\" font-size=\"14.00\">customerID : Unknown; index</text>\n<text text-anchor=\"start\" x=\"85.5\" y=\"-301.3\" font-family=\"Times,serif\" font-size=\"14.00\">billingID : Unknown; foreign_key</text>\n<text text-anchor=\"start\" x=\"85.5\" y=\"-286.3\" font-family=\"Times,serif\" font-size=\"14.00\">subscriptionID : Unknown; foreign_key</text>\n<text text-anchor=\"start\" x=\"85.5\" y=\"-271.3\" font-family=\"Times,serif\" font-size=\"14.00\">ContractStartDate : Datetime</text>\n<text text-anchor=\"start\" x=\"85.5\" y=\"-256.3\" font-family=\"Times,serif\" font-size=\"14.00\">ShortContract : Boolean</text>\n<text text-anchor=\"start\" x=\"85.5\" y=\"-241.3\" font-family=\"Times,serif\" font-size=\"14.00\">TotalProductCount : Double</text>\n<text text-anchor=\"start\" x=\"85.5\" y=\"-226.3\" font-family=\"Times,serif\" font-size=\"14.00\">UserCategory : Categorical</text>\n</g>\n<!-- billings -->\n<g id=\"node2\" class=\"node\">\n<title>billings</title>\n<polygon fill=\"none\" stroke=\"black\" points=\"0,-23 0,-159 181,-159 181,-23 0,-23\"/>\n<text text-anchor=\"middle\" x=\"90.5\" y=\"-143.8\" font-family=\"Times,serif\" font-size=\"14.00\">billings (4225 rows)</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"0,-136 181,-136\"/>\n<text text-anchor=\"start\" x=\"8\" y=\"-120.8\" font-family=\"Times,serif\" font-size=\"14.00\">billingID : Unknown; index</text>\n<text text-anchor=\"start\" x=\"8\" y=\"-105.8\" font-family=\"Times,serif\" font-size=\"14.00\">tenure : Double</text>\n<text text-anchor=\"start\" x=\"8\" y=\"-90.8\" font-family=\"Times,serif\" font-size=\"14.00\">Contract : Categorical</text>\n<text text-anchor=\"start\" x=\"8\" y=\"-75.8\" font-family=\"Times,serif\" font-size=\"14.00\">PaperlessBilling : Boolean</text>\n<text text-anchor=\"start\" x=\"8\" y=\"-60.8\" font-family=\"Times,serif\" font-size=\"14.00\">PaymentMethod : Categorical</text>\n<text text-anchor=\"start\" x=\"8\" y=\"-45.8\" font-family=\"Times,serif\" font-size=\"14.00\">MonthlyCharges : Double</text>\n<text text-anchor=\"start\" x=\"8\" y=\"-30.8\" font-family=\"Times,serif\" font-size=\"14.00\">TotalCharges : Double</text>\n</g>\n<!-- customers&#45;&gt;billings -->\n<g id=\"edge1\" class=\"edge\">\n<title>customers&#45;&gt;billings</title>\n<path fill=\"none\" stroke=\"black\" d=\"M129.25,-218.25C129.25,-218.25 129.25,-169.14 129.25,-169.14\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"132.75,-169.14 129.25,-159.14 125.75,-169.14 132.75,-169.14\"/>\n<text text-anchor=\"middle\" x=\"104.25\" y=\"-197.49\" font-family=\"Times,serif\" font-size=\"14.00\">billingID</text>\n</g>\n<!-- subscriptions -->\n<g id=\"node3\" class=\"node\">\n<title>subscriptions</title>\n<polygon fill=\"none\" stroke=\"black\" points=\"199,-0.5 199,-181.5 398,-181.5 398,-0.5 199,-0.5\"/>\n<text text-anchor=\"middle\" x=\"298.5\" y=\"-166.3\" font-family=\"Times,serif\" font-size=\"14.00\">subscriptions (4225 rows)</text>\n<polyline fill=\"none\" stroke=\"black\" points=\"199,-158.5 398,-158.5\"/>\n<text text-anchor=\"start\" x=\"207\" y=\"-143.3\" font-family=\"Times,serif\" font-size=\"14.00\">subscriptionID : Unknown; index</text>\n<text text-anchor=\"start\" x=\"207\" y=\"-128.3\" font-family=\"Times,serif\" font-size=\"14.00\">PhoneService : Categorical</text>\n<text text-anchor=\"start\" x=\"207\" y=\"-113.3\" font-family=\"Times,serif\" font-size=\"14.00\">MultipleLines : Boolean</text>\n<text text-anchor=\"start\" x=\"207\" y=\"-98.3\" font-family=\"Times,serif\" font-size=\"14.00\">InternetService : Categorical</text>\n<text text-anchor=\"start\" x=\"207\" y=\"-83.3\" font-family=\"Times,serif\" font-size=\"14.00\">OnlineSecurity : Categorical</text>\n<text text-anchor=\"start\" x=\"207\" y=\"-68.3\" font-family=\"Times,serif\" font-size=\"14.00\">OnlineBackup : Categorical</text>\n<text text-anchor=\"start\" x=\"207\" y=\"-53.3\" font-family=\"Times,serif\" font-size=\"14.00\">DeviceProtection : Categorical</text>\n<text text-anchor=\"start\" x=\"207\" y=\"-38.3\" font-family=\"Times,serif\" font-size=\"14.00\">TechSupport : Categorical</text>\n<text text-anchor=\"start\" x=\"207\" y=\"-23.3\" font-family=\"Times,serif\" font-size=\"14.00\">StreamingTV : Categorical</text>\n<text text-anchor=\"start\" x=\"207\" y=\"-8.3\" font-family=\"Times,serif\" font-size=\"14.00\">StreamingMovies : Categorical</text>\n</g>\n<!-- customers&#45;&gt;subscriptions -->\n<g id=\"edge2\" class=\"edge\">\n<title>customers&#45;&gt;subscriptions</title>\n<path fill=\"none\" stroke=\"black\" d=\"M255.25,-218.25C255.25,-218.25 255.25,-191.82 255.25,-191.82\"/>\n<polygon fill=\"black\" stroke=\"black\" points=\"258.75,-191.82 255.25,-181.82 251.75,-191.82 258.75,-191.82\"/>\n<text text-anchor=\"middle\" x=\"214.75\" y=\"-193.83\" font-family=\"Times,serif\" font-size=\"14.00\">subscriptionID</text>\n</g>\n</g>\n</svg>\n",
      "text/plain": [
       "<graphviz.graphs.Digraph at 0x1381ea220>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "es = entityset_wrapper(\n",
    "    id=\"customers_train\",\n",
    "    data_frame_name=\"customers\",\n",
    "    feature_dataset=X_train,\n",
    "    variable_types=CUSTOMERS | SUBSCRIPTIONS | BILLINGS,\n",
    "    entity_set_columns=entity_set_columns,\n",
    ")\n",
    "es.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the entity set in place, we can call the deep feature synthesis function to finally generate the features. For details on how the algorithm works, please refer to the paper of [Kanter and Veeramachaneni: Deep Feature Synthesis:\n",
    "Towards Automating Data Science Endeavors](https://dai.lids.mit.edu/wp-content/uploads/2017/10/DSAA_DSM_2015.pdf)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of the resulting feature matrix (4225, 118)\n"
     ]
    }
   ],
   "source": [
    "MAX_DEPTH = 2\n",
    "\n",
    "feature_matrix, feature_defs = ft.dfs(\n",
    "    entityset=es,\n",
    "    target_dataframe_name=\"customers\",\n",
    "    max_depth=MAX_DEPTH,\n",
    "    include_cutoff_time=False,\n",
    ")\n",
    "\n",
    "feature_matrix_enc, features_enc = ft.encode_features(feature_matrix, feature_defs)\n",
    "print(f\"Shape of the resulting feature matrix {feature_matrix_enc.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `max_depth` regulates the depth the generation step takes place, eg. depth one only existing features are used as input. Depth two takes the generated features from depth one and replays the algorithm to the results from depth one. \n",
    "\n",
    "The resulting feature encoded matrix contains 118 columns. After feature generation it's common to remove space features and features which do not contain a lot of information, to decrease the risk of overfitting and improve accuracy. In this example, we use two approaches to address the problem, first the build-in functions `remove_highly_null_features` and `remove_single_value_features` from feature tools and also a variance threshold (`VarianceThreshold`) method from sklearn. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Old shape vs new shape: (4225, 118) vs (4225, 114)\n",
      "Old shape vs new shape: (4225, 114) vs (4225, 82)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(4225, 82)"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "PCT_NULL_THRESHOLD = 0.85\n",
    "\n",
    "fm_wo_high_null, f_wo_high_null = remove_highly_null_features(\n",
    "    feature_matrix_enc, features=features_enc, pct_null_threshold=PCT_NULL_THRESHOLD\n",
    ")\n",
    "print(f\"Old shape vs new shape: {feature_matrix_enc.shape} vs {fm_wo_high_null.shape}\")\n",
    "\n",
    "fm_wo_sv, f_wo_sv = remove_single_value_features(\n",
    "    fm_wo_high_null, features=f_wo_high_null\n",
    ")\n",
    "print(f\"Old shape vs new shape: {fm_wo_high_null.shape} vs {fm_wo_sv.shape}\")\n",
    "\n",
    "X_train_feature_matrix = fm_wo_sv\n",
    "X_train_features = f_wo_sv\n",
    "\n",
    "X_train_feature_matrix.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the name indicates, `remove_highly_null_features ` removes all columns where the null share is above the threshold, in our case above 85%, in total four columns got removed. Additionally, we checked `remove_single_value_features ` which removes columns where all values are the same, but no additional column (of the remaining once) got removed.\n",
    "\n",
    "Column removal based on low variance is part of the sklearn pipeline, therefore will be done later. Other steps include NA replacements via `SimpleImputer` and scaling via `StandardScaler`, both steps are applied for all numeric features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-5 {color: black;background-color: white;}#sk-container-id-5 pre{padding: 0;}#sk-container-id-5 div.sk-toggleable {background-color: white;}#sk-container-id-5 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-5 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-5 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-5 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-5 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-5 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-5 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-5 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-5 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-5 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-5 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-5 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-5 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-5 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-5 div.sk-item {position: relative;z-index: 1;}#sk-container-id-5 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-5 div.sk-item::before, #sk-container-id-5 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-5 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-5 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-5 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-5 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-5 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-5 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-5 div.sk-label-container {text-align: center;}#sk-container-id-5 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-5 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-5\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(remainder=&#x27;passthrough&#x27;,\n",
       "                  transformers=[(&#x27;num&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;imputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;scaler&#x27;, StandardScaler())]),\n",
       "                                 [&#x27;TotalProductCount&#x27;, &#x27;billings.tenure&#x27;,\n",
       "                                  &#x27;billings.MonthlyCharges&#x27;,\n",
       "                                  &#x27;billings.TotalCharges&#x27;,\n",
       "                                  &#x27;billings.MAX(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;billings.MEAN(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;billings.MIN(cust...\n",
       "                                  &#x27;billings.SUM(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;subscriptions.MAX(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;subscriptions.MEAN(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;subscriptions.MIN(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;subscriptions.PERCENT_TRUE(customers.SeniorCitizen)&#x27;,\n",
       "                                  &#x27;subscriptions.PERCENT_TRUE(customers.ShortContract)&#x27;,\n",
       "                                  &#x27;subscriptions.SUM(customers.TotalProductCount)&#x27;])],\n",
       "                  verbose=True)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-29\" type=\"checkbox\" ><label for=\"sk-estimator-id-29\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(remainder=&#x27;passthrough&#x27;,\n",
       "                  transformers=[(&#x27;num&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;imputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;scaler&#x27;, StandardScaler())]),\n",
       "                                 [&#x27;TotalProductCount&#x27;, &#x27;billings.tenure&#x27;,\n",
       "                                  &#x27;billings.MonthlyCharges&#x27;,\n",
       "                                  &#x27;billings.TotalCharges&#x27;,\n",
       "                                  &#x27;billings.MAX(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;billings.MEAN(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;billings.MIN(cust...\n",
       "                                  &#x27;billings.SUM(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;subscriptions.MAX(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;subscriptions.MEAN(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;subscriptions.MIN(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;subscriptions.PERCENT_TRUE(customers.SeniorCitizen)&#x27;,\n",
       "                                  &#x27;subscriptions.PERCENT_TRUE(customers.ShortContract)&#x27;,\n",
       "                                  &#x27;subscriptions.SUM(customers.TotalProductCount)&#x27;])],\n",
       "                  verbose=True)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-30\" type=\"checkbox\" ><label for=\"sk-estimator-id-30\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">num</label><div class=\"sk-toggleable__content\"><pre>[&#x27;TotalProductCount&#x27;, &#x27;billings.tenure&#x27;, &#x27;billings.MonthlyCharges&#x27;, &#x27;billings.TotalCharges&#x27;, &#x27;billings.MAX(customers.TotalProductCount)&#x27;, &#x27;billings.MEAN(customers.TotalProductCount)&#x27;, &#x27;billings.MIN(customers.TotalProductCount)&#x27;, &#x27;billings.PERCENT_TRUE(customers.SeniorCitizen)&#x27;, &#x27;billings.PERCENT_TRUE(customers.ShortContract)&#x27;, &#x27;billings.SUM(customers.TotalProductCount)&#x27;, &#x27;subscriptions.MAX(customers.TotalProductCount)&#x27;, &#x27;subscriptions.MEAN(customers.TotalProductCount)&#x27;, &#x27;subscriptions.MIN(customers.TotalProductCount)&#x27;, &#x27;subscriptions.PERCENT_TRUE(customers.SeniorCitizen)&#x27;, &#x27;subscriptions.PERCENT_TRUE(customers.ShortContract)&#x27;, &#x27;subscriptions.SUM(customers.TotalProductCount)&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-31\" type=\"checkbox\" ><label for=\"sk-estimator-id-31\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-32\" type=\"checkbox\" ><label for=\"sk-estimator-id-32\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-33\" type=\"checkbox\" ><label for=\"sk-estimator-id-33\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">remainder</label><div class=\"sk-toggleable__content\"><pre></pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-34\" type=\"checkbox\" ><label for=\"sk-estimator-id-34\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(remainder='passthrough',\n",
       "                  transformers=[('num',\n",
       "                                 Pipeline(steps=[('imputer',\n",
       "                                                  SimpleImputer(strategy='median')),\n",
       "                                                 ('scaler', StandardScaler())]),\n",
       "                                 ['TotalProductCount', 'billings.tenure',\n",
       "                                  'billings.MonthlyCharges',\n",
       "                                  'billings.TotalCharges',\n",
       "                                  'billings.MAX(customers.TotalProductCount)',\n",
       "                                  'billings.MEAN(customers.TotalProductCount)',\n",
       "                                  'billings.MIN(cust...\n",
       "                                  'billings.SUM(customers.TotalProductCount)',\n",
       "                                  'subscriptions.MAX(customers.TotalProductCount)',\n",
       "                                  'subscriptions.MEAN(customers.TotalProductCount)',\n",
       "                                  'subscriptions.MIN(customers.TotalProductCount)',\n",
       "                                  'subscriptions.PERCENT_TRUE(customers.SeniorCitizen)',\n",
       "                                  'subscriptions.PERCENT_TRUE(customers.ShortContract)',\n",
       "                                  'subscriptions.SUM(customers.TotalProductCount)'])],\n",
       "                  verbose=True)"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fmt: off\n",
    "numeric_features_ft = get_cols_by_type(\n",
    "    in_df=X_train_feature_matrix,\n",
    "    types_to_search=[np.float64, np.int64],\n",
    "    exclude_ids=False,\n",
    ")\n",
    "\n",
    "numeric_transformer = Pipeline(\n",
    "    steps=[\n",
    "        (\"imputer\", SimpleImputer(missing_values=np.nan, strategy=NUMERIC_TRANSFORMER_REPLACEMENT)),\n",
    "        (\"scaler\", StandardScaler()),\n",
    "    ]\n",
    ")\n",
    "\n",
    "preprocessor = ColumnTransformer(\n",
    "    transformers=[(\"num\", numeric_transformer, numeric_features_ft),\n",
    "    ], verbose=True, remainder = \"passthrough\"\n",
    ")\n",
    "# fmt: on\n",
    "preprocessor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned at the beginning, the notebook mainly focuses on featertools, but to finish the task we are also gonna fit a Support Vector Machine classifier [SVC](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC) classifier. All the hyperparameter tuning, model selection, and how to improve prediction accuracy will be part of other notebooks. Therefore we mainly use the default settings here. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ColumnTransformer] ........... (1 of 2) Processing num, total=   0.0s\n",
      "[ColumnTransformer] ..... (2 of 2) Processing remainder, total=   0.0s\n",
      "[Pipeline] ...... (step 1 of 2) Processing preprocessor, total=   0.0s\n",
      "[Pipeline] ........ (step 2 of 2) Processing classifier, total=   0.6s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-6 {color: black;background-color: white;}#sk-container-id-6 pre{padding: 0;}#sk-container-id-6 div.sk-toggleable {background-color: white;}#sk-container-id-6 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-6 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-6 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-6 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-6 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-6 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-6 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-6 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-6 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-6 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-6 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-6 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-6 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-6 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-6 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-6 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-6 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-6 div.sk-item {position: relative;z-index: 1;}#sk-container-id-6 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-6 div.sk-item::before, #sk-container-id-6 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-6 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-6 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-6 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-6 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-6 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-6 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-6 div.sk-label-container {text-align: center;}#sk-container-id-6 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-6 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-6\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;preprocessor&#x27;,\n",
       "                 ColumnTransformer(remainder=&#x27;passthrough&#x27;,\n",
       "                                   transformers=[(&#x27;num&#x27;,\n",
       "                                                  Pipeline(steps=[(&#x27;imputer&#x27;,\n",
       "                                                                   SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                                  (&#x27;scaler&#x27;,\n",
       "                                                                   StandardScaler())]),\n",
       "                                                  [&#x27;TotalProductCount&#x27;,\n",
       "                                                   &#x27;billings.tenure&#x27;,\n",
       "                                                   &#x27;billings.MonthlyCharges&#x27;,\n",
       "                                                   &#x27;billings.TotalCharges&#x27;,\n",
       "                                                   &#x27;billings.MAX(customers.TotalProductCount)&#x27;,\n",
       "                                                   &#x27;billings.MEAN(customers.TotalP...\n",
       "                                                   &#x27;subscriptions.MAX(customers.TotalProductCount)&#x27;,\n",
       "                                                   &#x27;subscriptions.MEAN(customers.TotalProductCount)&#x27;,\n",
       "                                                   &#x27;subscriptions.MIN(customers.TotalProductCount)&#x27;,\n",
       "                                                   &#x27;subscriptions.PERCENT_TRUE(customers.SeniorCitizen)&#x27;,\n",
       "                                                   &#x27;subscriptions.PERCENT_TRUE(customers.ShortContract)&#x27;,\n",
       "                                                   &#x27;subscriptions.SUM(customers.TotalProductCount)&#x27;])],\n",
       "                                   verbose=True)),\n",
       "                (&#x27;classifier&#x27;, SVC())],\n",
       "         verbose=True)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-35\" type=\"checkbox\" ><label for=\"sk-estimator-id-35\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;preprocessor&#x27;,\n",
       "                 ColumnTransformer(remainder=&#x27;passthrough&#x27;,\n",
       "                                   transformers=[(&#x27;num&#x27;,\n",
       "                                                  Pipeline(steps=[(&#x27;imputer&#x27;,\n",
       "                                                                   SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                                  (&#x27;scaler&#x27;,\n",
       "                                                                   StandardScaler())]),\n",
       "                                                  [&#x27;TotalProductCount&#x27;,\n",
       "                                                   &#x27;billings.tenure&#x27;,\n",
       "                                                   &#x27;billings.MonthlyCharges&#x27;,\n",
       "                                                   &#x27;billings.TotalCharges&#x27;,\n",
       "                                                   &#x27;billings.MAX(customers.TotalProductCount)&#x27;,\n",
       "                                                   &#x27;billings.MEAN(customers.TotalP...\n",
       "                                                   &#x27;subscriptions.MAX(customers.TotalProductCount)&#x27;,\n",
       "                                                   &#x27;subscriptions.MEAN(customers.TotalProductCount)&#x27;,\n",
       "                                                   &#x27;subscriptions.MIN(customers.TotalProductCount)&#x27;,\n",
       "                                                   &#x27;subscriptions.PERCENT_TRUE(customers.SeniorCitizen)&#x27;,\n",
       "                                                   &#x27;subscriptions.PERCENT_TRUE(customers.ShortContract)&#x27;,\n",
       "                                                   &#x27;subscriptions.SUM(customers.TotalProductCount)&#x27;])],\n",
       "                                   verbose=True)),\n",
       "                (&#x27;classifier&#x27;, SVC())],\n",
       "         verbose=True)</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-36\" type=\"checkbox\" ><label for=\"sk-estimator-id-36\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">preprocessor: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(remainder=&#x27;passthrough&#x27;,\n",
       "                  transformers=[(&#x27;num&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;imputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;scaler&#x27;, StandardScaler())]),\n",
       "                                 [&#x27;TotalProductCount&#x27;, &#x27;billings.tenure&#x27;,\n",
       "                                  &#x27;billings.MonthlyCharges&#x27;,\n",
       "                                  &#x27;billings.TotalCharges&#x27;,\n",
       "                                  &#x27;billings.MAX(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;billings.MEAN(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;billings.MIN(cust...\n",
       "                                  &#x27;billings.SUM(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;subscriptions.MAX(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;subscriptions.MEAN(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;subscriptions.MIN(customers.TotalProductCount)&#x27;,\n",
       "                                  &#x27;subscriptions.PERCENT_TRUE(customers.SeniorCitizen)&#x27;,\n",
       "                                  &#x27;subscriptions.PERCENT_TRUE(customers.ShortContract)&#x27;,\n",
       "                                  &#x27;subscriptions.SUM(customers.TotalProductCount)&#x27;])],\n",
       "                  verbose=True)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-37\" type=\"checkbox\" ><label for=\"sk-estimator-id-37\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">num</label><div class=\"sk-toggleable__content\"><pre>[&#x27;TotalProductCount&#x27;, &#x27;billings.tenure&#x27;, &#x27;billings.MonthlyCharges&#x27;, &#x27;billings.TotalCharges&#x27;, &#x27;billings.MAX(customers.TotalProductCount)&#x27;, &#x27;billings.MEAN(customers.TotalProductCount)&#x27;, &#x27;billings.MIN(customers.TotalProductCount)&#x27;, &#x27;billings.PERCENT_TRUE(customers.SeniorCitizen)&#x27;, &#x27;billings.PERCENT_TRUE(customers.ShortContract)&#x27;, &#x27;billings.SUM(customers.TotalProductCount)&#x27;, &#x27;subscriptions.MAX(customers.TotalProductCount)&#x27;, &#x27;subscriptions.MEAN(customers.TotalProductCount)&#x27;, &#x27;subscriptions.MIN(customers.TotalProductCount)&#x27;, &#x27;subscriptions.PERCENT_TRUE(customers.SeniorCitizen)&#x27;, &#x27;subscriptions.PERCENT_TRUE(customers.ShortContract)&#x27;, &#x27;subscriptions.SUM(customers.TotalProductCount)&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-38\" type=\"checkbox\" ><label for=\"sk-estimator-id-38\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-39\" type=\"checkbox\" ><label for=\"sk-estimator-id-39\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-40\" type=\"checkbox\" ><label for=\"sk-estimator-id-40\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">remainder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;SeniorCitizen&#x27;, &#x27;ShortContract&#x27;, &#x27;UserCategory = mid&#x27;, &#x27;UserCategory = heavy&#x27;, &#x27;UserCategory = light&#x27;, &#x27;MONTH(ContractStartDate) = 2&#x27;, &#x27;MONTH(ContractStartDate) = 1&#x27;, &#x27;MONTH(ContractStartDate) = 4&#x27;, &#x27;MONTH(ContractStartDate) = 12&#x27;, &#x27;MONTH(ContractStartDate) = 9&#x27;, &#x27;MONTH(ContractStartDate) = 11&#x27;, &#x27;MONTH(ContractStartDate) = 3&#x27;, &#x27;MONTH(ContractStartDate) = 5&#x27;, &#x27;MONTH(ContractStartDate) = 6&#x27;, &#x27;MONTH(ContractStartDate) = 8&#x27;, &#x27;MONTH(ContractStartDate) is unknown&#x27;, &#x27;WEEKDAY(ContractStartDate) = 1&#x27;, &#x27;WEEKDAY(ContractStartDate) = 5&#x27;, &#x27;WEEKDAY(ContractStartDate) = 2&#x27;, &#x27;WEEKDAY(ContractStartDate) = 0&#x27;, &#x27;WEEKDAY(ContractStartDate) = 4&#x27;, &#x27;WEEKDAY(ContractStartDate) = 6&#x27;, &#x27;WEEKDAY(ContractStartDate) = 3&#x27;, &#x27;YEAR(ContractStartDate) = 2022&#x27;, &#x27;YEAR(ContractStartDate) = 2027&#x27;, &#x27;YEAR(ContractStartDate) = 2023&#x27;, &#x27;YEAR(ContractStartDate) = 2024&#x27;, &#x27;YEAR(ContractStartDate) = 2026&#x27;, &#x27;YEAR(ContractStartDate) = 2025&#x27;, &#x27;YEAR(ContractStartDate) = 2028&#x27;, &#x27;billings.Contract = Month-to-month&#x27;, &#x27;billings.Contract = Two year&#x27;, &#x27;billings.Contract = One year&#x27;, &#x27;billings.PaymentMethod = Electronic check&#x27;, &#x27;billings.PaymentMethod = Mailed check&#x27;, &#x27;billings.PaymentMethod = Bank transfer (automatic)&#x27;, &#x27;billings.PaymentMethod = Credit card (automatic)&#x27;, &#x27;subscriptions.PhoneService = Yes&#x27;, &#x27;subscriptions.PhoneService = No&#x27;, &#x27;subscriptions.InternetService = Fiber optic&#x27;, &#x27;subscriptions.InternetService = DSL&#x27;, &#x27;subscriptions.InternetService = No&#x27;, &#x27;subscriptions.OnlineSecurity = No&#x27;, &#x27;subscriptions.OnlineSecurity = Yes&#x27;, &#x27;subscriptions.OnlineSecurity = No internet service&#x27;, &#x27;subscriptions.OnlineBackup = No&#x27;, &#x27;subscriptions.OnlineBackup = Yes&#x27;, &#x27;subscriptions.OnlineBackup = No internet service&#x27;, &#x27;subscriptions.DeviceProtection = No&#x27;, &#x27;subscriptions.DeviceProtection = Yes&#x27;, &#x27;subscriptions.DeviceProtection = No internet service&#x27;, &#x27;subscriptions.TechSupport = No&#x27;, &#x27;subscriptions.TechSupport = Yes&#x27;, &#x27;subscriptions.TechSupport = No internet service&#x27;, &#x27;subscriptions.StreamingTV = No&#x27;, &#x27;subscriptions.StreamingTV = Yes&#x27;, &#x27;subscriptions.StreamingTV = No internet service&#x27;, &#x27;subscriptions.StreamingMovies = Yes&#x27;, &#x27;subscriptions.StreamingMovies = No&#x27;, &#x27;subscriptions.StreamingMovies = No internet service&#x27;, &#x27;billings.MODE(customers.UserCategory) = mid&#x27;, &#x27;billings.MODE(customers.UserCategory) = heavy&#x27;, &#x27;billings.MODE(customers.UserCategory) = light&#x27;, &#x27;subscriptions.MODE(customers.UserCategory) = mid&#x27;, &#x27;subscriptions.MODE(customers.UserCategory) = heavy&#x27;, &#x27;subscriptions.MODE(customers.UserCategory) = light&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-41\" type=\"checkbox\" ><label for=\"sk-estimator-id-41\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-42\" type=\"checkbox\" ><label for=\"sk-estimator-id-42\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SVC</label><div class=\"sk-toggleable__content\"><pre>SVC()</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('preprocessor',\n",
       "                 ColumnTransformer(remainder='passthrough',\n",
       "                                   transformers=[('num',\n",
       "                                                  Pipeline(steps=[('imputer',\n",
       "                                                                   SimpleImputer(strategy='median')),\n",
       "                                                                  ('scaler',\n",
       "                                                                   StandardScaler())]),\n",
       "                                                  ['TotalProductCount',\n",
       "                                                   'billings.tenure',\n",
       "                                                   'billings.MonthlyCharges',\n",
       "                                                   'billings.TotalCharges',\n",
       "                                                   'billings.MAX(customers.TotalProductCount)',\n",
       "                                                   'billings.MEAN(customers.TotalP...\n",
       "                                                   'subscriptions.MAX(customers.TotalProductCount)',\n",
       "                                                   'subscriptions.MEAN(customers.TotalProductCount)',\n",
       "                                                   'subscriptions.MIN(customers.TotalProductCount)',\n",
       "                                                   'subscriptions.PERCENT_TRUE(customers.SeniorCitizen)',\n",
       "                                                   'subscriptions.PERCENT_TRUE(customers.ShortContract)',\n",
       "                                                   'subscriptions.SUM(customers.TotalProductCount)'])],\n",
       "                                   verbose=True)),\n",
       "                ('classifier', SVC())],\n",
       "         verbose=True)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# fmt: off\n",
    "clf = Pipeline(\n",
    "    steps=[ (\"preprocessor\",   preprocessor), \n",
    "            (\"classifier\",     svm.SVC(kernel='rbf'))\n",
    "            ],\n",
    "            verbose=True\n",
    "            \n",
    ")\n",
    "# fmt: on\n",
    "\n",
    "clf.fit(X_train_feature_matrix, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we need to keep training, test, and validation data separate, we must also apply all transformations done by feature tools to the test and validation datasets. Therefore we again create an entity set and together with the feature definitions from the training set we can generate the same features for the other datasets. \n",
    "\n",
    "As the last step, we only have to call our pipeline for the transformed test and validation dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model score: 0.7842\n",
      "model score: 0.8034\n"
     ]
    }
   ],
   "source": [
    "es_val = entityset_wrapper(\n",
    "    id=\"customers_validation\",\n",
    "    data_frame_name=\"customers\",\n",
    "    feature_dataset=X_val,\n",
    "    variable_types=CUSTOMERS | SUBSCRIPTIONS | BILLINGS,\n",
    "    entity_set_columns=entity_set_columns,\n",
    ")\n",
    "val_transformed = ft.calculate_feature_matrix(X_train_features, es_val)\n",
    "\n",
    "\n",
    "es_test = entityset_wrapper(\n",
    "    id=\"customers_test\",\n",
    "    data_frame_name=\"customers\",\n",
    "    feature_dataset=X_test,\n",
    "    variable_types=CUSTOMERS | SUBSCRIPTIONS | BILLINGS,\n",
    "    entity_set_columns=entity_set_columns,\n",
    ")\n",
    "test_transformed = ft.calculate_feature_matrix(X_train_features, es_test)\n",
    "\n",
    "\n",
    "print(\"model score: %.4f\" % clf.score(val_transformed, y_val))\n",
    "print(\"model score: %.4f\" % clf.score(test_transformed, y_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmarking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As the last step, we gonna experiment with different depths of the features and what's happening to the model performance when we don't use feature selection algorithms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "@dataclass\n",
    "class BenchmarkData:\n",
    "    \"\"\"Small Dataclass to handle data for each set.\"\"\"\n",
    "\n",
    "    es: ft.EntitySet\n",
    "    X: pd.DataFrame\n",
    "    y: pd.Series\n",
    "\n",
    "\n",
    "def _wrap_col_transformer(feature_matrix: pd.DataFrame) -> ColumnTransformer:\n",
    "    numeric_ft = get_cols_by_type(\n",
    "        in_df=feature_matrix,\n",
    "        types_to_search=[np.float64, np.int64],\n",
    "        exclude_ids=False,\n",
    "    )\n",
    "\n",
    "    # fmt:off\n",
    "    numeric_transformer = Pipeline(\n",
    "        steps=[\n",
    "            (\"imputer\", SimpleImputer(missing_values=np.nan, strategy=NUMERIC_TRANSFORMER_REPLACEMENT)),\n",
    "            (\"scaler\", StandardScaler()),\n",
    "        ]\n",
    "    )\n",
    "    \n",
    "    preprocessor = ColumnTransformer(\n",
    "        transformers=[\n",
    "                   (\"num\", numeric_transformer, numeric_ft),\n",
    "        ],\n",
    "        verbose=True,\n",
    "        remainder=\"passthrough\",\n",
    "    )\n",
    "    # fmt: on\n",
    "    return preprocessor\n",
    "\n",
    "\n",
    "def benchmark(\n",
    "    train_data: BenchmarkData,\n",
    "    test_data: BenchmarkData,\n",
    "    val_data: BenchmarkData,\n",
    "    ft_selection_functions: tuple,\n",
    "    fitting_pipe: Pipeline,\n",
    "    target_df_name: str = None,\n",
    "    max_depth: int = 2,\n",
    "    **kwargs_dfs,\n",
    ") -> dict:\n",
    "    \"\"\"\n",
    "    Function to test different setups for max_depth and feature selection functions.\n",
    "    Additional parameters for the Featuretools dfs function can be supplied via keyword arguments.\n",
    "\n",
    "    :param BenchmarkData train_data: the training data\n",
    "    :param BenchmarkData test_data: the test data\n",
    "    :param BenchmarkData val_data: validation data\n",
    "    :param tuple ft_selection_functions: any number of feature selection functions provided by Featuretools\n",
    "    :param Pipeline fitting_pipe: the sklearn Pipline for fitting the model\n",
    "    :param str target_df_name: the target dataframe name for constructing the Featuretools EntitySet, defaults to None\n",
    "    :param int max_depth: max_depth parameter of Featuretools dfs, defaults to 2\n",
    "    :return dict: result dic\n",
    "    \"\"\"\n",
    "\n",
    "    ft_matrix = ft.dfs(\n",
    "        entityset=train_data.es,\n",
    "        target_dataframe_name=target_df_name,\n",
    "        max_depth=max_depth,\n",
    "        include_cutoff_time=False,\n",
    "        **kwargs_dfs,\n",
    "    )\n",
    "\n",
    "    ft_matrix_enc = ft.encode_features(*ft_matrix)\n",
    "    ft_matrix_reduced = (\n",
    "        ft_matrix_enc\n",
    "        if ft_selection_functions is None\n",
    "        else reduce(lambda res, f: f(*res), ft_selection_functions, ft_matrix_enc)\n",
    "    )\n",
    "\n",
    "    print(f\"Feature Matrix shape with {max_depth}: {ft_matrix_reduced[0].shape}\")\n",
    "    val_tf = ft.calculate_feature_matrix(ft_matrix_reduced[1], val_data.es)\n",
    "    test_tf = ft.calculate_feature_matrix(ft_matrix_reduced[1], test_data.es)\n",
    "    train_tf = ft_matrix_reduced[0]\n",
    "\n",
    "    col_transformer_new = _wrap_col_transformer(train_tf)\n",
    "    fitting_pipe.steps.pop(0)  # remove preprocessor step\n",
    "    fitting_pipe.steps.insert(\n",
    "        0, (\"preprocessor\", col_transformer_new)\n",
    "    )  # insert new preprocessor step\n",
    "    fitting_pipe.fit(X=train_tf, y=train_data.y)\n",
    "\n",
    "    return {\n",
    "        \"model_score_validation\": fitting_pipe.score(val_tf, val_data.y),\n",
    "        \"model_score_test\": fitting_pipe.score(test_tf, test_data.y),\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = BenchmarkData(es=es, X=X_train, y=y_train)\n",
    "test_data = BenchmarkData(es=es_test, X=X_test, y=y_test)\n",
    "val_data = BenchmarkData(es=es_val, X=X_val, y=y_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We run the experiment with max depth from 1 to 3 and each with and without feature selection ([remove_highly_null_features](https://featuretools.alteryx.com/en/stable/generated/featuretools.selection.remove_highly_null_features.html) and [remove_single_value_features](https://featuretools.alteryx.com/en/stable/generated/featuretools.selection.remove_single_value_features.html?highlight=remove_single_value_features)):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Feature Matrix shape with 1: (4225, 64)\n",
      "[ColumnTransformer] ........... (1 of 2) Processing num, total=   0.0s\n",
      "[ColumnTransformer] ..... (2 of 2) Processing remainder, total=   0.0s\n",
      "[Pipeline] ...... (step 1 of 2) Processing preprocessor, total=   0.0s\n",
      "[Pipeline] ........ (step 2 of 2) Processing classifier, total=   0.6s\n",
      "Feature Matrix shape with 1: (4225, 84)\n",
      "[ColumnTransformer] ........... (1 of 2) Processing num, total=   0.0s\n",
      "[ColumnTransformer] ..... (2 of 2) Processing remainder, total=   0.0s\n",
      "[Pipeline] ...... (step 1 of 2) Processing preprocessor, total=   0.0s\n",
      "[Pipeline] ........ (step 2 of 2) Processing classifier, total=   0.6s\n",
      "Feature Matrix shape with 2: (4225, 82)\n",
      "[ColumnTransformer] ........... (1 of 2) Processing num, total=   0.0s\n",
      "[ColumnTransformer] ..... (2 of 2) Processing remainder, total=   0.0s\n",
      "[Pipeline] ...... (step 1 of 2) Processing preprocessor, total=   0.0s\n",
      "[Pipeline] ........ (step 2 of 2) Processing classifier, total=   0.7s\n",
      "Feature Matrix shape with 2: (4225, 118)\n",
      "[ColumnTransformer] ........... (1 of 2) Processing num, total=   0.0s\n",
      "[ColumnTransformer] ..... (2 of 2) Processing remainder, total=   0.0s\n",
      "[Pipeline] ...... (step 1 of 2) Processing preprocessor, total=   0.0s\n",
      "[Pipeline] ........ (step 2 of 2) Processing classifier, total=   0.8s\n",
      "Feature Matrix shape with 3: (4225, 132)\n",
      "[ColumnTransformer] ........... (1 of 2) Processing num, total=   0.0s\n",
      "[ColumnTransformer] ..... (2 of 2) Processing remainder, total=   0.0s\n",
      "[Pipeline] ...... (step 1 of 2) Processing preprocessor, total=   0.0s\n",
      "[Pipeline] ........ (step 2 of 2) Processing classifier, total=   0.9s\n",
      "Feature Matrix shape with 3: (4225, 184)\n",
      "[ColumnTransformer] ........... (1 of 2) Processing num, total=   0.0s\n",
      "[ColumnTransformer] ..... (2 of 2) Processing remainder, total=   0.0s\n",
      "[Pipeline] ...... (step 1 of 2) Processing preprocessor, total=   0.0s\n",
      "[Pipeline] ........ (step 2 of 2) Processing classifier, total=   1.1s\n"
     ]
    }
   ],
   "source": [
    "result_dic = dict()\n",
    "\n",
    "with_f_select = {\n",
    "    \"train_data\": train_data,\n",
    "    \"test_data\": test_data,\n",
    "    \"val_data\": val_data,\n",
    "    \"ft_selection_functions\": (\n",
    "        remove_highly_null_features,\n",
    "        remove_single_value_features,\n",
    "    ),\n",
    "    \"target_df_name\": \"customers\",\n",
    "    \"fitting_pipe\": clf,\n",
    "    \"max_depth\": None,\n",
    "}\n",
    "\n",
    "without_f_select = dict(with_f_select)\n",
    "without_f_select[\"ft_selection_functions\"] = None\n",
    "\n",
    "\n",
    "for i in [1, 2, 3]:\n",
    "    with_f_select[\"max_depth\"] = i\n",
    "    without_f_select[\"max_depth\"] = i\n",
    "\n",
    "    result_dic[f\"max_depth_{i}_with_feature_selection\"] = benchmark(**with_f_select)\n",
    "    result_dic[f\"max_depth_{i}_without_feature_selection\"] = benchmark(\n",
    "        **without_f_select\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model_score_validation</th>\n",
       "      <th>model_score_test</th>\n",
       "      <th>diff</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>max_depth_1_with_feature_selection</th>\n",
       "      <td>0.779986</td>\n",
       "      <td>0.799858</td>\n",
       "      <td>0.019872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max_depth_1_without_feature_selection</th>\n",
       "      <td>0.778566</td>\n",
       "      <td>0.802697</td>\n",
       "      <td>0.024131</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max_depth_2_with_feature_selection</th>\n",
       "      <td>0.784244</td>\n",
       "      <td>0.803407</td>\n",
       "      <td>0.019163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max_depth_2_without_feature_selection</th>\n",
       "      <td>0.782825</td>\n",
       "      <td>0.803407</td>\n",
       "      <td>0.020582</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max_depth_3_with_feature_selection</th>\n",
       "      <td>0.777857</td>\n",
       "      <td>0.794180</td>\n",
       "      <td>0.016324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max_depth_3_without_feature_selection</th>\n",
       "      <td>0.777147</td>\n",
       "      <td>0.798439</td>\n",
       "      <td>0.021292</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       model_score_validation  \\\n",
       "max_depth_1_with_feature_selection                   0.779986   \n",
       "max_depth_1_without_feature_selection                0.778566   \n",
       "max_depth_2_with_feature_selection                   0.784244   \n",
       "max_depth_2_without_feature_selection                0.782825   \n",
       "max_depth_3_with_feature_selection                   0.777857   \n",
       "max_depth_3_without_feature_selection                0.777147   \n",
       "\n",
       "                                       model_score_test      diff  \n",
       "max_depth_1_with_feature_selection             0.799858  0.019872  \n",
       "max_depth_1_without_feature_selection          0.802697  0.024131  \n",
       "max_depth_2_with_feature_selection             0.803407  0.019163  \n",
       "max_depth_2_without_feature_selection          0.803407  0.020582  \n",
       "max_depth_3_with_feature_selection             0.794180  0.016324  \n",
       "max_depth_3_without_feature_selection          0.798439  0.021292  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df = pd.DataFrame(result_dic).transpose()\n",
    "results_df[\"diff\"] = results_df.model_score_test - results_df.model_score_validation\n",
    "results_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results show that the difference is rather small but in our case max depth with feature selection. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Conclusion:\n",
    "\n",
    "The Featuretools framework provides a nice and easy-to-use approach for automated feature generation which requires very little coding effort compared to manually generate all the features. Especially if data comes in normalized form, eg. from a database, the entity set generation and setting up the relations is rather easy and possibly can be mimicked from the database relations. For time-indexed data, test-train splits can be done more easily by setting the splitting index compared to our example. When it comes to performance tuning of the feature generation process we only scratched the surface of the framework, Infos on fine tuning can be found [here](https://featuretools.alteryx.com/en/stable/guides/tuning_dfs.html).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Acknowledgement: \n",
    "\n",
    "- Kaggle example of how to use featuretools on the titanc dataset: [Automated feature engineering for Titanic dataset](https://www.kaggle.com/code/liananapalkova/automated-feature-engineering-for-titanic-dataset/notebook)\n",
    "- [IBM GitHup repos with the data](https://github.com/IBM/telco-customer-churn-on-icp4d/blob/master/data/Telco-Customer-Churn.csv)\n",
    "- [Featuretools](https://featuretools.alteryx.com/en/stable/#)\n",
    "\n",
    "___\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.10 ('data-science-playbooks-Yk-1xU1Q-py3.10')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "919e6aa0d89df1282d444dc7cd808811badc65a732d2c47612a254902fd6721c"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}