docs/target_encoding_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A large number of machine learning algorithms can not work with categorical data out-of-the-box. One approach to go over this obstacle is to use one-hot encoding. However, if categorical feature takes `k` distinct values and there are `n` objects in a dataset, `n` * `k` values will be stored after dense one-hot encoding. In case of high `k`, this can cause `MemoryError`. Of course, there are several approaches that do not result in extreme growth of learning sample size even if categorical features have high cardinality. For example, hashing trick is implemented in `sklearn.feature_extraction.FeatureHasher`. In this demo, another memory-efficient approach is shown.\n",
    "\n",
    "Suppose that we want to replace categorical feature with mean of target variable conditional on this feature (i.e., group by the feature, average target within each group, and replace each value of the feature with average of its group). Is it a good idea? At the first glance, yes, it is. Nevertheless, the devil is in the detail. The smaller a group of objects with a particular value of the feature is, the higher contribution of each object's target to within-group average is. Random noise from target variable leaks to a new feature and it is not milded by averaging over a sample if this sample is not big enough. Hence, this new feature gives an advantage that is unfair, because this advantage is present during training, but it is absent when new examples with unknown target are provided.\n",
    "\n",
    "A cure to this problem is to compute aggregates of target with usage of other object's targets, but without target of an object for which a new feature is being computed right now. For example, new feature can be generated out-of-fold: data are split into folds and a new feature for a fold is computed on all other folds only. It is like stacking where a first-stage model does not use any features directly and just applies an aggregating function to a target with respect to grouping by a categorical feature under processing.\n",
    "\n",
    "Classes `OutOfFoldTargetEncodingRegressor` and `OutOfFoldTargetEncodingClassifier` that can be imported from `dsawl.target_encoding` are plug-and-play implementations of this trick. They have `sklearn`-compatible API such that cross validation scores measured by means of `sklearn` are realistic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look at [this Kaggle post](https://www.kaggle.com/general/16927#95887) for additional details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# General Preparations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Statements & Random Seed Setting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from typing import List, Tuple\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.metrics import r2_score\n",
    "from sklearn.model_selection import KFold\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.preprocessing import StandardScaler, PolynomialFeatures\n",
    "\n",
    "from dsawl.target_encoding import OutOfFoldTargetEncodingRegressor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "np.random.seed(361)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Synthetic Dataset Generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate_data(\n",
    "        slopes: List[float],\n",
    "        group_size: int,\n",
    "        noise_stddev: float\n",
    "        ) -> pd.DataFrame:\n",
    "    \"\"\"\n",
    "    Generate `len(slopes)` * `group_size` examples\n",
    "    with dependency y = slope * x + noise.\n",
    "    \"\"\"\n",
    "    dfs = []\n",
    "    for i, slope in enumerate(slopes):\n",
    "        curr_df = pd.DataFrame(columns=['x', 'category', 'y'])\n",
    "        curr_df['x'] = range(group_size)\n",
    "        curr_df['category'] = i\n",
    "        curr_df['y'] = curr_df['x'].apply(\n",
    "            lambda x: slope * x + np.random.normal(scale=noise_stddev)\n",
    "        )\n",
    "        dfs.append(curr_df)\n",
    "    df = pd.concat(dfs)\n",
    "    return df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us create a situation where in-fold generation of target-based features leads to overfitting. To do so, make a lot of small categories and set noise variance to a high value. Such settings result in leakage of noise from target into in-fold generated mean. Thus, regressor learns how to use this leakage, which is useless on hold-out sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "slopes = [2, 1, 3, 4, -1, -2, 3, 2, 1, 5, -2, -3, -5, 8, 1, -7, 0, 2, 0]\n",
    "group_size = 5\n",
    "noise_stddev = 10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>category</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3.314436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>-4.654332</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1.642263</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>15.854888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>21.873901</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   x  category          y\n",
       "0  0         0   3.314436\n",
       "1  1         0  -4.654332\n",
       "2  2         0   1.642263\n",
       "3  3         0  15.854888\n",
       "4  4         0  21.873901"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df = generate_data(slopes, group_size, noise_stddev)\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate test set from the same distribution (in a way that preserves balance between categories)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>x</th>\n",
       "      <th>category</th>\n",
       "      <th>y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.682203</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>9.033670</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>8.911399</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>2.345571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>3.408391</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   x  category         y\n",
       "0  0         0 -0.682203\n",
       "1  1         0  9.033670\n",
       "2  2         0  8.911399\n",
       "3  3         0  2.345571\n",
       "4  4         0  3.408391"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df = generate_data(slopes, group_size, noise_stddev)\n",
    "test_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Benchmark: Training with In-Fold Generated Mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "encoding_df = (\n",
    "    train_df\n",
    "    .groupby('category', as_index=False)\n",
    "    .agg({'y': np.mean})\n",
    "    .rename(columns={'y': 'infold_mean'})\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_target_and_features(df: pd.DataFrame) -> Tuple[pd.DataFrame]:\n",
    "    merged_df = df.merge(encoding_df, on='category')\n",
    "    X = merged_df[['x', 'infold_mean']]\n",
    "    y = merged_df['y']\n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, y_train = get_target_and_features(train_df)\n",
    "X_test, y_test = get_target_and_features(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rgr = LinearRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.42987610029007461"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rgr.fit(X_train, y_train)\n",
    "y_hat_train = rgr.predict(X_train)\n",
    "r2_score(y_train, y_hat_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.061377490670754042"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_hat = rgr.predict(X_test)\n",
    "r2_score(y_test, y_hat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obviously, overfitting is detected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Out-of-Fold Target Encoding Estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, y_train = train_df[['x', 'category']], train_df['y']\n",
    "X_test, y_test = test_df[['x', 'category']], test_df['y']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "splitter = KFold(shuffle=True, random_state=361)\n",
    "rgr = OutOfFoldTargetEncodingRegressor(\n",
    "    LinearRegression,  # It is a type, not an instance of a class.\n",
    "    dict(),  # If neeeded, pass constructor arguments here as a dictionary.\n",
    "    # Separation of constructor arguments from estimator makes code\n",
    "    # with involvement of tools such as `GridSearchCV` more consistent.\n",
    "    splitter=splitter,  # Define how to make folds for features generation.\n",
    "    smoothing_strength=0,  # New feature can be smoothed towards unconditional aggregate.\n",
    "    min_frequency=1,  # Unconditional aggregate is used for rare enough values.\n",
    "    drop_source_features=True  # To use or not to use features from conditioning.\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is below is a wrong way to measure train score. Regressor uses in-fold generated features for predictions, not the features that are used for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.27857429182727422"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rgr.fit(X_train, y_train, source_positions=[1])\n",
    "y_hat_train = rgr.predict(X_train)\n",
    "r2_score(y_train, y_hat_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actually, imparity between training set and all other sets is the main reason why special estimators are implemented instead of making `dsawl.target_encoding.TargetEncoder` be able to work inside `sklearn.pipeline.Pipeline`. If you want to use an estimator with target encoding inside a pipeline, pass pipeline instance as an internal estimator, i.e., as the first argument. For more details, please go to *Appendix II* of this demo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us look at the right way to measure performance on train set. In `OutOfFoldTargetEncodingRegressor` and `OutOfFoldTargetEncodingClassifier`, `fit_predict` method is not just a combination of `fit` and `predict` methods, because it is designed specially for correct work with training sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.18233620473442524"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_hat_train = rgr.fit_predict(X_train, y_train, source_positions=[1])\n",
    "r2_score(y_train, y_hat_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.14721978147696491"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rgr.fit(X_train, y_train, source_positions=[1])\n",
    "y_hat = rgr.predict(X_test)\n",
    "r2_score(y_test, y_hat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, there is no blatant overfitting — train set score and test set score are close to each other (for some other random seeds the gap between them might be lower, but, anyway, it is not too considerable even now). Also it is appropriate to highlight that test set score is significantly higher than that of the benchmark regressor trained with mean generated in-fold."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix I. Some Snippets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How to Run Grid Search?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'estimator_params': {}, 'min_frequency': 0, 'smoothing_strength': 0}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "\n",
    "grid_params = {\n",
    "    'estimator_params': [{}, {'fit_intercept': False}],\n",
    "    'smoothing_strength': [0, 10],\n",
    "    'min_frequency': [0, 10]\n",
    "}\n",
    "rgr = GridSearchCV(\n",
    "    OutOfFoldTargetEncodingRegressor(),\n",
    "    grid_params\n",
    ")\n",
    "rgr.fit(X_train, y_train, source_positions=[1])\n",
    "rgr.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Appendix II. Advanced Integration with Pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start with, remind that the key reason of difficulties with target encoding is imparity between training set and other sets for which predictions are made. Training set is split into folds, whereas other sets are not, because targets from them are not used for target encoding.\n",
    "\n",
    "If correct measuring of train scores is not crucial for you, you can use this quick-and-impure trick."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from dsawl.target_encoding import TargetEncoder\n",
    "\n",
    "\n",
    "class PipelinedTargetEncoder(TargetEncoder):\n",
    "    pass\n",
    "\n",
    "\n",
    "PipelinedTargetEncoder.fit_transform = PipelinedTargetEncoder.fit_transform_out_of_fold\n",
    "\n",
    "# Now instances of `PipelinedTargetEncoder` can be used as transformers in pipelines\n",
    "# and target encoding inside these pipelines is implemented in an out-of-fold fashion."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Anyway, look at a way that allows working with train scores methodocogically right.\n",
    "\n",
    "Suppose that by some unknown reasons there is a need in learning a model from the dataset under consideration such that:\n",
    "* first of all, both features are scaled,\n",
    "* then the categorical feature is target-encoded,\n",
    "* then squares, cubes, and interactions of order not higher than three between all terms are included as a new features,\n",
    "* and, finally, linear regression is run.\n",
    "\n",
    "This is not so easy, because if `Pipeline` instance is passed as an internal estimator, target encoding is the first transformation, yet in this case it must go after scaling. Below snippet demonstrates how to use `OutOfFoldTargetEncodingRegressor` inside a pipeline that meets above specifications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('scaling', StandardScaler(copy=True, with_mean=True, with_std=True)), ('target_encoding_regression', OutOfFoldTargetEncodingRegressor(aggregators=None, drop_source_features=True,\n",
       "                 estimator_params=None, estimator_type=None,\n",
       "                 min_frequency=1, smoothing_strength=0.0,\n",
       "                 splitter=KFold(n_splits=3, random_state=None, shuffle=True)))])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "splitter = KFold(shuffle=True, random_state=361)\n",
    "rgr = OutOfFoldTargetEncodingRegressor(\n",
    "    Pipeline,\n",
    "    {\n",
    "        'steps': [\n",
    "            ('poly_enrichment', PolynomialFeatures()),\n",
    "            ('linear_model', LinearRegression())\n",
    "        ],\n",
    "        'poly_enrichment__degree': 3\n",
    "    },\n",
    "    splitter=splitter\n",
    ")\n",
    "\n",
    "pipeline = Pipeline([\n",
    "    ('scaling', StandardScaler()),\n",
    "    ('target_encoding_regression', rgr)\n",
    "])\n",
    "\n",
    "X_train = X_train.astype(np.float64)  # Avoid `DataConversionWarning`.\n",
    "pipeline.fit(\n",
    "    X_train, y_train,\n",
    "    target_encoding_regression__source_positions=[1]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.263271432939716"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r2_score(\n",
    "    y_train,\n",
    "    pipeline.fit_predict(\n",
    "        X_train, y_train,\n",
    "        target_encoding_regression__source_positions=[1]\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Actually, relatively good training score is achieved by chance and this odd pipeline does not have any advantages over regular estimator. The cell above just demonstrates how train score can be measured."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dsawl_env",
   "language": "python",
   "name": "dsawl_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}