example/Example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial for hccEncoding\n",
    "\n",
    "\n",
    "This notebook will exhibit how to use hcc-encoding package for classification and regression problem using dataset from Kaggle competition 'Prudential Life Insurance Assessment'\n",
    "\n",
    "In hcc-encoding, the basic principle motivating the processing is to map individual values of a high-cardinality categorical independent attribute to an estimate of the probability or the expected value of dependent attribute. However, just simply transfer the high-cardinality categorical to target statistics often result in information leaking.
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "dataset download: https://www.kaggle.com/c/prudential-life-insurance-assessment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this dataset, you are provided over a hundred variables describing attributes of life insurance applicants. \n",
    "The task is to predict the \"Response\" variable for each Id in the test set. \n",
    "\"Response\" is an ordinal measure of risk that has 8 levels, which means the problem can be treated as both regression problem and classification problems (classify to 8 classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# load raw data\n",
    "import pandas as pd\n",
    "train=pd.read_csv('train.csv')\n",
    "test=pd.read_csv('test.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Product_Info_1</th>\n",
       "      <th>Product_Info_2</th>\n",
       "      <th>Product_Info_3</th>\n",
       "      <th>Product_Info_4</th>\n",
       "      <th>Product_Info_5</th>\n",
       "      <th>Product_Info_6</th>\n",
       "      <th>Product_Info_7</th>\n",
       "      <th>Ins_Age</th>\n",
       "      <th>Ht</th>\n",
       "      <th>...</th>\n",
       "      <th>Medical_Keyword_40</th>\n",
       "      <th>Medical_Keyword_41</th>\n",
       "      <th>Medical_Keyword_42</th>\n",
       "      <th>Medical_Keyword_43</th>\n",
       "      <th>Medical_Keyword_44</th>\n",
       "      <th>Medical_Keyword_45</th>\n",
       "      <th>Medical_Keyword_46</th>\n",
       "      <th>Medical_Keyword_47</th>\n",
       "      <th>Medical_Keyword_48</th>\n",
       "      <th>Response</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>D3</td>\n",
       "      <td>10</td>\n",
       "      <td>0.076923</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0.641791</td>\n",
       "      <td>0.581818</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>A1</td>\n",
       "      <td>26</td>\n",
       "      <td>0.076923</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0.059701</td>\n",
       "      <td>0.600000</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>E1</td>\n",
       "      <td>26</td>\n",
       "      <td>0.076923</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0.029851</td>\n",
       "      <td>0.745455</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>D4</td>\n",
       "      <td>10</td>\n",
       "      <td>0.487179</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0.164179</td>\n",
       "      <td>0.672727</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>1</td>\n",
       "      <td>D2</td>\n",
       "      <td>26</td>\n",
       "      <td>0.230769</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0.417910</td>\n",
       "      <td>0.654545</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 128 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  Product_Info_1 Product_Info_2  Product_Info_3  Product_Info_4  \\\n",
       "0   2               1             D3              10        0.076923   \n",
       "1   5               1             A1              26        0.076923   \n",
       "2   6               1             E1              26        0.076923   \n",
       "3   7               1             D4              10        0.487179   \n",
       "4   8               1             D2              26        0.230769   \n",
       "\n",
       "   Product_Info_5  Product_Info_6  Product_Info_7   Ins_Age        Ht  \\\n",
       "0               2               1               1  0.641791  0.581818   \n",
       "1               2               3               1  0.059701  0.600000   \n",
       "2               2               3               1  0.029851  0.745455   \n",
       "3               2               3               1  0.164179  0.672727   \n",
       "4               2               3               1  0.417910  0.654545   \n",
       "\n",
       "     ...     Medical_Keyword_40  Medical_Keyword_41  Medical_Keyword_42  \\\n",
       "0    ...                      0                   0                   0   \n",
       "1    ...                      0                   0                   0   \n",
       "2    ...                      0                   0                   0   \n",
       "3    ...                      0                   0                   0   \n",
       "4    ...                      0                   0                   0   \n",
       "\n",
       "   Medical_Keyword_43  Medical_Keyword_44  Medical_Keyword_45  \\\n",
       "0                   0                   0                   0   \n",
       "1                   0                   0                   0   \n",
       "2                   0                   0                   0   \n",
       "3                   0                   0                   0   \n",
       "4                   0                   0                   0   \n",
       "\n",
       "   Medical_Keyword_46  Medical_Keyword_47  Medical_Keyword_48  Response  \n",
       "0                   0                   0                   0         8  \n",
       "1                   0                   0                   0         4  \n",
       "2                   0                   0                   0         8  \n",
       "3                   0                   0                   0         8  \n",
       "4                   0                   0                   0         8  \n",
       "\n",
       "[5 rows x 128 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train['Product_Info_2'].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be seen that the feature 'Product_Info_2' can be treated as a high-cardinal feature. To exhibit how to use hcc-encoding more easily, we ignore most irrelavant features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train=train[['Id','Response','Product_Info_2']]\n",
    "test=test[['Id','Product_Info_2']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Response</th>\n",
       "      <th>Product_Info_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>D3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>A1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>E1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>D4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>D2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  Response Product_Info_2\n",
       "0   2         8             D3\n",
       "1   5         4             A1\n",
       "2   6         8             E1\n",
       "3   7         8             D4\n",
       "4   8         8             D2"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1. Encoding for classification problems"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from hccEncoding.EncoderForClassification import BayesEncoding,BayesEncodingKfold,LOOEncoding,LOOEncodingKfold\n",
    "\n",
    "train_BayesEncoding,test_BayesEncoding=BayesEncoding(train,test,'Response','Product_Info_2')\n",
    "train_LOOEncoding,test_LOOEncoding=LOOEncoding(train,test,'Response','Product_Info_2')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Response</th>\n",
       "      <th>Product_Info_2</th>\n",
       "      <th>bayes_Product_Info_2_1</th>\n",
       "      <th>bayes_Product_Info_2_2</th>\n",
       "      <th>bayes_Product_Info_2_3</th>\n",
       "      <th>bayes_Product_Info_2_4</th>\n",
       "      <th>bayes_Product_Info_2_5</th>\n",
       "      <th>bayes_Product_Info_2_6</th>\n",
       "      <th>bayes_Product_Info_2_7</th>\n",
       "      <th>bayes_Product_Info_2_8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>D3</td>\n",
       "      <td>0.100520</td>\n",
       "      <td>0.116242</td>\n",
       "      <td>0.016612</td>\n",
       "      <td>0.029456</td>\n",
       "      <td>0.087625</td>\n",
       "      <td>0.227483</td>\n",
       "      <td>0.145368</td>\n",
       "      <td>0.274486</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>A1</td>\n",
       "      <td>0.055488</td>\n",
       "      <td>0.100343</td>\n",
       "      <td>0.021790</td>\n",
       "      <td>0.028626</td>\n",
       "      <td>0.083012</td>\n",
       "      <td>0.148888</td>\n",
       "      <td>0.115147</td>\n",
       "      <td>0.445322</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>E1</td>\n",
       "      <td>0.076636</td>\n",
       "      <td>0.078272</td>\n",
       "      <td>0.012823</td>\n",
       "      <td>0.029673</td>\n",
       "      <td>0.067797</td>\n",
       "      <td>0.162853</td>\n",
       "      <td>0.150906</td>\n",
       "      <td>0.425601</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>D4</td>\n",
       "      <td>0.064025</td>\n",
       "      <td>0.065807</td>\n",
       "      <td>0.007659</td>\n",
       "      <td>0.011436</td>\n",
       "      <td>0.075465</td>\n",
       "      <td>0.166885</td>\n",
       "      <td>0.135976</td>\n",
       "      <td>0.477254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>D2</td>\n",
       "      <td>0.119590</td>\n",
       "      <td>0.147281</td>\n",
       "      <td>0.017612</td>\n",
       "      <td>0.024653</td>\n",
       "      <td>0.067270</td>\n",
       "      <td>0.247489</td>\n",
       "      <td>0.145203</td>\n",
       "      <td>0.233645</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  Response Product_Info_2  bayes_Product_Info_2_1  \\\n",
       "0   2         8             D3                0.100520   \n",
       "1   5         4             A1                0.055488   \n",
       "2   6         8             E1                0.076636   \n",
       "3   7         8             D4                0.064025   \n",
       "4   8         8             D2                0.119590   \n",
       "\n",
       "   bayes_Product_Info_2_2  bayes_Product_Info_2_3  bayes_Product_Info_2_4  \\\n",
       "0                0.116242                0.016612                0.029456   \n",
       "1                0.100343                0.021790                0.028626   \n",
       "2                0.078272                0.012823                0.029673   \n",
       "3                0.065807                0.007659                0.011436   \n",
       "4                0.147281                0.017612                0.024653   \n",
       "\n",
       "   bayes_Product_Info_2_5  bayes_Product_Info_2_6  bayes_Product_Info_2_7  \\\n",
       "0                0.087625                0.227483                0.145368   \n",
       "1                0.083012                0.148888                0.115147   \n",
       "2                0.067797                0.162853                0.150906   \n",
       "3                0.075465                0.166885                0.135976   \n",
       "4                0.067270                0.247489                0.145203   \n",
       "\n",
       "   bayes_Product_Info_2_8  \n",
       "0                0.274486  \n",
       "1                0.445322  \n",
       "2                0.425601  \n",
       "3                0.477254  \n",
       "4                0.233645  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_BayesEncoding.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: In BayesEncoding for classification problem, after encoding, new feature will be the probilities of each class, the header of generated new feature will be 'bayes_ __ _(Origin Feature name)_ __ _(name of the class)'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Response</th>\n",
       "      <th>Product_Info_2</th>\n",
       "      <th>loo_Product_Info_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>D3</td>\n",
       "      <td>5.568211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>A1</td>\n",
       "      <td>6.176825</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>E1</td>\n",
       "      <td>6.171721</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>D4</td>\n",
       "      <td>6.373370</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>D2</td>\n",
       "      <td>5.260191</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  Response Product_Info_2  loo_Product_Info_2\n",
       "0   2         8             D3            5.568211\n",
       "1   5         4             A1            6.176825\n",
       "2   6         8             E1            6.171721\n",
       "3   7         8             D4            6.373370\n",
       "4   8         8             D2            5.260191"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_LOOEncoding.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_BayesEncodingKfold,test_BayesEncodingKfold=BayesEncodingKfold(train,test,'Response','Product_Info_2',fold=5)\n",
    "train_LOOEncodingKfold,test_LOOEncodingKfold=LOOEncodingKfold(train,test,'Response','Product_Info_2',fold=5)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Response</th>\n",
       "      <th>Product_Info_2</th>\n",
       "      <th>bayes_Product_Info_2_1</th>\n",
       "      <th>bayes_Product_Info_2_2</th>\n",
       "      <th>bayes_Product_Info_2_3</th>\n",
       "      <th>bayes_Product_Info_2_4</th>\n",
       "      <th>bayes_Product_Info_2_5</th>\n",
       "      <th>bayes_Product_Info_2_6</th>\n",
       "      <th>bayes_Product_Info_2_7</th>\n",
       "      <th>bayes_Product_Info_2_8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>D3</td>\n",
       "      <td>0.103135</td>\n",
       "      <td>0.116970</td>\n",
       "      <td>0.016271</td>\n",
       "      <td>0.029247</td>\n",
       "      <td>0.086499</td>\n",
       "      <td>0.229355</td>\n",
       "      <td>0.147193</td>\n",
       "      <td>0.268195</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>A1</td>\n",
       "      <td>0.054170</td>\n",
       "      <td>0.094039</td>\n",
       "      <td>0.023956</td>\n",
       "      <td>0.030422</td>\n",
       "      <td>0.087120</td>\n",
       "      <td>0.148125</td>\n",
       "      <td>0.115648</td>\n",
       "      <td>0.443238</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>E1</td>\n",
       "      <td>0.072549</td>\n",
       "      <td>0.079161</td>\n",
       "      <td>0.013412</td>\n",
       "      <td>0.028990</td>\n",
       "      <td>0.063683</td>\n",
       "      <td>0.161461</td>\n",
       "      <td>0.150065</td>\n",
       "      <td>0.433177</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>D4</td>\n",
       "      <td>0.063238</td>\n",
       "      <td>0.064417</td>\n",
       "      <td>0.007972</td>\n",
       "      <td>0.011082</td>\n",
       "      <td>0.076236</td>\n",
       "      <td>0.165874</td>\n",
       "      <td>0.131380</td>\n",
       "      <td>0.478303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>D2</td>\n",
       "      <td>0.114566</td>\n",
       "      <td>0.146584</td>\n",
       "      <td>0.018327</td>\n",
       "      <td>0.024603</td>\n",
       "      <td>0.068299</td>\n",
       "      <td>0.245421</td>\n",
       "      <td>0.148178</td>\n",
       "      <td>0.233748</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  Response Product_Info_2  bayes_Product_Info_2_1  \\\n",
       "0   2         8             D3                0.103135   \n",
       "1   5         4             A1                0.054170   \n",
       "2   6         8             E1                0.072549   \n",
       "3   7         8             D4                0.063238   \n",
       "4   8         8             D2                0.114566   \n",
       "\n",
       "   bayes_Product_Info_2_2  bayes_Product_Info_2_3  bayes_Product_Info_2_4  \\\n",
       "0                0.116970                0.016271                0.029247   \n",
       "1                0.094039                0.023956                0.030422   \n",
       "2                0.079161                0.013412                0.028990   \n",
       "3                0.064417                0.007972                0.011082   \n",
       "4                0.146584                0.018327                0.024603   \n",
       "\n",
       "   bayes_Product_Info_2_5  bayes_Product_Info_2_6  bayes_Product_Info_2_7  \\\n",
       "0                0.086499                0.229355                0.147193   \n",
       "1                0.087120                0.148125                0.115648   \n",
       "2                0.063683                0.161461                0.150065   \n",
       "3                0.076236                0.165874                0.131380   \n",
       "4                0.068299                0.245421                0.148178   \n",
       "\n",
       "   bayes_Product_Info_2_8  \n",
       "0                0.268195  \n",
       "1                0.443238  \n",
       "2                0.433177  \n",
       "3                0.478303  \n",
       "4                0.233748  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_BayesEncodingKfold.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Response</th>\n",
       "      <th>Product_Info_2</th>\n",
       "      <th>loo_Product_Info_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>D3</td>\n",
       "      <td>5.510350</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>A1</td>\n",
       "      <td>6.147088</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>E1</td>\n",
       "      <td>6.165965</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>D4</td>\n",
       "      <td>6.389432</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>D2</td>\n",
       "      <td>5.280445</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  Response Product_Info_2  loo_Product_Info_2\n",
       "0   2         8             D3            5.510350\n",
       "1   5         4             A1            6.147088\n",
       "2   6         8             E1            6.165965\n",
       "3   7         8             D4            6.389432\n",
       "4   8         8             D2            5.280445"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_LOOEncodingKfold.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: the difference between BayesEncoding and BayesEncodingKfold (also LOOEncoding and LOOEncodingKfold) is how to encode train dataset. In BayesEncoding (or LOOEncoding), the train dataset is encoded using statistics of full train dataset. In BayesEncodingKfold (or LOOEncodingKfold), the train dataset is encoded using statistics of part of train dataset. For example, when fold=5, Baye0%sEncodingKfold (or LOOEncodingKfold) use 80% of train dataset to encode the rest 20% train dataset. This can further reduce the risk of information leaking, the cons is to use less information from train dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2. Encoding for Regression problems"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from hccEncoding.EncoderForRegression import BayesEncoding,BayesEncodingKfold,LOOEncoding,LOOEncodingKfold\n",
    "\n",
    "train_BayesEncoding,test_BayesEncoding=BayesEncoding(train,test,'Response','Product_Info_2')\n",
    "train_LOOEncoding,test_LOOEncoding=LOOEncoding(train,test,'Response','Product_Info_2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Response</th>\n",
       "      <th>Product_Info_2</th>\n",
       "      <th>bayes_Product_Info_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>D3</td>\n",
       "      <td>5.554615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>A1</td>\n",
       "      <td>6.212693</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>E1</td>\n",
       "      <td>6.226905</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>D4</td>\n",
       "      <td>6.337270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>D2</td>\n",
       "      <td>5.261874</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  Response Product_Info_2  bayes_Product_Info_2\n",
       "0   2         8             D3              5.554615\n",
       "1   5         4             A1              6.212693\n",
       "2   6         8             E1              6.226905\n",
       "3   7         8             D4              6.337270\n",
       "4   8         8             D2              5.261874"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_BayesEncoding.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Response</th>\n",
       "      <th>Product_Info_2</th>\n",
       "      <th>loo_Product_Info_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>D3</td>\n",
       "      <td>5.553309</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>A1</td>\n",
       "      <td>6.042045</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>E1</td>\n",
       "      <td>6.065838</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>D4</td>\n",
       "      <td>6.423081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>D2</td>\n",
       "      <td>5.302704</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  Response Product_Info_2  loo_Product_Info_2\n",
       "0   2         8             D3            5.553309\n",
       "1   5         4             A1            6.042045\n",
       "2   6         8             E1            6.065838\n",
       "3   7         8             D4            6.423081\n",
       "4   8         8             D2            5.302704"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_LOOEncoding.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_BayesEncodingKfold,test_BayesEncodingKfold=BayesEncodingKfold(train,test,'Response','Product_Info_2',fold=5)\n",
    "train_LOOEncodingKfold,test_LOOEncodingKfold=LOOEncodingKfold(train,test,'Response','Product_Info_2',fold=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Response</th>\n",
       "      <th>Product_Info_2</th>\n",
       "      <th>bayes_Product_Info_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>D3</td>\n",
       "      <td>5.528438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>A1</td>\n",
       "      <td>6.217039</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>E1</td>\n",
       "      <td>6.175437</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>D4</td>\n",
       "      <td>6.322527</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>D2</td>\n",
       "      <td>5.288024</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  Response Product_Info_2  bayes_Product_Info_2\n",
       "0   2         8             D3              5.528438\n",
       "1   5         4             A1              6.217039\n",
       "2   6         8             E1              6.175437\n",
       "3   7         8             D4              6.322527\n",
       "4   8         8             D2              5.288024"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_BayesEncodingKfold.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Id</th>\n",
       "      <th>Response</th>\n",
       "      <th>Product_Info_2</th>\n",
       "      <th>loo_Product_Info_2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>D3</td>\n",
       "      <td>5.511082</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>A1</td>\n",
       "      <td>6.147173</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>8</td>\n",
       "      <td>E1</td>\n",
       "      <td>6.164431</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>D4</td>\n",
       "      <td>6.390441</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>D2</td>\n",
       "      <td>5.280106</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Id  Response Product_Info_2  loo_Product_Info_2\n",
       "0   2         8             D3            5.511082\n",
       "1   5         4             A1            6.147173\n",
       "2   6         8             E1            6.164431\n",
       "3   7         8             D4            6.390441\n",
       "4   8         8             D2            5.280106"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_LOOEncodingKfold.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "for more detailed explanation about parameters, please check online documentation: http://hccencoding-project.readthedocs.io/en/latest/"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}