Skip to content

improve count encoder userguide #742

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
solegalli merged 2 commits into from
Apr 19, 2024
Merged

improve count encoder userguide #742

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d980723
improve count encoder userguide
solegalli Apr 19, 2024
39be5e4
remove url|
solegalli Apr 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions docs/conf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -189,10 +189,10 @@
# intersphinx configuration
intersphinx_mapping = {
"python": ("https://docs.python.org/{.major}".format(sys.version_info), None),
"numpy": ("https://docs.scipy.org/doc/numpy/", None),
# "numpy": ("https://numpy.org/doc/", None),
"pandas": ("https://pandas.pydata.org/docs/", None),
"scipy": ("https://docs.scipy.org/doc/scipy/reference", None),
"matplotlib": ("https://matplotlib.org/", None),
"scipy": ("https://docs.scipy.org/doc/scipy/", None),
# "matplotlib": ("https://matplotlib.org/", None),
"sklearn": ("https://scikit-learn.org/stable/", None),
}

Expand Down
238 changes: 194 additions & 44 deletions docs/user_guide/encoding/CountFrequencyEncoder.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,80 @@
CountFrequencyEncoder
=====================

The :class:`CountFrequencyEncoder()` replaces categories by either the count or the
percentage of observations per category. For example in the variable colour, if 10
observations are blue, blue will be replaced by 10. Alternatively, if 10% of the
observations are blue, blue will be replaced by 0.1.
Count encoding and frequency encoding are 2 categorical encoding techniques that were
commonly used during data preprocessing in Kaggle's data science competitions, even when
their predictive value is not immediately obvious.

Let's look at an example using the Titanic Dataset.
Count encoding consists of replacing the categories of categorical features by their
counts, which are estimated from the training set. For example, in the variable color,
if 10 observations are blue and 5 observations are red, blue will be replaced by 10 and
red by 5.

First, let's load the data and separate it into train and test:
Frequency encoding consists of replacing the labels of categorical data with their
frequency, which is also estimated from the training set. Then, in the variable City,
if London appears in 10% of the observations and Bristol in 1%, London will be replaced
by 0.1 and Bristol with 0.01.

Count and frequency encoding in machine learning
------------------------------------------------

We'd use count encoding or frequency encoding when we think that the representation of
the categories in the dataset has some sort of predictive value. To be honest, the only
example that I can think of where count encoding could be useful is in sales forecasting
or sales data analysis scenarios, where the count of a product or an item represents its
popularity. In other words, we may be more likely to sell a product with a high count.

Count encoding and frequency encoding can be suitable for categorical variables with high
cardinality because these types of categorical encoding will cause what is called
collisions: categories that are present in a similar number of observations will be
replaced with similar, if not the same values, which reduces the variability.

This, of course, can result in the loss of information by placing two categories that
are otherwise different in the same pot. But on the other hand, if we are using count
encoding or frequency encoding, we have reasons to believe that the count or the frequency
are a good indicator of predictive performance or somehow capture data insight, so that
categories with similar counts would show similar patterns or behaviors.

Count and Frequency encoding with Feature-engine
------------------------------------------------

The :class:`CountFrequencyEncoder()` replaces categories of categorical features by
either the count or the percentage of observations each category shows in the training set.

With :class:`CountFrequencyEncoder()` we can automatically encode all categorical
features in the dataset, or only a subset of them, by passing the variable names in a
list to the `variables` parameter when we set up the encoder.

By default, :class:`CountFrequencyEncoder()` will encode only categorical data. If we
want to encode numerical values, we need to explicitly say so by setting the parameter
`ignore_format` to True.

Count and frequency encoding with unseen categories
---------------------------------------------------

When we learn mappings from strings to numbers, either with count encoding or other
encoding techniques like ordinal encoding or target encoding, we do so by observing the
categories in the training set. Hence, we won't have mappings for categories that appear
only in the test set. These are the so-called "unseen categories."

When encountering unseen categories, :class:`CountFrequencyEncoder()` will ignore them
by default, which means that unseen categories will be replaced with missing values.
We can instruct the encoder to raise an error when a new category is encountered, or
alternatively, to encode unseen categories with zero.

Count encoding vs other encoding methods
----------------------------------------

Count and frequency encoding, similar to ordinal encoding and contrarily to one-hot
encoding, feature hashing, or binary encoding, does not increase the dataset dimensionality.
From one categorical variable, we obtain one numerical feature.

Python example
--------------

Let's examine the functionality of :class:`CountFrequencyEncoder()` by using the Titanic
dataset. We'll start by loading the libraries and functions, loading the dataset, and then
splitting it into a training and a testing set.

.. code:: python

Expand All @@ -33,7 +99,7 @@ First, let's load the data and separate it into train and test:

print(X_train.head())

We see the resulting data below:
We see the resulting dataframe with the predictor variables below:

.. code:: python

Expand All @@ -44,50 +110,50 @@ We see the resulting data below:
1193 3 male 29.881135 0 0 7.7250 M Q
686 3 female 22.000000 0 0 7.7250 M Q

Now, we set up the :class:`CountFrequencyEncoder()` to replace the categories by their
frequencies, only in the 3 indicated variables:
This dataset has three obvious categorical features: cabin, embarked, and sex, and in
addition, pclass could also be handled as a categorical.

Count encoding
~~~~~~~~~~~~~~

We'll start by encoding the three categorical variables using their counts, that is,
replacing the strings with the number of times each category is present in the training
dataset.

.. code:: python

encoder = CountFrequencyEncoder(
encoding_method='frequency',
variables=['cabin', 'pclass', 'embarked'],
ignore_format=True,
encoding_method='count',
variables=['cabin', 'sex', 'embarked'],
)

encoder.fit(X_train)

With `fit()` the encoder learns the frequencies of each category, which are stored in
its `encoder_dict_` parameter:
With `fit()`, the count encoder learns the counts of each category. We can inspect the
counts as follows:

.. code:: python

encoder.encoder_dict_
encoder.encoder_dict_

In the `encoder_dict_` we find the frequencies for each one of the categories of each
variable that we want to encode. This way, we can map the original value to the new
value.
We see the counts of each category for each of the three variables in the following output:

.. code:: python

{'cabin': {'M': 0.7663755458515283,
'C': 0.07751091703056769,
'B': 0.04585152838427948,
'E': 0.034934497816593885,
'D': 0.034934497816593885,
'A': 0.018558951965065504,
'F': 0.016375545851528384,
'G': 0.004366812227074236,
'T': 0.001091703056768559},
'pclass': {3: 0.5436681222707423,
1: 0.25109170305676853,
2: 0.2052401746724891},
'embarked': {'S': 0.7117903930131004,
'C': 0.19541484716157206,
'Q': 0.0906113537117904,
'Missing': 0.002183406113537118}}

We can now go ahead and replace the original strings with the numbers:

{'cabin': {'M': 702,
'C': 71,
'B': 42,
'E': 32,
'D': 32,
'A': 17,
'F': 15,
'G': 4,
'T': 1},
'sex': {'male': 581, 'female': 335},
'embarked': {'S': 652, 'C': 179, 'Q': 83, 'Missing': 2}}

Now, we can go ahead and encode the variables:

.. code:: python

Expand All @@ -96,16 +162,91 @@ We can now go ahead and replace the original strings with the numbers:

print(train_t.head())

Below we see the that the original variables were replaced with the frequencies:
We see the resulting dataframe where the categorical features are now replaced with
integer values corresponding to the category counts:

.. code:: python

pclass sex age sibsp parch fare cabin embarked
501 2 335 13.000000 0 1 19.5000 702 652
588 2 335 4.000000 1 1 23.0000 702 652
402 2 335 30.000000 1 0 13.8583 702 179
1193 3 581 29.881135 0 0 7.7250 702 83
686 3 335 22.000000 0 0 7.7250 702 83

We can now use the encoded dataframes to train machine learning models.

Frequency encoding
~~~~~~~~~~~~~~~~~~

Let's now perform frequency encoding. We'll encode 2 categorical and 1 numerical variable,
hence, we need to set the encoder to ignore the variable's type:

.. code:: python

encoder = CountFrequencyEncoder(
encoding_method='frequency',
variables=['cabin', 'pclass', 'embarked'],
ignore_format=True,
)


Now, we fit the frequency encoder to the train set and transform it straightaway, and
then we transform the test set:

.. code:: python

t_train = encoder.fit_transform(X_train)
t_test = encoder.transform(X_test)

test.head()

In the following output we see the transformed dataframe, where the categorical features
are now encoded into their frequencies:

.. code:: python

pclass sex age sibsp parch fare cabin embarked
501 0.205240 female 13.000000 0 1 19.5000 0.766376 0.711790
588 0.205240 female 4.000000 1 1 23.0000 0.766376 0.711790
402 0.205240 female 30.000000 1 0 13.8583 0.766376 0.195415
1193 0.543668 male 29.881135 0 0 7.7250 0.766376 0.090611
686 0.543668 female 22.000000 0 0 7.7250 0.766376 0.090611
1139 0.543668 male 38.000000 0 0 7.8958 0.766376 0.71179
533 0.205240 female 21.000000 0 1 21.0000 0.766376 0.71179
459 0.205240 male 42.000000 1 0 27.0000 0.766376 0.71179
1150 0.543668 male 29.881135 0 0 14.5000 0.766376 0.71179
393 0.205240 male 25.000000 0 0 31.5000 0.766376 0.71179

With `fit()` the encoder learns the frequencies of each category, which are stored in
its `encoder_dict_` parameter. We can inspect them like this:

.. code:: python

encoder.encoder_dict_

In the `encoder_dict_` we find the frequencies for each one of the unique categories of
each variable to encode. This way, we can map the original value to the new value.

.. code:: python

{'cabin': {'M': 0.7663755458515283,
'C': 0.07751091703056769,
'B': 0.04585152838427948,
'E': 0.034934497816593885,
'D': 0.034934497816593885,
'A': 0.018558951965065504,
'F': 0.016375545851528384,
'G': 0.004366812227074236,
'T': 0.001091703056768559},
'pclass': {3: 0.5436681222707423,
1: 0.25109170305676853,
2: 0.2052401746724891},
'embarked': {'S': 0.7117903930131004,
'C': 0.19541484716157206,
'Q': 0.0906113537117904,
'Missing': 0.002183406113537118}}

We can now use these dataframes to train machine learning algorithms.

With the method `inverse_transform`, we can transform the encoded dataframes back to their
original representation, that is, we can replace the encoding with the original categorical
values.

Additional resources
--------------------
Expand All @@ -126,6 +267,15 @@ For more details about this and other feature engineering methods check out thes

Feature Engineering for Machine Learning

.. figure:: ../../images/fetsf.png
:width: 300
:figclass: align-center
:align: right
:target: https://www.trainindata.com/p/feature-engineering-for-forecasting

Feature Engineering for Time Series Forecasting


|
|
|
Expand All @@ -137,7 +287,7 @@ For more details about this and other feature engineering methods check out thes
|
|

Or read our book:
Our book:

.. figure:: ../../images/cookbook.png
:width: 200
Expand All @@ -161,5 +311,5 @@ Or read our book:
|
|

Both our book and course are suitable for beginners and more advanced data scientists
Both our book and courses are suitable for beginners and more advanced data scientists
alike. By purchasing them you are supporting Sole, the main developer of Feature-engine.