diff --git a/docs/images/equalfrequencydiscretisation.png b/docs/images/equalfrequencydiscretisation.png index febbf5ce8..9995229e6 100644 Binary files a/docs/images/equalfrequencydiscretisation.png and b/docs/images/equalfrequencydiscretisation.png differ diff --git a/docs/images/equalfrequencydiscretisation_gaussian.png b/docs/images/equalfrequencydiscretisation_gaussian.png new file mode 100644 index 000000000..423801207 Binary files /dev/null and b/docs/images/equalfrequencydiscretisation_gaussian.png differ diff --git a/docs/images/equalfrequencydiscretisation_skewed.png b/docs/images/equalfrequencydiscretisation_skewed.png new file mode 100644 index 000000000..8cbf50caf Binary files /dev/null and b/docs/images/equalfrequencydiscretisation_skewed.png differ diff --git a/docs/images/equalwidthdiscretisation.png b/docs/images/equalwidthdiscretisation.png index e30427d40..a90b01dbf 100644 Binary files a/docs/images/equalwidthdiscretisation.png and b/docs/images/equalwidthdiscretisation.png differ diff --git a/docs/user_guide/discretisation/EqualFrequencyDiscretiser.rst b/docs/user_guide/discretisation/EqualFrequencyDiscretiser.rst index be23124eb..5fa9d6ce2 100644 --- a/docs/user_guide/discretisation/EqualFrequencyDiscretiser.rst +++ b/docs/user_guide/discretisation/EqualFrequencyDiscretiser.rst @@ -5,122 +5,377 @@ EqualFrequencyDiscretiser ========================= -The :class:`EqualFrequencyDiscretiser()` sorts continuous numerical variables into -contiguous equal frequency intervals, that is, intervals that contain approximately the -same proportion of observations. The limits of the intervals are calculated according -to percentiles or quantiles utilising `pandas.qcut()`. You decide the number of -intervals. +Equal frequency discretization consists of dividing continuous attributes into equal-frequency bins. These bins +contain roughly the same number of observations, with boundaries set at specific quantile values determined by the desired +number of bins. -**A note on number of intervals** +Equal frequency discretization ensures a uniform distribution of data points across the range of values, enhancing the +handling of skewed data and outliers. -Common values are 5 and 10. Note that if the variable is highly skewed or not continuous -smaller intervals maybe required. Otherwise, the transformer will introduce np.nan. +Discretization is a common data preprocessing technique used in data science. It's also known as binning data (or simply "binning"). -The :class:`EqualFrequencyDiscretiser()` works only with numerical variables. A list of -variables can be indicated, or the discretiser will automatically select all numerical -variables in the train set. +Advantages and Limitations +-------------------------- -**Example** +Equal frequency discretization has some advantages and shortcomings: -Let's look at an example using the House Prices Dataset (more details about the -dataset :ref:`here `). +Advantages +~~~~~~~~~~ -Let's load the house prices dataset and separate it into train and test sets: +Some advantages of equal frequency binning: + +- **Algorithm Efficiency:** Enhances the performance of data mining and machine learning algorithms by providing a simplified representation of the dataset. +- **Outlier Management:** Efficiently mitigates the effect of outliers by grouping them into the extreme bins. +- **Data Smoothing:** Helps smooth the data, reduces noise, and improves the model's ability to generalize. +- **Improved value distribution:** Returns an uniform distribution of values across the value range. + +Equal frequency discretization improves the data distribution, **optimizing the spread of values**. This is particularly +beneficial for datasets with skewed distributions (see the Python example code). + +Limitations +~~~~~~~~~~~ + +On the other hand, equal frequency binning can lead to a loss of information by aggregating data into broader categories. +This is particularly concerning if the data in the same bin has predictive information about the target. + +Let's consider a binary classifier task using a decision tree model. A bin with a high proportion of both target categories +would potentially impact the model's performance in this scenario. + +EqualFrequencyDiscretiser +------------------------- + +Feature-engine's :class:`EqualFrequencyDiscretiser` applies equal frequency discretization to numerical variables. It uses +the `pandas.qcut()` function under the hood, to determine the interval limits. + +You can specify the variables to be discretized by passing their names in a list when you set up the transformer. Alternatively, +:class:`EqualFrequencyDiscretiser` will automatically infer the data types to compute the interval limits for all numeric variables. + +**Optimal number of intervals:** With :class:`EqualFrequencyDiscretiser`, the user defines the number of bins. Smaller intervals +may be required if the variable is highly skewed or not continuous. + +**Integration with scikit-learn:** :class:`EqualFrequencyDiscretiser` and all other feature-engine transformers seamlessly integrate +with scikit-learn `pipelines `_. + +Python code example +------------------- + +In this section, we'll show the main functionality of :class:`EqualFrequencyDiscretiser` + +Load dataset +~~~~~~~~~~~~ + +In this example, we'll use the Ames House Prices' Dataset. First, let's load the dataset and split it into train and +test sets: .. code:: python - import numpy as np - import pandas as pd import matplotlib.pyplot as plt + from sklearn.datasets import fetch_openml from sklearn.model_selection import train_test_split from feature_engine.discretisation import EqualFrequencyDiscretiser # Load dataset - data = data = pd.read_csv('houseprice.csv') + X, y = fetch_openml(name='house_prices', version=1, return_X_y=True, as_frame=True) + X.set_index('Id', inplace=True) # Separate into train and test sets - X_train, X_test, y_train, y_test = train_test_split( - data.drop(['Id', 'SalePrice'], axis=1), - data['SalePrice'], test_size=0.3, random_state=0) + X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42) + + +Equal-frequency Discretization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Now we want to discretise the 2 variables indicated below into 10 intervals of equal -number of observations: +In this example, let's discretize two variables, LotArea and GrLivArea, into 10 intervals of approximately equal +number of observations. .. code:: python + + # List the target numeric variables to be transformed + TARGET_NUMERIC_FEATURES= ['LotArea','GrLivArea'] - # set up the discretisation transformer - disc = EqualFrequencyDiscretiser(q=10, variables=['LotArea', 'GrLivArea']) + # Set up the discretization transformer + disc = EqualFrequencyDiscretiser(q=10, variables=TARGET_NUMERIC_FEATURES) - # fit the transformer + # Fit the transformer disc.fit(X_train) -With `fit()` the transformer learns the boundaries of each interval. Then, we can go -ahead and sort the values into the intervals: -.. code:: python - - # transform the data - train_t= disc.transform(X_train) - test_t= disc.transform(X_test) +Note that if we do not specify the variables (default=`None`), :class:`EqualFrequencyDiscretiser` will automatically +infer the data types to compute the interval limits for all numeric variables. -The `binner_dict_` stores the interval limits identified for each variable. +With the `fit()` method, the discretizer learns the bin boundaries and saves them into a dictionary so we can use them +to transform unseen data: .. code:: python + # Learned limits for each variable disc.binner_dict_ + .. code:: python {'LotArea': [-inf, - 5007.1, - 7164.6, - 8165.700000000001, - 8882.0, - 9536.0, - 10200.0, - 11046.300000000001, - 12166.400000000001, - 14373.9, + 5000.0, + 7105.6, + 8099.200000000003, + 8874.0, + 9600.0, + 10318.400000000001, + 11173.5, + 12208.2, + 14570.699999999999, inf], 'GrLivArea': [-inf, - 912.0, - 1069.6000000000001, - 1211.3000000000002, - 1344.0, - 1479.0, - 1603.2000000000003, - 1716.0, + 918.5, + 1080.4, + 1218.0, + 1348.4, + 1476.5, + 1601.6000000000001, + 1717.6999999999998, 1893.0000000000005, 2166.3999999999996, inf]} -With equal frequency discretisation, each bin contains approximately the same number of observations. +Note that the lower and upper boundaries are set to -inf and inf, respectively. his behavior ensures that the transformer +will be able to allocate to the extreme bins values that are smaller or greater than the observed minimum and maximum +values in the training set. + +:class:`EqualFrequencyDiscretiser` will not work in the presence of missing values. Therefore, we should either remove or +impute missing values before fitting the transformer. .. code:: python - train_t.groupby('GrLivArea')['GrLivArea'].count().plot.bar() - plt.ylabel('Number of houses') + # Transform the data + train_t = disc.transform(X_train) + test_t = disc.transform(X_test) -We can see below that the intervals contain approximately the same number of -observations. -.. image:: ../../images/equalfrequencydiscretisation.png +Let's visualize the first rows of the raw data and the transformed data: -| +.. code:: python + + # Raw data + print(X_train[TARGET_NUMERIC_FEATURES].head()) + +Here we see the original variables: + +.. code:: python + + LotArea GrLivArea + Id + 136 10400 1682 + 1453 3675 1072 + 763 8640 1547 + 933 11670 1905 + 436 10667 1661 + +.. code:: python + + # Transformed data + print(train_t[TARGET_NUMERIC_FEATURES].head()) + +Here we observe the variables after discretization: + +.. code:: python + + LotArea GrLivArea + Id + 136 6 6 + 1453 0 1 + 763 3 5 + 933 7 8 + 436 6 6 + + +The transformed data now contains discrete values corresponding to the ordered computed buckets (0 being the first and q-1 the last). + +Now, let's visualize the plots for equal-width intervals with a histogram and the transformed data with equal-frequency discretiser: + +.. code:: python + + # Instantiate a figure with two axes + fig, axes = plt.subplots(ncols=2, figsize=(10,5)) + + # Plot raw distribution + X_train['GrLivArea'].plot.hist(bins=disc.q, ax=axes[0]) + axes[0].set_title('Raw data with equal width binning') + axes[0].set_xlabel('GrLivArea') + + # Plot transformed distribution + train_t['GrLivArea'].value_counts().sort_index().plot.bar(ax=axes[1]) + axes[1].set_title('Transformed data with equal frequency binning') + + plt.tight_layout(w_pad=2) + plt.show() + +As we see in the following image, the intervals contain approximately the same number of observations: + +.. image:: ../../images/equalfrequencydiscretisation_gaussian.png + + +Finally, as the default value for the `return_object` parameter is `False`, the transformer outputs integer variables: + +.. code:: python + + train_t[TARGET_NUMERIC_FEATURES].dtypes + + +.. code:: python + + LotArea int64 + GrLivArea int64 + dtype: object + + +Return variables as object +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Categorical encoders in Feature-engine are designed to work by default with variables of type object. Therefore, to further +encode the discretised output with Feature-engine, we can set `return_object=True` instead. This will return the transformed +variables as object. + +Let's say we want to obtain monotonic relationships between the variable and the target. We can do that seamlessly by setting +`return_object` to True. A tutorial of how to use this functionality is available +`here `_. + +Return bin boundaries +~~~~~~~~~~~~~~~~~~~~~ + +If we want to output the intervals limits instead of integers, we can set `return_boundaries` to `True`: + +.. code:: python + + # Set up the discretization transformer + disc = EqualFrequencyDiscretiser( + q=10, + variables=TARGET_NUMERIC_FEATURES, + return_boundaries=True) + + # Fit the transformer + disc.fit(X_train) + + # Transform test set & visualize limit + test_t = disc.transform(X_test) + + # Visualize output (boundaries) + print(test_t[TARGET_NUMERIC_FEATURES].head()) + +The transformed variables now show the interval limits in the output. We can immediately see that the bin width for these +intervals varies. In other words, they don't have the same width, contrarily to what we see with :ref:`equal width discretization `. + +Unlike the variables discretized into integers, these variables cannot be used to train machine learning models; however, +they are still highly helpful for data analysis in this format, and they may be sent to any Feature-engine encoder for +additional processing. + +.. code:: python + + LotArea GrLivArea + Id + 893 (8099.2, 8874.0] (918.5, 1080.4] + 1106 (12208.2, 14570.7] (2166.4, inf] + 414 (8874.0, 9600.0] (918.5, 1080.4] + 523 (-inf, 5000.0] (1601.6, 1717.7] + 1037 (12208.2, 14570.7] (1601.6, 1717.7] + + +Binning skewed data +~~~~~~~~~~~~~~~~~~~ + +Let's now show the benefits of equal frequency discretization for skewed variables. We'll +start by importing the libraries and classes: + +.. code:: python + + import numpy as np + import pandas as pd + import matplotlib.pyplot as plt + from feature_engine.discretisation import EqualFrequencyDiscretiser + +Now, we'll create a toy dataset with a variable that is normally distributed and another +one that is skewed: + +.. code:: python + + # Set seed for reproducibility + np.random.seed(42) + + # Generate a normally distributed data + normal_data = np.random.normal(loc=0, scale=1, size=1000) + + # Generate a right-skewed data using exponential distribution + skewed_data = np.random.exponential(scale=1, size=1000) + + # Create dataframe with simulated data + X = pd.DataFrame({'feature1': normal_data, 'feature2': skewed_data}) + +Let's discretize both variables into 5 equal frequency bins: + +.. code:: python + + # Instantiate discretizer + disc = EqualFrequencyDiscretiser(q=5) + + # Transform simulated data + X_transformed = disc.fit_transform(X) + +Let's plot the original distribution and the distribution after discretization for the variable that was normally +distributed: + +.. code:: python + + fig, axes = plt.subplots(1, 2, figsize=(12, 4)) + + axes[0].hist(X.feature1, bins=disc.q) + axes[0].set(xlabel='feature1', ylabel='count', title='Raw data') + + X_transformed.feature1.value_counts().sort_index().plot.bar(ax=axes[1]) + axes[1].set_title('Transformed data') + + plt.suptitle('Normal distributed data', weight='bold', size='large', y=1.05) + + plt.show() + +In the following image, we see that after the discretization there is an even distribution of the values across +the value range, hence, the variable does no look normally distributed any more. + +.. image:: ../../images/equalfrequencydiscretisation_gaussian.png + +Let's now plot the original distribution and the distribution after discretization for the variable that was skewed: + +.. code:: python + + fig, axes = plt.subplots(1, 2, figsize=(12, 4)) + + axes[0].hist(X.feature2, bins=disc.q) + axes[0].set(xlabel='feature2', ylabel='count', title='Raw data') + + X_transformed.feature2.value_counts().sort_index().plot.bar(ax=axes[1]) + axes[1].set_title('Transformed data') + + plt.suptitle('Skewed distributed data', weight='bold', size='large', y=1.05) + + plt.show() + +In the following image, we see that after the discretization there is an even distribution of the values across +the value range. + +.. image:: ../../images/equalfrequencydiscretisation_skewed.png + +See Also +-------- + +For alternative binning techniques, check out the following resources: + +- Further feature-engine :ref:`discretizers / binning methods ` +- Scikit-learn's `KBinsDiscretizer `_. -**Discretisation plus encoding** +Check out also: -If we return the interval values as integers, the discretiser has the option to return -the transformed variable as integer or as object. Why would we want the transformed -variables as object? +- `Pandas cut `_. -Categorical encoders in Feature-engine are designed to work with variables of type -object by default. Thus, if you wish to encode the returned bins further, say to try and -obtain monotonic relationships between the variable and the target, you can do so -seamlessly by setting `return_object` to True. You can find an example of how to use -this functionality `here `_. Additional resources -------------------- @@ -128,7 +383,7 @@ Additional resources Check also for more details on how to use this transformer: - `Jupyter notebook `_ -- `Jupyter notebook - Discretiser plus Weight of Evidence encoding `_ +- `Jupyter notebook - Discretizer plus Weight of Evidence encoding `_ For more details about this and other feature engineering methods check out these resources: diff --git a/docs/user_guide/discretisation/EqualWidthDiscretiser.rst b/docs/user_guide/discretisation/EqualWidthDiscretiser.rst index 5c2cc42a0..c937d43db 100644 --- a/docs/user_guide/discretisation/EqualWidthDiscretiser.rst +++ b/docs/user_guide/discretisation/EqualWidthDiscretiser.rst @@ -5,77 +5,118 @@ EqualWidthDiscretiser ===================== -The :class:`EqualWidthDiscretiser()` sorts the variable values into contiguous intervals -of equal size. The size of the interval is calculated as: +Equal width discretization consist of dividing continuous variables into intervals of equal width, calculated +using the following formula: -( max(X) - min(X) ) / bins +.. math:: -where bins, which is the number of intervals, should be determined by the user. The -interval limits are determined using `pandas.cut()`. + bin_{width} = ( max(X) - min(X) ) / bins -**A note on number of intervals** +Here, `bins` is the number of intervals specified by the user and `max(X)` and `min(X)` are the minimum and maximum values +of the variable to discretize. -Common values are 5 and 10. Note that if the variable is highly skewed or not continuous -smaller intervals maybe required. Otherwise, the transformer will introduce np.nan. +Discretization is a common data preprocessing technique used in data science. It's also known as data binning (or simply +"binning"). -The :class:`EqualWidthDiscretiser()` works only with numerical variables. A list of -variables to discretise can be indicated, or the discretiser will automatically select -all numerical variables in the train set. +Advantages and Limitations +-------------------------- -**Example** +Equal binning discretization has some advantages and also shortcomings. -Let's look at an example using the House Prices Dataset (more details about the -dataset :ref:`here `). +Advantages +~~~~~~~~~~ -Let's load the house prices dataset and separate it into train and test sets: +Some advantages of equal width binning: + +- **Algorithm Efficiency:** Enhances the performance of data mining and machine learning algorithms by providing a simplified representation of the dataset. +- **Outlier Management:** Efficiently mitigates the effect of outliers by grouping them into the extreme bins, thus preserving the integrity of the main data distribution. +- **Data Smoothing:** Helps smooth the data, reduces noise, and improves the model's ability to generalize. + +Limitations +~~~~~~~~~~~ + +On the other hand, equal width discretzation can lead to a loss of information by aggregating data into broader categories. +This is particularly concerning if the data in the same bin has predictive information about the target. + +Let's consider a binary classifier task using a decision tree model. A bin with a high proportion of both target categories would +potentially impact the model's performance in this scenario. + +EqualWidthDiscretiser +--------------------- + +Feture-engine's :class:`EqualWidthDiscretiser()` applies equal width discretization to numerical variables. It uses +the `pandas.cut()` function under the hood to find the interval limits and then sort the continuous variables into +the bins. + +You can specify the variables to be discretized by passing their names in a list when you set up the transformer. Alternatively, +:class:`EqualWidthDiscretiser()` will automatically infer the data types to compute the interval limits for all numeric +variables. + +**Optimal number of intervals:** With :class:`EqualWidthDiscretiser()`, the user defines the number of bins. Smaller intervals +may be required if the variable is highly skewed or not continuous. + +**Integration with scikit-learn:** :class:`EqualWidthDiscretiser()` and all other Feature-engine transformers seamlessly +integrate with scikit-learn `pipelines `_. + +Python code example +------------------- + +In this section, we'll show the main functionality of :class:`EqualWidthDiscretiser()`. + +Load dataset +~~~~~~~~~~~~ + +In this example, we'll use the Ames House Prices' Dataset. First, let's load the dataset and split it into train and +test sets: .. code:: python - import numpy as np - import pandas as pd import matplotlib.pyplot as plt + from sklearn.datasets import fetch_openml from sklearn.model_selection import train_test_split - from feature_engine.discretisation import EqualWidthDiscretiser + from feature_engine.discretisation import EqualFrequencyDiscretiser # Load dataset - data = data = pd.read_csv('houseprice.csv') + X, y = fetch_openml(name='house_prices', version=1, return_X_y=True, as_frame=True) + X.set_index('Id', inplace=True) # Separate into train and test sets - X_train, X_test, y_train, y_test = train_test_split( - data.drop(['Id', 'SalePrice'], axis=1), - data['SalePrice'], test_size=0.3, random_state=0) + X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42) -Now we want to discretise the 2 variables indicated below into 10 intervals of equal -width: +Equal-width Discretization +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In this example, let's discretize two variables, LotArea and GrLivArea, into 10 intervals of equal width: .. code:: python - # set up the discretisation transformer - disc = EqualWidthDiscretiser(bins=10, variables=['LotArea', 'GrLivArea']) + # List the target numeric variables for equal-width discretization + TARGET_NUMERIC_FEATURES= ['LotArea','GrLivArea'] - # fit the transformer - disc.fit(X_train) + # Set up the discretization transformer + disc = EqualWidthDiscretiser(bins=10, variables=TARGET_NUMERIC_FEATURES) -With `fit()` the transformer learns the boundaries of each interval. Then, we can go -ahead and sort the values into the intervals: + # Fit the transformer + disc.fit(X_train) -.. code:: python - # transform the data - train_t= disc.transform(X_train) - test_t= disc.transform(X_test) +Note that if we do not specify the variables (default=`None`), :class:`EqualWidthDiscretiser` will automatically infer +the data types to compute the interval limits for all numeric variables. -The `binner_dict_` stores the interval limits identified for each variable. +With the `fit()` method, the discretizer learns the bin boundaries and saves them into a dictionary so we can use them +to transform unseen data: .. code:: python + # Learned limits for each variable disc.binner_dict_ + .. code:: python - 'LotArea': [-inf, + {'LotArea': [-inf, 22694.5, 44089.0, 65483.5, @@ -87,41 +128,163 @@ The `binner_dict_` stores the interval limits identified for each variable. 193850.5, inf], 'GrLivArea': [-inf, - 768.2, - 1202.4, - 1636.6, - 2070.8, - 2505.0, - 2939.2, - 3373.4, - 3807.6, - 4241.799999999999, + 864.8, + 1395.6, + 1926.3999999999999, + 2457.2, + 2988.0, + 3518.7999999999997, + 4049.5999999999995, + 4580.4, + 5111.2, inf]} -With equal width discretisation, each bin does not necessarily contain the same number of observations. + +Note that the lower and upper boundaries are set to -inf and inf, respectively. This behavior ensures that the transformer +will be able to allocate to the extreme bins values that are smaller or greater than the observed minimum and maximum +values in the training set. + +:class:`EqualWidthDiscretiser` will not work in the presence of missing values. Therefore, we should either remove or +impute missing values before fitting the transformer. + +.. code:: python + + # Transform the data (data discretization) + train_t = disc.transform(X_train) + test_t = disc.transform(X_test) + +Let's visualize the first rows of the raw data and the transformed data: + +.. code:: python + + # Raw data + print(X_train[TARGET_NUMERIC_FEATURES].head()) + +Here we see the original variables: .. code:: python - train_t.groupby('GrLivArea')['GrLivArea'].count().plot.bar() + LotArea GrLivArea + Id + 136 10400 1682 + 1453 3675 1072 + 763 8640 1547 + 933 11670 1905 + 436 10667 1661 + + +.. code:: python + + # Transformed data + print(train_t[TARGET_NUMERIC_FEATURES].head()) + +Here we observe the variables after discretization: + +.. code:: python + + LotArea GrLivArea + Id + 136 0 2 + 1453 0 1 + 763 0 2 + 933 0 2 + 436 0 2 + + +The transformed data now contains discrete values corresponding to the ordered computed buckets (0 being the first and +bins-1 the last). + +Now, let's check out the number of observations per bin by creating a bar plot: + +.. code:: python + + train_t['GrLivArea'].value_counts().sort_index().plot.bar() plt.ylabel('Number of houses') + plt.show() -We can see below that the intervals contain different number of observations. +As we see in the following image, the intervals contain different number of observations. It's a similar output to a +histogram: .. image:: ../../images/equalwidthdiscretisation.png | -**Discretisation plus encoding** +Equal width discretization does not improve the spread of values over the value range. If the variable is skewed, it will +still be skewed after the discretization. + +Finally, since the default value for the `return_object` parameter is `False`, the transformer outputs integer variables: + +.. code:: python + + train_t[TARGET_NUMERIC_FEATURES].dtypes + + +.. code:: python + + LotArea int64 + GrLivArea int64 + dtype: object + + +Return variables as object +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Categorical encoders in Feature-engine are designed to work by default with variables of type object. Therefore, to +further encode the discretized output with Feature-engine's encoders, we can set `return_object=True` instead. This will +return the transformed variables as object. + +Let's say we want to obtain monotonic relationships between the variable and the target. We can do that seamlessly by +setting `return_object` to `True`. A tutorial of how to use this functionality is available +`here `_. + +Return bin boundaries +~~~~~~~~~~~~~~~~~~~~~ + +If we want to output the intervals limits instead of integers, we can set `return_boundaries` to `True`: + +.. code:: python + + # Set up the discretization transformer + disc = EqualFrequencyDiscretiser( + bins=10, + variables=TARGET_NUMERIC_FEATURES, + return_boundaries=True) + + # Fit the transformer + disc.fit(X_train) + + # Transform test set & visualize limit + test_t = disc.transform(X_test) + + # Visualize output (boundaries) + print(test_t[TARGET_NUMERIC_FEATURES].head()) + +In the following output we see that the transformed variables now display the interval limits. While we can't use these +variables to train machine learning models, as opposed to the variables discretized into integers, they are very useful +in this format for data analysis, and they can also be passed on to any Feature-engine encoder for further processing. + +.. code:: python + + LotArea GrLivArea + Id + 893 (-inf, 22694.5] (864.8, 1395.6] + 1106 (-inf, 22694.5] (2457.2, 2988.0] + 414 (-inf, 22694.5] (864.8, 1395.6] + 523 (-inf, 22694.5] (1395.6, 1926.4] + 1037 (-inf, 22694.5] (1395.6, 1926.4] + + +See Also +-------- + +For alternative binning techniques, check out the following resources: + +- Further feature-engine :ref:`discretizers / binning methods ` +- Scikit-learn's `KBinsDiscretizer `_. -If we return the interval values as integers, the discretiser has the option to return -the transformed variable as integer or as object. Why would we want the transformed -variables as object? +Check out also: -Categorical encoders in Feature-engine are designed to work with variables of type -object by default. Thus, if you wish to encode the returned bins further, say to try and -obtain monotonic relationships between the variable and the target, you can do so -seamlessly by setting `return_object` to True. You can find an example of how to use -this functionality `here `_. +- `Pandas cut `_. Additional resources -------------------- @@ -129,7 +292,7 @@ Additional resources Check also for more details on how to use this transformer: - `Jupyter notebook `_ -- `Jupyter notebook - Discretiser plus Ordinal encoding `_ +- `Jupyter notebook - Discretizer plus Ordinal encoding `_ For more details about this and other feature engineering methods check out these resources: @@ -178,4 +341,4 @@ Or read our book: | Both our book and course are suitable for beginners and more advanced data scientists -alike. By purchasing them you are supporting Sole, the main developer of Feature-engine. \ No newline at end of file +alike. By purchasing them you are supporting Sole, the main developer of Feature-engine. diff --git a/docs/user_guide/discretisation/index.rst b/docs/user_guide/discretisation/index.rst index 9a7fc91c6..559dc751a 100644 --- a/docs/user_guide/discretisation/index.rst +++ b/docs/user_guide/discretisation/index.rst @@ -1,3 +1,5 @@ +.. _discretization_transformers: + .. -*- mode: rst -*- Discretisation