diff --git a/docs/images/treemonotonicprediction.png b/docs/images/treemonotonicprediction.png new file mode 100644 index 000000000..aac36656a Binary files /dev/null and b/docs/images/treemonotonicprediction.png differ diff --git a/docs/images/treepredictionrounded.png b/docs/images/treepredictionrounded.png new file mode 100644 index 000000000..db9c718ff Binary files /dev/null and b/docs/images/treepredictionrounded.png differ diff --git a/docs/user_guide/discretisation/DecisionTreeDiscretiser.rst b/docs/user_guide/discretisation/DecisionTreeDiscretiser.rst index e4e141320..3f913f194 100644 --- a/docs/user_guide/discretisation/DecisionTreeDiscretiser.rst +++ b/docs/user_guide/discretisation/DecisionTreeDiscretiser.rst @@ -5,138 +5,465 @@ DecisionTreeDiscretiser ======================= -The :class:`DecisionTreeDiscretiser()` replaces numerical variables by discrete, i.e., -finite variables, which values are the predictions of a decision tree. The method is -based on the winning solution of the KDD 2009 competition: +Discretization consists of transforming continuous variables into discrete features by creating +a set of contiguous intervals, or bins, that span the range of the variable values. + +Discretization is a common data preprocessing step in many data science projects, as it simplifies +continuous attributes and has the potential to improve model performance or speed up model training. + +Decision tree discretization +---------------------------- + +Decision trees make decisions based on discrete partitions over continuous features. During +training, a decision tree evaluates all possible feature values to find the best cut-point, that is, +the feature value at which the split maximizes the information gain, or in other words, reduces the +impurity. It repeats the procedure at each node until it allocates all samples to certain leaf +nodes or end nodes. Hence, classification and regression trees can naturally find the optimal limits +of the intervals to maximize class coherence. + +Discretization with decision trees consists of using a decision tree algorithm to identify the optimal +partitions for each continuous variable. After finding the optimal partitions, we sort the variable's +values into those intervals. + +Discretization with decision trees is a supervised discretization method, in that, the interval +limits are found based on class or target coherence. In simpler words, we need the target variable +to train the decision trees. + +Advantages +~~~~~~~~~~ + +- The output returned by the decision tree is monotonically related to the target. +- The tree end nodes, or bins, show decreased entropy, that is, the observations within each bin are more similar among themselves than to those of other bins. + +Limitations +~~~~~~~~~~~ + +- Could cause over-fitting +- We need to tune some of the decision tree parameters to obtain the optimal number of intervals. + + +Decision tree discretizer +------------------------- + +The :class:`DecisionTreeDiscretiser()` applies discretization based on the interval limits found +by decision trees algorithms. It uses decision trees to find the optimal interval limits. Next, +it sorts the variable into those intervals. + +The transformed variable can either have the limits of the intervals as values, an ordinal number +representing the interval into which the value was sorted, or alternatively, the prediction of the +decision tree. In any case, the number of values of the variable will be finite. + +In theory, decision tree discretization creates discrete variables with a monotonic relationship +with the target, and hence, the transformed features would be more suitable to train linear models, +like linear or logistic regression. + +Original idea +------------- + +The method of decision tree discretization is based on the winning solution of the KDD 2009 competition: `Niculescu-Mizil, et al. "Winning the KDD Cup Orange Challenge with Ensemble Selection". JMLR: Workshop and Conference Proceedings 7: 23-34. KDD 2009 `_. -In the original article, each feature in the challenge dataset was re-coded by training -a decision tree of limited depth (2, 3 or 4) using that feature alone, and letting the -tree predict the target. The probabilistic predictions of this decision tree were used -as an additional feature, that was now linearly (or at least monotonically) correlated -with the target. +In the original article, each feature in the dataset was re-coded by training a decision tree of limited +depth (2, 3 or 4) using that feature alone, and letting the tree predict the target. The probabilistic +predictions of this decision tree were used as an additional feature that was now linearly (or at least +monotonically) related with the target. According to the authors, the addition of these new features had a significant impact on the performance of linear models. -**Example** +Code examples +------------- + +In the following sections, we will do decision tree discretization to showcase the functionality of +the :class:`DecisionTreeDiscretiser()`. We will discretize 2 numerical variables of the Ames house +prices dataset using decision trees. -In the following example, we re-code 2 numerical variables using decision trees. +First, we will transform the variables using the predictions of the decision trees, next, we will +return the interval limits, and finally, we will return the bin order. -First we load the data and separate it into train and test: +Discretization with the predictions of the decision tree +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +First we load the data and separate it into a training set and a test set: .. code:: python - import numpy as np - import pandas as pd - import matplotlib.pyplot as plt - from sklearn.model_selection import train_test_split + from sklearn.datasets import fetch_openml + from sklearn.model_selection import train_test_split + + data = fetch_openml(name='house_prices', as_frame=True) + data = data.frame + + X = data.drop(['SalePrice', 'Id'], axis=1) + y = data['SalePrice'] - from feature_engine.discretisation import DecisionTreeDiscretiser - # Load dataset - data = data = pd.read_csv('houseprice.csv') + X_train, X_test, y_train, y_test = train_test_split( + X, y, test_size=0.2, random_state=42) - # Separate into train and test sets - X_train, X_test, y_train, y_test = train_test_split( - data.drop(['Id', 'SalePrice'], axis=1), - data['SalePrice'], test_size=0.3, random_state=0) + print(X_train.head()) -Now we set up the discretiser. We will optimise the decision tree's depth using 3 fold -cross-validation. +In the following output we see the predictor variables of the house prices dataset: .. code:: python - # set up the discretisation transformer - disc = DecisionTreeDiscretiser(cv=3, + MSSubClass MSZoning LotFrontage LotArea Street Alley LotShape \ + 254 20 RL 70.0 8400 Pave NaN Reg + 1066 60 RL 59.0 7837 Pave NaN IR1 + 638 30 RL 67.0 8777 Pave NaN Reg + 799 50 RL 60.0 7200 Pave NaN Reg + 380 50 RL 50.0 5000 Pave Pave Reg + + LandContour Utilities LotConfig ... ScreenPorch PoolArea PoolQC Fence \ + 254 Lvl AllPub Inside ... 0 0 NaN NaN + 1066 Lvl AllPub Inside ... 0 0 NaN NaN + 638 Lvl AllPub Inside ... 0 0 NaN MnPrv + 799 Lvl AllPub Corner ... 0 0 NaN MnPrv + 380 Lvl AllPub Inside ... 0 0 NaN NaN + + MiscFeature MiscVal MoSold YrSold SaleType SaleCondition + 254 NaN 0 6 2010 WD Normal + 1066 NaN 0 5 2009 WD Normal + 638 NaN 0 5 2008 WD Normal + 799 NaN 0 6 2007 WD Normal + 380 NaN 0 5 2010 WD Normal + + [5 rows x 79 columns] + + +We set up the decision tree discretiser to find the optimal intervals using decision trees. + +The :class:`DecisionTreeDiscretiser()` will optimize the depth of the decision tree classifier +or regressor by default and using cross-validation. That's why we need to select the appropriate +metric for the optimization. In this example, we are using decision tree regression, so we select +the mean squared error metric. + +We specify in the `bin_output` that we want to replace the continuous attribute values with the +predictions of the decision tree. + +.. code:: python + + from feature_engine.discretisation import DecisionTreeDiscretiser + + disc = DecisionTreeDiscretiser(bin_output="prediction", + cv=3, scoring='neg_mean_squared_error', variables=['LotArea', 'GrLivArea'], regression=True) - # fit the transformer - disc.fit(X_train, y_train) + disc.fit(X_train, y_train) +The scoring and cv parameter work exactly as those from any scikit-learn estimator. So we can pass +any value that is also valid for those estimators. Check scikit-learn's documentation for more information. -With `fit()` the transformer fits a decision tree per variable. Then, we can go -ahead replace the variable values by the predictions of the trees: +With `fit()` the transformer fits a decision tree for each one of the continuous features. Then, +we can go ahead replace the variable values by the predictions of the trees and display the transformed +variables: .. code:: python - # transform the data - train_t= disc.transform(X_train) - test_t= disc.transform(X_test) + train_t = disc.transform(X_train) + test_t = disc.transform(X_test) + + print(train_t[['LotArea', 'GrLivArea']].head()) + +In this case, the original values were replaced with the predictions of each one of the decision trees: + +.. code:: python + + LotArea GrLivArea + 254 144174.283688 152471.713568 + 1066 144174.283688 191760.966667 + 638 176117.741848 97156.250000 + 799 144174.283688 202178.409091 + 380 144174.283688 202178.409091 + +Decision trees make discrete predictions, that's why we'll see a limited number of values in the +transformed variables: + +.. code:: python + + train_t[['LotArea', 'GrLivArea']].nunique() + +.. code:: python + + LotArea 4 + GrLivArea 16 + dtype: int64 The `binner_dict_` stores the details of each decision tree. .. code:: python - disc.binner_dict_ + disc.binner_dict_ .. code:: python - {'LotArea': GridSearchCV(cv=3, error_score='raise-deprecating', - estimator=DecisionTreeRegressor(criterion='mse', max_depth=None, - max_features=None, - max_leaf_nodes=None, - min_impurity_decrease=0.0, - min_impurity_split=None, - min_samples_leaf=1, - min_samples_split=2, - min_weight_fraction_leaf=0.0, - presort=False, random_state=None, - splitter='best'), - iid='warn', n_jobs=None, param_grid={'max_depth': [1, 2, 3, 4]}, - pre_dispatch='2*n_jobs', refit=True, return_train_score=False, - scoring='neg_mean_squared_error', verbose=0), - 'GrLivArea': GridSearchCV(cv=3, error_score='raise-deprecating', - estimator=DecisionTreeRegressor(criterion='mse', max_depth=None, - max_features=None, - max_leaf_nodes=None, - min_impurity_decrease=0.0, - min_impurity_split=None, - min_samples_leaf=1, - min_samples_split=2, - min_weight_fraction_leaf=0.0, - presort=False, random_state=None, - splitter='best'), - iid='warn', n_jobs=None, param_grid={'max_depth': [1, 2, 3, 4]}, - pre_dispatch='2*n_jobs', refit=True, return_train_score=False, - scoring='neg_mean_squared_error', verbose=0)} + {'LotArea': GridSearchCV(cv=3, estimator=DecisionTreeRegressor(), + param_grid={'max_depth': [1, 2, 3, 4]}, + scoring='neg_mean_squared_error'), + 'GrLivArea': GridSearchCV(cv=3, estimator=DecisionTreeRegressor(), + param_grid={'max_depth': [1, 2, 3, 4]}, + scoring='neg_mean_squared_error')} -With tree discretisation, each bin, that is, each prediction value, does not necessarily -contain the same number of observations. +With decision tree discretization, each bin, that is, each prediction value in this case, does not +necessarily contain the same number of observations. Let's check that out with a visualization: .. code:: python - # with tree discretisation, each bin does not necessarily contain - # the same number of observations. - train_t.groupby('GrLivArea')['GrLivArea'].count().plot.bar() - plt.ylabel('Number of houses') + import matplotlib.pyplot as plt + train_t.groupby('GrLivArea')['GrLivArea'].count().plot.bar() + plt.ylabel('Number of houses') + plt.show() .. image:: ../../images/treediscretisation.png -**Note** +Finally, we can determine if we have a monotonic relationship with the target after the transformation: + +.. code:: python + + plt.scatter(test_t['GrLivArea'], y_test) + plt.xlabel('GrLivArea') + plt.ylabel('Sale Price') + plt.show() + +.. image:: ../../images/treemonotonicprediction.png + +Rounding the prediction value +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Sometimes, the output of the prediction can have multiple values after the comma, which makes the +visualization and interpretation a bit uncomfortable. Fortunately, we can round those values through +the `precision` parameter: + +.. code:: python + + disc = DecisionTreeDiscretiser( + bin_output="prediction", + precision=1, + cv=3, + scoring='neg_mean_squared_error', + variables=['LotArea', 'GrLivArea'], + regression=True) + + disc.fit(X_train, y_train) + + train_t= disc.transform(X_train) + test_t= disc.transform(X_test) + + train_t.groupby('GrLivArea')['GrLivArea'].count().plot.bar() + plt.ylabel('Number of houses') + plt.show() + +.. image:: ../../images/treepredictionrounded.png + +In this example, we are predicting house prices, which is a continuous target. The procedure for +classification models is identical, we just need to set the parameter `regression` to False. + +Discretization with interval limits +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In this section, instead of replacing the original variable values with the predictions of the +decision tree, we will return the limits of the intervals. When returning interval boundaries, +we need to set the precision to a positive integer. + +.. code:: python + + disc = DecisionTreeDiscretiser( + bin_output="boundaries", + precision=3, + cv=3, + scoring='neg_mean_squared_error', + variables=['LotArea', 'GrLivArea'], + regression=True) + + # fit the transformer + disc.fit(X_train, y_train) + +In this case, when we explore the `binner_dict_` attribute, we will see the interval limits instead +of the decision trees: + +.. code:: python + + disc.binner_dict_ + +.. code:: python + + {'LotArea': [-inf, 8637.5, 10924.0, 13848.5, inf], + 'GrLivArea': [-inf, + 749.5, + 808.0, + 1049.0, + 1144.5, + 1199.0, + 1413.0, + 1438.5, + 1483.0, + 1651.5, + 1825.0, + 1969.5, + 2386.0, + 2408.0, + 2661.0, + 4576.0, + inf]} + +The :class:`DecisionTreeDiscretiser()` will use these limits with `pandas.cut` to discretize the +continuous variable values during transform: + +.. code:: python + + train_t = disc.transform(X_train) + test_t = disc.transform(X_test) + + print(train_t[['LotArea', 'GrLivArea']].head()) + +In the following output we see the interval limits into which the values of the continuous attributes were sorted: + +.. code:: python -Our implementation of the :class:`DecisionTreeDiscretiser()` will replace the original -values of the variable by the predictions of the trees. This is not strictly identical -to what the winners of the KDD competition did. They added the predictions of the features -as new variables, while keeping the original ones. + LotArea GrLivArea + 254 (-inf, 8637.5] (1199.0, 1413.0] + 1066 (-inf, 8637.5] (1483.0, 1651.5] + 638 (8637.5, 10924.0] (749.5, 808.0] + 799 (-inf, 8637.5] (1651.5, 1825.0] + 380 (-inf, 8637.5] (1651.5, 1825.0] -More details -^^^^^^^^^^^^ +To train machine learning algorithms we would follow that up with any categorical data encoding method. + +Discretization with ordinal numbers +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +In the last part of this guide, we will replace the variable values with the number of bin into +which the value was sorted. Here, 0 is the first bin, 1 the second, and so on. + +.. code:: python + + disc = DecisionTreeDiscretiser( + bin_output="bin_number", + cv=3, + scoring='neg_mean_squared_error', + variables=['LotArea', 'GrLivArea'], + regression=True, + ) + + # fit the transformer + disc.fit(X_train, y_train) + +The `binner_dict_` will also contain the limits of the intervals: + +.. code:: python + + disc.binner_dict_ + +.. code:: python + + {'LotArea': [-inf, 8637.5, 10924.0, 13848.5, inf], + 'GrLivArea': [-inf, + 749.5, + 808.0, + 1049.0, + 1144.5, + 1199.0, + 1413.0, + 1438.5, + 1483.0, + 1651.5, + 1825.0, + 1969.5, + 2386.0, + 2408.0, + 2661.0, + 4576.0, + inf]} + +When we apply transform, :class:`DecisionTreeDiscretiser()` will use these limits with `pandas.cut` to +discretize the continuous variable: + +.. code:: python + + train_t = disc.transform(X_train) + test_t = disc.transform(X_test) + + print(train_t[['LotArea', 'GrLivArea']].head()) + +In the following output we see the interval numbers into which the values of the continuous attributes +were sorted: + +.. code:: python + + LotArea GrLivArea + 254 0 5 + 1066 0 8 + 638 1 1 + 799 0 9 + 380 0 9 + +Additional considerations +------------------------- + +Decision tree discretization uses scikit-learn's DecisionTreeRegressor or DecisionTreeClassifier under +the hood to find the optimal interval limits. These models do not support missing data. Hence, we need +to replace missing values with numbers before proceeding with the disrcretization. + +Tutorials, books and courses +---------------------------- Check also for more details on how to use this transformer: - `Jupyter notebook `_ - `tree_pipe in cell 21 of this Kaggle kernel `_ -For more details about this and other feature engineering methods check out these resources: - -- `Feature engineering for machine learning `_, online course. -- `Python Feature Engineering Cookbook `_, book. \ No newline at end of file +For tutorials about this and other discretization methods and feature engineering techniques check out our online course: + +.. figure:: ../../images/feml.png + :width: 300 + :figclass: align-center + :align: left + :target: https://www.trainindata.com/p/feature-engineering-for-machine-learning + + Feature Engineering for Machine Learning + +| +| +| +| +| +| +| +| +| +| + +Or read our book: + +.. figure:: ../../images/cookbook.png + :width: 200 + :figclass: align-center + :align: left + :target: https://packt.link/0ewSo + + Python Feature Engineering Cookbook + +| +| +| +| +| +| +| +| +| +| +| +| +| + +Both our book and course are suitable for beginners and more advanced data scientists +alike. By purchasing them you are supporting Sole, the main developer of Feature-engine. \ No newline at end of file diff --git a/docs/whats_new/v_170.rst b/docs/whats_new/v_170.rst index adf8e5471..cc001fcff 100644 --- a/docs/whats_new/v_170.rst +++ b/docs/whats_new/v_170.rst @@ -1,6 +1,30 @@ Version 1.7.X ============= + +Version 1.7.1 +------------- + +Deployed: XXth XX 2024 + +Contributors +~~~~~~~~~~~~ + +- `Soledad Galli `_ + +TBD + +New functionality +~~~~~~~~~~~~~~~~~ + +TBD + +Enhancements +~~~~~~~~~~~~ + +- The `DecisionTreeDiscretiser()` can now replace the continuous attributes with the decision tree predictions, interval limits, or bin numnber (`Soledad Galli `_) + + Version 1.7.0 ------------- diff --git a/feature_engine/discretisation/decision_tree.py b/feature_engine/discretisation/decision_tree.py index decef6a39..13822650f 100644 --- a/feature_engine/discretisation/decision_tree.py +++ b/feature_engine/discretisation/decision_tree.py @@ -3,6 +3,7 @@ from typing import Dict, List, Optional, Union +import numpy as np import pandas as pd from sklearn.model_selection import GridSearchCV from sklearn.tree import DecisionTreeClassifier, DecisionTreeRegressor @@ -35,17 +36,19 @@ class DecisionTreeDiscretiser(BaseNumericalTransformer): """ The DecisionTreeDiscretiser() replaces numerical variables by discrete, i.e., - finite variables, which values are the predictions of a decision tree. + finite variables, whose values are the predictions of a decision tree, the bin + number, or the bin limits. The method is inspired by the following article from the winners of the KDD 2009 competition: http://www.mtome.com/Publications/CiML/CiML-v3-book.pdf - The DecisionTreeDiscretiser() trains a decision tree per variable. Then, it - transforms the variables, with predictions of the decision tree. + The DecisionTreeDiscretiser() trains a decision tree per variable. Then it finds + the boundaries of each bin. Finally, it replaces the variable values with + the predictions of the decision tree, the bin number, or the bin limits. - The DecisionTreeDiscretiser() works only with numerical variables. A list of - variables to transform can be indicated. Alternatively, the discretiser will + The DecisionTreeDiscretiser() works only with numerical variables. You can pass a + list with the variables you wish to transform. Alternatively, the discretiser will automatically select all numerical variables. More details in the :ref:`User Guide `. @@ -54,6 +57,17 @@ class DecisionTreeDiscretiser(BaseNumericalTransformer): ---------- {variables} + bin_output: str, default = "prediction" + Whether to return the predictions of the tree, the bin number, or the interval + boundaries. Takes values "prediction", "bin_number" and "boundaries", + respectively. + + precision: int, default=None + The precision at which to store and display the bins labels. In other words, + the number of decimals after the comma. Only used when `bin_output` is + "prediction" or "boundaries". If `bin_output="boundaries"` then precision + cannot be None. + cv: int, cross-validation generator or an iterable, default=3 Determines the cross-validation splitting strategy. Possible inputs for cv are: @@ -97,7 +111,8 @@ class DecisionTreeDiscretiser(BaseNumericalTransformer): Attributes ---------- binner_dict_: - Dictionary containing the fitted tree per variable. + Dictionary with the interval limits per variable or the fitted tree per + variable, depending on how `bin_output` was set up. scores_dict_: Dictionary with the score of the best decision tree per variable. @@ -111,12 +126,12 @@ class DecisionTreeDiscretiser(BaseNumericalTransformer): Methods ------- fit: - Fit a decision tree per variable. + Fit a decision tree per variable and find the interval limits. {fit_transform} transform: - Replace continuous variable values by the predictions of the decision tree. + Sort continuous variables into intervals or replace them with the predictions. See Also -------- @@ -132,8 +147,8 @@ class DecisionTreeDiscretiser(BaseNumericalTransformer): Examples -------- - >>> import pandas as pd >>> import numpy as np + >>> import pandas as pd >>> from feature_engine.discretisation import DecisionTreeDiscretiser >>> np.random.seed(42) >>> X = pd.DataFrame(dict(x= np.random.randint(1,100, 100))) @@ -159,6 +174,8 @@ class DecisionTreeDiscretiser(BaseNumericalTransformer): def __init__( self, variables: Union[None, int, str, List[Union[str, int]]] = None, + bin_output: str = "prediction", + precision: Union[int, None] = None, cv=3, scoring: str = "neg_mean_squared_error", param_grid: Optional[Dict[str, Union[str, int, float, List[int]]]] = None, @@ -166,9 +183,30 @@ def __init__( random_state: Optional[int] = None, ) -> None: + if bin_output not in ["prediction", "bin_number", "boundaries"]: + raise ValueError( + "bin_output takes values 'prediction', 'bin_number' or 'boundaries'. " + f"Got {bin_output} instead." + ) + + if precision is not None and (not isinstance(precision, int) or precision < 1): + raise ValueError( + "precision must be None or a positive integer. " + f"Got {precision} instead." + ) + + if bin_output == "boundaries" and precision is None: + raise ValueError( + "When `bin_output == 'boundaries', `precision` cannot be None. " + "Change precision's value to a positive integer." + ) if not isinstance(regression, bool): - raise ValueError("regression can only take True or False") + raise ValueError( + f"regression can only take True or False. Got {regression} instead." + ) + self.bin_output = bin_output + self.precision = precision self.cv = cv self.scoring = scoring self.regression = regression @@ -210,8 +248,8 @@ def fit(self, X: pd.DataFrame, y: pd.Series): # type: ignore else: param_grid = {"max_depth": [1, 2, 3, 4]} - self.binner_dict_ = {} - self.scores_dict_ = {} + binner_dict_ = {} + scores_dict_ = {} for var in self.variables_: @@ -227,9 +265,21 @@ def fit(self, X: pd.DataFrame, y: pd.Series): # type: ignore # fit the model to the variable tree_model.fit(X[var].to_frame(), y) - self.binner_dict_[var] = tree_model - self.scores_dict_[var] = tree_model.score(X[var].to_frame(), y) - + binner_dict_[var] = tree_model + scores_dict_[var] = tree_model.score(X[var].to_frame(), y) + + if self.bin_output != "prediction": + for var in self.variables_: + clf = binner_dict_[var].best_estimator_ + threshold = clf.tree_.threshold + feature = clf.tree_.feature + feature_threshold = threshold[feature == 0] + thresholds = sorted(feature_threshold) + thresholds = [-np.inf] + thresholds + [np.inf] + binner_dict_[var] = thresholds + + self.binner_dict_ = binner_dict_ + self.scores_dict_ = scores_dict_ return self def transform(self, X: pd.DataFrame) -> pd.DataFrame: @@ -251,12 +301,42 @@ def transform(self, X: pd.DataFrame) -> pd.DataFrame: # check input dataframe and if class was fitted X = self._check_transform_input_and_state(X) - for feature in self.variables_: - if self.regression: - X[feature] = self.binner_dict_[feature].predict(X[feature].to_frame()) - else: - tmp = self.binner_dict_[feature].predict_proba(X[feature].to_frame()) - X[feature] = tmp[:, 1] + if self.bin_output == "prediction": + for feature in self.variables_: + if self.regression: + preds = self.binner_dict_[feature].predict(X[feature].to_frame()) + if self.precision is None: + X[feature] = preds + else: + X[feature] = np.round(preds, self.precision) + else: + tmp = self.binner_dict_[feature].predict_proba( + X[feature].to_frame() + ) + preds = tmp[:, 1] + if self.precision is None: + X[feature] = preds + else: + X[feature] = np.round(preds, self.precision) + + elif self.bin_output == "boundaries": + for feature in self.variables_: + X[feature] = pd.cut( + X[feature], + self.binner_dict_[feature], + precision=self.precision, + include_lowest=True, + ) + X[self.variables_] = X[self.variables_].astype(str) + + else: + for feature in self.variables_: + X[feature] = pd.cut( + X[feature], + self.binner_dict_[feature], + labels=False, + include_lowest=True, + ) return X diff --git a/tests/test_discretisation/test_decision_tree_discretiser.py b/tests/test_discretisation/test_decision_tree_discretiser.py index 286238968..a90d64ab8 100644 --- a/tests/test_discretisation/test_decision_tree_discretiser.py +++ b/tests/test_discretisation/test_decision_tree_discretiser.py @@ -6,7 +6,82 @@ from feature_engine.discretisation import DecisionTreeDiscretiser, EqualWidthDiscretiser -def test_classification(df_normal_dist): +# init parameters +@pytest.mark.parametrize( + "params", + [("prediction", 3, True), ("bin_number", 10, False), ("boundaries", 1, False)], +) +def test_init_param_assignment(params): + dsc = DecisionTreeDiscretiser( + bin_output=params[0], + precision=params[1], + regression=params[2], + ) + assert dsc.bin_output == params[0] + assert dsc.precision == params[1] + assert dsc.regression == params[2] + + +@pytest.mark.parametrize("bin_output_", ["arbitrary", False, 1]) +def test_error_if_binoutput_not_permitted_value(bin_output_): + msg = ( + "bin_output takes values 'prediction', 'bin_number' or 'boundaries'. " + f"Got {bin_output_} instead." + ) + with pytest.raises(ValueError) as record: + DecisionTreeDiscretiser(bin_output=bin_output_) + assert str(record.value) == msg + + +@pytest.mark.parametrize("precision_", ["arbitrary", -1, 0.3]) +def test_error_if_precision_not_permitted_value(precision_): + msg = "precision must be None or a positive integer. " f"Got {precision_} instead." + with pytest.raises(ValueError) as record: + DecisionTreeDiscretiser(precision=precision_) + assert str(record.value) == msg + + +def test_precision_errors_if_none_when_bin_output_is_boundaries(): + msg = ( + "When `bin_output == 'boundaries', `precision` cannot be None. " + "Change precision's value to a positive integer." + ) + with pytest.raises(ValueError) as record: + DecisionTreeDiscretiser(precision=None, bin_output="boundaries") + assert str(record.value) == msg + + dsc = DecisionTreeDiscretiser(precision=None, bin_output="bin_number") + assert dsc.precision is None + + +@pytest.mark.parametrize("regression_", ["arbitrary", -1, 0.3]) +def test_error_if_regression_is_not_bool(regression_): + msg = "regression can only take True or False. " f"Got {regression_} instead." + with pytest.raises(ValueError) as record: + DecisionTreeDiscretiser(regression=regression_) + assert str(record.value) == msg + + +# fit +def test_error_if_y_not_passed(df_normal_dist): + encoder = DecisionTreeDiscretiser() + with pytest.raises(TypeError): + encoder.fit(df_normal_dist) + + +def test_error_when_regression_is_true_and_target_is_binary(df_discretise): + msg = ( + "Trying to fit a regression to a binary target is not " + "allowed by this transformer. Check the target values " + "or set regression to False." + ) + transformer = DecisionTreeDiscretiser(regression=True) + with pytest.raises(ValueError) as record: + transformer.fit(df_discretise[["var_A", "var_B"]], df_discretise["target"]) + assert str(record.value) == msg + + +def test_classification_predictions(df_normal_dist): transformer = DecisionTreeDiscretiser( cv=3, @@ -36,6 +111,96 @@ def test_classification(df_normal_dist): ) +@pytest.mark.parametrize( + "params", + [ + (1, [1.0, 0.7, 0.9, 0.0]), + (2, [1.0, 0.71, 0.93, 0.0]), + (3, [1.0, 0.712, 0.933, 0.0]), + ], +) +def test_classification_rounds_predictions(df_normal_dist, params): + + transformer = DecisionTreeDiscretiser( + precision=params[0], + cv=3, + scoring="roc_auc", + variables=None, + param_grid={"max_depth": [1, 2, 3, 4]}, + regression=False, + random_state=0, + ) + np.random.seed(0) + y = pd.Series(np.random.binomial(1, 0.7, 100)) + X = transformer.fit_transform(df_normal_dist, y) + bins = params[1] + + assert list(X["var"].unique()) == bins + + +def test_classification_bin_number(df_normal_dist): + transformer = DecisionTreeDiscretiser( + bin_output="bin_number", + scoring="roc_auc", + param_grid={"max_depth": [1, 2, 3, 4]}, + regression=False, + random_state=0, + ) + np.random.seed(0) + y = pd.Series(np.random.binomial(1, 0.7, 100)) + X = transformer.fit_transform(df_normal_dist, y) + bins = [4, 2, 1, 0, 3] + limits = [ + -np.inf, + -0.22668930888175964, + -0.09422881528735161, + 0.10165948793292046, + 0.11590901389718056, + np.inf, + ] + + assert transformer.binner_dict_["var"] == limits + assert np.round(transformer.scores_dict_["var"], 3) == np.round( + 0.717391304347826, 3 + ) + assert list(X["var"].unique()) == bins + + +def test_classification_boundaries(df_normal_dist): + transformer = DecisionTreeDiscretiser( + bin_output="boundaries", + precision=3, + scoring="roc_auc", + param_grid={"max_depth": [1, 2, 3, 4]}, + regression=False, + random_state=0, + ) + np.random.seed(0) + y = pd.Series(np.random.binomial(1, 0.7, 100)) + X = transformer.fit_transform(df_normal_dist, y) + bins = [ + "(0.116, inf]", + "(-0.0942, 0.102]", + "(-0.227, -0.0942]", + "(-inf, -0.227]", + "(0.102, 0.116]", + ] + limits = [ + -np.inf, + -0.22668930888175964, + -0.09422881528735161, + 0.10165948793292046, + 0.11590901389718056, + np.inf, + ] + + assert transformer.binner_dict_["var"] == limits + assert np.round(transformer.scores_dict_["var"], 3) == np.round( + 0.717391304347826, 3 + ) + assert list(X["var"].unique()) == bins + + def test_regression(df_normal_dist): transformer = DecisionTreeDiscretiser( @@ -83,18 +248,53 @@ def test_regression(df_normal_dist): assert all(x for x in np.round(X["var"].unique(), 2) if x not in X_t) -def test_error_when_regression_is_not_bool(): - with pytest.raises(ValueError): - DecisionTreeDiscretiser(regression="other") +@pytest.mark.parametrize( + "params", + [ + (1, [0.2, 0.0, 0.1, -0.1, -0.3, -0.2]), + ( + 2, + [ + 0.19, + 0.04, + 0.11, + 0.23, + -0.09, + -0.02, + 0.01, + 0.15, + 0.07, + -0.26, + 0.09, + -0.07, + -0.16, + -0.2, + -0.04, + -0.12, + ], + ), + ], +) +def test_regression_rounds_predictions(df_normal_dist, params): + transformer = DecisionTreeDiscretiser( + precision=params[0], + cv=3, + scoring="neg_mean_squared_error", + variables=None, + param_grid={"max_depth": [1, 2, 3, 4]}, + regression=True, + random_state=0, + ) + np.random.seed(0) + y = pd.Series(pd.Series(np.random.normal(0, 0.1, 100))) + X = transformer.fit_transform(df_normal_dist, y) + bins = params[1] -def test_error_if_y_not_passed(df_normal_dist): - # test case 3: raises error if target is not passed - with pytest.raises(TypeError): - encoder = DecisionTreeDiscretiser() - encoder.fit(df_normal_dist) + assert list(X["var"].unique()) == bins +# transform def test_non_fitted_error(df_vartypes): with pytest.raises(NotFittedError): transformer = EqualWidthDiscretiser() @@ -119,16 +319,10 @@ def df_discretise(): return df -def test_error_when_regression_is_true_and_target_is_binary(df_discretise): - with pytest.raises(ValueError): - transformer = DecisionTreeDiscretiser(regression=True) - transformer.fit(df_discretise[["var_A", "var_B"]], df_discretise["target"]) - - def test_error_when_regression_is_false_and_target_is_continuous(df_discretise): np.random.seed(42) mu, sigma = 0, 3 y = np.random.normal(mu, sigma, len(df_discretise)) + transformer = DecisionTreeDiscretiser(regression=False) with pytest.raises(ValueError): - transformer = DecisionTreeDiscretiser(regression=False) transformer.fit(df_discretise[["var_A", "var_B"]], y) diff --git a/tests/test_encoding/test_count_frequency_encoder.py b/tests/test_encoding/test_count_frequency_encoder.py index f98f8ccf5..a724344a1 100644 --- a/tests/test_encoding/test_count_frequency_encoder.py +++ b/tests/test_encoding/test_count_frequency_encoder.py @@ -27,12 +27,16 @@ def test_error_if_unseen_gets_not_permitted_value(errors): "params", [("count", "raise", True), ("frequency", "ignore", False)] ) def test_init_param_assignment(params): - CountFrequencyEncoder( + enc = CountFrequencyEncoder( encoding_method=params[0], missing_values=params[1], ignore_format=params[2], unseen=params[1], ) + assert enc.encoding_method == params[0] + assert enc.missing_values == params[1] + assert enc.ignore_format == params[2] + assert enc.unseen == params[1] # fit and transform