fix logcptransformer so that it adds constants to strictly non positi…

docs/user_guide/transformation/LogCpTransformer.rst

@@ -5,14 +5,32 @@
 LogCpTransformer
 ================
 
-The :class:`LogCpTransformer()` applies the transformation log(x + C), where C is a
-positive constant.
+:class:`LogCpTransformer()` applies the transformation log(x + C), where x is the
+variable to transform and C is a positive constant that shifts the distribution towards
+positive values.
 
-You can enter the positive quantity to add to the variable. Alternatively, the transformer
-will find the necessary quantity to make all values of the variable positive.
+:class:`LogCpTransformer()` is an extension of :class:`LogTransformer()` that allows
+adding a constant to move distributions towards positive values. For more details about
+the logarithm transformation, check out the :ref:`LogTransformer()'s user Guide <log_transformer>`.
 
-Example
--------
+Defining C
+----------
+
+You can enter the positive quantity to add to the variable as a dictionary, where the
+keys are the variable names, and the values are the constant to add to each variable. If you
+want to add the same value to all variables, you can pass an integer or float, instead.
+
+Alternatively, the :class:`LogCpTransformer()` will find the necessary value to make all
+values of the variable positive. For strictly positive variables, C will be 0, and the
+transformation will be log(x).
+
+Python example
+--------------
+
+Let's check out the functionality of :class:`LogCpTransformer()`.
+
+Transforming strictly positive variables
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Let's load the California housing dataset that comes with Scikit-learn and separate it
 into train and test sets.
@@ -54,15 +72,15 @@ an attribute. We can visualise the learned parameters as follows:
     # learned constant C
     tf.C_
 
-.. code:: python
+As these variables are strictly positive, the transformer will add 0 to the variables
+before applying the logarithm transformation:
 
-    {'MedInc': 1.4999, 'HouseAge': 2.0}
+.. code:: python
 
-Applying the log of a variable plus a constant in this dataset does not make much sense
-because all variables are positive, that is why the constant values C for the former
-variables are possible.
+    {'MedInc': 0, 'HouseAge': 0}
 
-We will carry on with the demo anyways.
+In this case, the transformation applied by :class:`LogCpTransformer()` is the same as
+using :class:`LogTransformer()` because these variables are strictly positive.
 
 We can now go ahead and transform the variables:
 
@@ -83,6 +101,8 @@ Then we can plot the original variable distribution:
 
 .. image:: ../../images/logcpraw.png
 
+|
+
 And the distribution of the transformed variable:
 
 .. code:: python
@@ -94,6 +114,171 @@ And the distribution of the transformed variable:
 
 .. image:: ../../images/logcptransform.png
 
+|
+
+Transforming non-strictly positive variables
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Let's now show the functionality of :class:`LogCpTransformer()` with variables that contain
+values lower or equal to 0. Let's load the diabetes dataset:
+
+.. code:: python
+
+    import pandas as pd
+    import matplotlib.pyplot as plt
+    from sklearn.model_selection import train_test_split
+    from sklearn.datasets import load_diabetes
+    from feature_engine.transformation import LogCpTransformer
+
+    # Load dataset
+    X, y = load_diabetes( return_X_y=True, as_frame=True)
+
+    # Separate into train and test sets
+    X_train, X_test, y_train, y_test =  train_test_split(
+        X, y, test_size=0.3, random_state=0)
+
+Let's print out a summary of the main characteristics of 2 of the variables:
+
+.. code:: python
+
+    print(X_train[["bmi", "s3"]].describe())
+
+In the following output we see that the variables contain negative values:
+
+.. code:: python
+
+                  bmi          s3
+    count  309.000000  309.000000
+    mean    -0.001298    0.000511
+    std      0.048368    0.048294
+    min     -0.084886   -0.102307
+    25%     -0.036385   -0.032356
+    50%     -0.008362   -0.006584
+    75%      0.030440    0.030232
+    max      0.170555    0.181179
+
+Let's now set up :class:`LogCpTransformer()` to shift the variables' distribution to
+positive values and then apply the logarithm:
+
+.. code:: python
+
+    tf = LogCpTransformer(variables = ["bmi", "s3"], C="auto")
+    tf.fit(X_train)
+
+We can inspect the constant values that will be added to each variable:
+
+.. code:: python
+
+    tf.C_
+
+Since these variables were not strictly positive, :class:`LogCpTransformer()` found
+the minimum value needed to make their values positive:
+
+.. code:: python
+
+    {'bmi': 1.0848862355291056, 's3': 1.102307050517416}
+
+We can now transform the data:
+
+.. code:: python
+
+    train_t= tf.transform(X_train)
+    test_t= tf.transform(X_test)
+
+Let's plot `bmi` before the transformation:
+
+.. code:: python
+
+    X_train["bmi"].hist(bins=20)
+    plt.title("bmi - original distribution")
+    plt.ylabel("Number of observations")
+
+In the following image we see the original distribution of `bmi`:
+
+.. image:: ../../images/bmiraw.png
+
+|
+
+Let's now plot the transformed variable:
+
+.. code:: python
+
+    # transformed variable
+    train_t["bmi"].hist(bins=20)
+    plt.title("bmi - transformed distribution")
+    plt.ylabel("Number of observations")
+
+In the following image we see the distribution of `bmi` after the transformation:
+
+.. image:: ../../images/bmilogcp.png
+
+|
+
+
+Adding the same constant to all variables
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can add the same constant to all variables by setting up :class:`LogCpTransformer()`
+as follows:
+
+.. code:: python
+
+    tf = LogCpTransformer(C=5)
+    tf.fit(X_train)
+
+In this case, all numerical variables will be transformed. We can find the variables that
+will be transformed in the `variables_` attribute:
+
+.. code:: python
+
+    tf.variables_
+
+All numerical variables were selected for the transformation:
+
+.. code:: python
+
+    ['age', 'sex', 'bmi', 'bp', 's1', 's2', 's3', 's4', 's5', 's6']
+
+You can now apply `transform()` to transform all these variables.
+
+Adding different user defined constants
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you want to add specific values to specific variables, you can do so by setting
+:class:`LogCpTransformer()` as follows:
+
+.. code:: python
+
+    tf = LogCpTransformer(C={"bmi": 2, "s3": 3, "s4": 4})
+    tf.fit(X_train)
+
+In this case, :class:`LogCpTransformer()` will only modify the variables indicated in the
+dictionary:
+
+.. code:: python
+
+    tf.variables_
+
+The variables in the dictionary will be transformed:
+
+.. code:: python
+
+    ['bmi', 's3', 's4']
+
+And the constant values will be those from the dictionary:
+
+.. code:: python
+
+    tf.C_
+
+`C_` coincides with the values entered in `C`:
+
+.. code:: python
+
+    {'bmi': 2, 's3': 3, 's4': 4}
+
+You can now apply `transform()` to transform all these variables.
+
 
 Tutorials, books and courses
 ----------------------------

feature_engine/transformation/log.py

@@ -225,13 +225,18 @@ def _more_tags(self):
 )
 class LogCpTransformer(BaseNumericalTransformer, FitFromDictMixin):
     """
-    The LogCpTransformer() applies the transformation log(x + C), where C is a positive
-    constant, to the input variable. It applies the natural logarithm or the base 10
-    logarithm, where the natural logarithm is logarithm in base e.
+    LogCpTransformer() applies the transformation log(x + C), where x is the
+    variable to transform and C is a positive constant. It can apply the natural
+    logarithm or the base 10 logarithm, where the natural logarithm is logarithm in
+    base e.
 
-    The logarithm can only be applied to numerical non-negative values. If the
-    variable contains a zero or a negative value after adding a constant C, the
-    transformer will return an error.
+    As the logarithm can only be applied to numerical non-negative values,
+    LogCpTransformer() extends the functionality of LogTransformer, by adding a
+    constant to shift the distribution of the variables towards positive values.
+
+    Note that if the variable contains a zero or a negative value after adding a
+    constant C, the transformer will return an error. This can occur if the values of
+    the variables in the test set are smaller than those seen during `fit()`.
 
     A list of variables can be passed as an argument. Alternatively, the transformer
     will automatically select and transform all variables of type numeric.
@@ -265,7 +270,7 @@ class LogCpTransformer(BaseNumericalTransformer, FitFromDictMixin):
 
     C_:
         The constant C to add to each variable. If C = "auto" a dictionary with
-        C = abs(min(variable)) + 1.
+        C = abs(min(variable)) + 1. For strictly positive variables, C = 0.
 
     {feature_names_in_}
 
@@ -286,21 +291,23 @@ class LogCpTransformer(BaseNumericalTransformer, FitFromDictMixin):
     Examples
     --------
 
-    >>> import numpy as np
     >>> import pandas as pd
     >>> from feature_engine.transformation import LogCpTransformer
-    >>> np.random.seed(42)
-    >>> X = pd.DataFrame(dict(x = np.random.lognormal(size = 100)))
+    >>> X = pd.DataFrame(dict(
+    >>>    vara=[0, 1, 2, 3],
+    >>>    varb=[5, 5, 6, 7],
+    >>>    varc=[-2, -1, 0, 4],
+    >>>    vard=[-3, -2, -1, -5],
+    >>>    vare=["a", "b", "c", "d"]))
     >>> lct = LogCpTransformer()
     >>> lct.fit(X)
     >>> X = lct.transform(X)
-    >>> X.head()
-              x
-    0  0.944097
-    1  0.586701
-    2  1.043204
-    3  1.707159
-    4  0.541405
+    >>> X
+           vara      varb      varc      vard vare
+    0  0.000000  1.609438  0.000000  1.098612    a
+    1  0.693147  1.609438  0.693147  1.386294    b
+    2  1.098612  1.791759  1.098612  1.609438    c
+    3  1.386294  1.945910  1.945910  0.000000    d
     """
 
     def __init__(
@@ -311,10 +318,14 @@ def __init__(
     ) -> None:
 
         if base not in ["e", "10"]:
-            raise ValueError("base can take only '10' or 'e' as values")
+            raise ValueError(
+                f"base can take only '10' or 'e' as values. Got {base} instead."
+            )
 
         if not isinstance(C, (int, float, dict)) and not C == "auto":
-            raise ValueError("C can take only 'auto', integers or floats")
+            raise ValueError(
+                f"C can take only 'auto', integers or floats. Got {C} instead."
+            )
 
         self.variables = _check_variables_input_value(variables)
         self.base = base
@@ -349,14 +360,15 @@ def fit(self, X: pd.DataFrame, y: Optional[pd.Series] = None):
 
         # calculate C to add to each variable
         if self.C == "auto":
-            self.C_ = dict(X[self.variables_].min(axis=0).abs() + 1)
+            # we add 0 to positive variables
+            c_dict = {var: 0 for var in self.variables_ if X[var].min() > 0}
 
-        # check variables are positive after adding C
-        if (X[self.variables_] + self.C_ <= 0).any().any():
-            raise ValueError(
-                "Some variables contain zero or negative values after adding"
-                + "constant C, can't apply log"
-            )
+            # we add the minimum plus 1 to non-positive variables
+            non_positive_vars = [
+                var for var in self.variables_ if var not in c_dict.keys()
+            ]
+            c_dict.update(dict(X[non_positive_vars].min(axis=0).abs() + 1))
+            self.C_ = c_dict  # type:ignore
 
         return self
 
@@ -379,18 +391,20 @@ def transform(self, X: pd.DataFrame) -> pd.DataFrame:
         X = self._check_transform_input_and_state(X)
 
         # check variable is positive after adding c
+        error_msg = (
+            "Some variables contain zero or negative values after adding"
+            + " constant C, can't apply log."
+        )
+
         if (X[self.variables_] + self.C_ <= 0).any().any():
-            raise ValueError(
-                "Some variables contain zero or negative values after adding"
-                + "constant C, can't apply log"
-            )
+            raise ValueError(error_msg)
 
         X[self.variables_] = X[self.variables_].astype(float)
 
         # transform
         if self.base == "e":
             X.loc[:, self.variables_] = np.log(X.loc[:, self.variables_] + self.C_)
-        elif self.base == "10":
+        else:
             X.loc[:, self.variables_] = np.log10(X.loc[:, self.variables_] + self.C_)
 
         return X
@@ -416,7 +430,7 @@ def inverse_transform(self, X: pd.DataFrame) -> pd.DataFrame:
         # inverse transform
         if self.base == "e":
             X.loc[:, self.variables_] = np.exp(X.loc[:, self.variables_]) - self.C_
-        elif self.base == "10":
+        else:
             X.loc[:, self.variables_] = 10 ** X.loc[:, self.variables_] - self.C_
 
         return X