fix multiple bugs introduced by pandas, sklearn and scipy latest versions

README.md

@@ -10,8 +10,7 @@
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 [![GitHub contributors](https://img.shields.io/github/contributors/feature-engine/feature_engine?logo=GitHub)](https://github.com/feature-engine/feature_engine/graphs/contributors)
 [![Gitter](https://img.shields.io/gitter/room/feature-engine/feaure_engine?logo=Gitter)](https://gitter.im/feature_engine/community)
-[![Total Downloads](https://pepy.tech/badge/feature-engine)](https://pepy.tech/project/feature-engine)
-[![Monthly Downloads](https://pepy.tech/badge/feature-engine/month)](https://pepy.tech/project/feature-engine)
+[![Monthly Downloads](https://img.shields.io/pypi/dm/feature-engine)](https://img.shields.io/pypi/dm/feature-engine)
 [![DOI](https://zenodo.org/badge/163630824.svg)](https://zenodo.org/badge/latestdoi/163630824)
 [![JOSS](https://joss.theoj.org/papers/10.21105/joss.03642/status.svg)](https://doi.org/10.21105/joss.03642)
 [![first-timers-only](https://img.shields.io/badge/first--timers--only-friendly-blue.svg?style=flat)](https://www.firsttimersonly.com/)

docs/conf.py

@@ -195,7 +195,7 @@
     "pandas": ("https://pandas.pydata.org/docs/", None),
     "scipy": ("https://docs.scipy.org/doc/scipy/reference", None),
     "matplotlib": ("https://matplotlib.org/", None),
-    "sklearn": ("http://scikit-learn.org/stable", None),
+    "sklearn": ("https://scikit-learn.org/stable/", None),
 }
 
 # -- Options for LaTeX output ---------------------------------------------
@@ -273,9 +273,6 @@
 # A list of files that should not be packed into the epub file.
 epub_exclude_files = ["search.html"]
 
-# Example configuration for intersphinx: refer to the Python standard library.
-intersphinx_mapping = {"https://docs.python.org/": None}
-
 # The following is used by sphinx.ext.linkcode to provide links to github
 linkcode_resolve = make_linkcode_resolve(
     "feature_engine",

docs/user_guide/datetime/DatetimeFeatures.rst

@@ -77,7 +77,7 @@ First, we will create a toy dataframe with 2 date variables:
 
     toy_df = pd.DataFrame({
         "var_date1": ['May-1989', 'Dec-2020', 'Jan-1999', 'Feb-2002'],
-        "var_date2": ['06/21/12', '02/10/98', '08/03/10', '10/31/20'],
+        "var_date2": ['06/21/2012', '02/10/1998', '08/03/2010', '10/31/2020'],
     })
 
 Now, we will extract the variables month, month-end and the day of the year from the
@@ -148,9 +148,14 @@ First, let's create a toy dataset with 2 time variables and an object variable.
 datetime. So if we want to extract time features from all our datetime variables, we
 don't need to specify them.
 
+Note that from version 2.0.0 pandas deprecated the parameter `infer_datetime_format`.
+Hence, if you want pandas to infer the datetime format and you have different formats,
+you need to explicitly say so by passing `"mixed"` to the `format` parameter as shown
+below.
+
 .. code:: python
 
-    dfts = DatetimeFeatures(features_to_extract=["minute"])
+    dfts = DatetimeFeatures(features_to_extract=["minute"], format="mixed")
 
     df_transf = dfts.fit_transform(toy_df)
 
@@ -227,10 +232,11 @@ the features.
         variables=["var_dt1", "var_dt3"],
         features_to_extract=["year", "hour"],
         drop_original=False,
+        format="mixed",
     )
     df_transf = dfts.fit_transform(toy_df)
 
-    print(df_transf)
+    df_transf
 
 We can see the resulting dataframe in the following output:
 
@@ -373,10 +379,11 @@ And now we mistakenly extract only date features:
 
     dfts = DatetimeFeatures(
         features_to_extract=["year", "month", "day_of_week"],
+        format="mixed",
     )
     df_transf = dfts.fit_transform(toy_df)
 
-    print(df_transf)
+    df_transf
 
 .. code:: python
 
@@ -413,6 +420,7 @@ And we mistakenly extract the hour and the minute:
 
     dfts = DatetimeFeatures(
         features_to_extract=["hour", "minute"],
+        format="mixed",
     )
     df_transf = dfts.fit_transform(toy_df)
 
@@ -466,7 +474,7 @@ To do this, we leave the parameter `features_to_extract` to `None`.
 
     df_transf = dfts.fit_transform(toy_df)
 
-    print(df_transf)
+    df_transf
 
 .. code:: python
 
@@ -614,7 +622,7 @@ from the dataset.
 .. code:: python
 
     pipe = Pipeline([
-        ('datetime', DatetimeFeatures()),
+        ('datetime', DatetimeFeatures(format="mixed")),
         ('drop_constant', DropConstantFeatures()),
     ])
 
@@ -683,12 +691,13 @@ converts all data to UTC timezone.
     dfts = DatetimeFeatures(
         features_to_extract=["hour", "minute"],
         drop_original=False,
-        utc=True
+        utc=True,
+        format="mixed",
     )
 
     df_transf = dfts.fit_transform(toy_df)
 
-    print(df_transf)
+    df_transf
 
 .. code:: python
 
@@ -709,7 +718,7 @@ the datetime information extracted as if it were in UTC timezone.
     from feature_engine.datetime import DatetimeFeatures
 
     var_tz = pd.Series(['08/31/00 12:34:45', '12/01/90 23:01:02', '04/25/01 11:59:21'])
-    var_tz = pd.to_datetime(var_tz)
+    var_tz = pd.to_datetime(var_tz, format="mixed")
     var_tz = var_tz.dt.tz_localize("US/eastern")
     var_tz
 
@@ -735,7 +744,7 @@ timezone.
 
     df_transf = dfts.fit_transform(toy_df)
 
-    print(df_transf)
+    df_transf
 
 .. code:: python
 

docs/user_guide/datetime/DatetimeSubtraction.rst

@@ -81,8 +81,8 @@ example shows how to use this syntax:
 
 .. code:: python
 
-    data["diff"] = data["date1"].sub(data["date2"], axis=0).apply(
-        lambda x: x / np.timedelta64(1, "Y"))
+    data["diff"] = data["date1"].sub(data["date2"], axis=0).div(
+        np.timedelta64(1, "Y").astype("timedelta64[ns]"))
 
     print(data)
 
@@ -235,6 +235,7 @@ parameter `missing_values` to `"ignore"`. Here is a code example:
 
 .. code:: python
 
+    import numpy as np
     import pandas as pd
     from feature_engine.datetime import DatetimeSubtraction
 
@@ -293,6 +294,7 @@ the time difference in microseconds:
         reference="date2",
         utc=True,
         output_unit="ms",
+        format="mixed"
     )
 
     new = dfts.fit_transform(data)

docs/user_guide/encoding/RareLabelEncoder.rst

@@ -55,6 +55,7 @@ First, let's load the data and separate it into train and test:
         predictors_only=True,
         cabin="letter_only",
     )
+    X["pclass"] = X["pclass"].astype("O")
 
     X_train, X_test, y_train, y_test = train_test_split(
         X, y, test_size=0.3, random_state=0,
@@ -73,19 +74,29 @@ We see the resulting data below:
     1193       3    male  29.881135      0      0   7.7250     M        Q
     686        3  female  22.000000      0      0   7.7250     M        Q
 
+Let's explore the number of uniue categories in the variable `"cabin"`.
+
+.. code:: python
+
+    X_train["cabin"].unique()
+
+We see the number of unique categories in the output below:
+
+.. code:: python
+
+    array(['M', 'E', 'C', 'D', 'B', 'A', 'F', 'T', 'G'], dtype=object)
+
 Now, we set up the :class:`RareLabelEncoder()` to group categories shown by less than 3%
 of the observations into a new group or category called 'Rare'. We will group the
 categories in the indicated variables if they have more than 2 unique categories each.
 
 .. code:: python
 
-    # set up the encoder
     encoder = RareLabelEncoder(
         tol=0.03,
         n_categories=2,
         variables=['cabin', 'pclass', 'embarked'],
         replace_with='Rare',
-        ignore_format=True,
     )
 
     # fit the encoder
@@ -116,6 +127,20 @@ Now we can go ahead and transform the variables:
     train_t = encoder.transform(X_train)
     test_t = encoder.transform(X_test)
 
+Let's now inspect the number of unique categories in the variable `"cabin"` after the
+transformation:
+
+.. code:: python
+
+    train_t["cabin"].unique()
+
+In the output below, we see that the infrequent categories have been replaced by
+`"Rare"`.
+
+.. code:: python
+
+    array(['M', 'E', 'C', 'D', 'B', 'Rare'], dtype=object)
+
 We can also specify the maximum number of categories that can be considered frequent
 using the `max_n_categories` parameter.
 

docs/user_guide/selection/RecursiveFeatureAddition.rst

@@ -49,12 +49,12 @@ First, we load the data:
     import pandas as pd
     from sklearn.datasets import load_diabetes
     from sklearn.linear_model import LinearRegression
-    from feature_engine.selection import RecursiveFeatureElimination
+    from feature_engine.selection import RecursiveFeatureAddition
 
     # load dataset
     diabetes_X, diabetes_y = load_diabetes(return_X_y=True)
     X = pd.DataFrame(diabetes_X)
-    y = pd.DataFrame(diabetes_y)
+    y = pd.Series(diabetes_y)
 
 Now, we set up :class:`RecursiveFeatureAddition` to select features based on the r2
 returned by a Linear Regression model, using 3 fold cross-validation. In this case,
@@ -66,7 +66,7 @@ we leave the parameter `threshold` to the default value which is 0.01.
     linear_model = LinearRegression()
 
     # initialize feature selector
-    tr = RecursiveFeatureElimination(estimator=linear_model, scoring="r2", cv=3)
+    tr = RecursiveFeatureAddition(estimator=linear_model, scoring="r2", cv=3)
 
 With `fit()` the model finds the most useful features, that is, features that when added
 cause an increase in model performance bigger than 0.01. With `transform()`, the transformer
@@ -100,15 +100,16 @@ adding each feature.
 ..  code:: python
 
     {4: 0,
-     8: 0.2837159006046677,
-     2: 0.1377700238871593,
-     5: 0.0023329006089969906,
-     3: 0.0187608758643259,
-     1: 0.0027994385024313617,
-     7: 0.0026951300105543807,
-     6: 0.002683967832484757,
-     9: 0.0003040126429713075,
-     0: -0.007386876030245182}
+     8: 0.28371458794131676,
+     2: 0.1377714799388745,
+     5: 0.0023327265047610735,
+     3: 0.018759914615172735,
+     1: 0.0027996354657459643,
+     7: 0.002695149440021638,
+     6: 0.002683934134630306,
+     9: 0.000304067408860742,
+     0: -0.007387230783454768}
+
 
 :class:`RecursiveFeatureAddition` also stores the features that will be dropped based
 n the given threshold.
@@ -120,7 +121,7 @@ n the given threshold.
 
 ..  code:: python
 
-    [0, 6, 7, 9]
+    [0, 1, 5, 6, 7, 9]
 
 If we now print the transformed data, we see that the features above were removed.
 
@@ -130,10 +131,10 @@ If we now print the transformed data, we see that the features above were remove
 
 ..  code:: python
 
-              4         8         2         3
-    0 -0.044223  0.019908  0.061696  0.021872
-    1 -0.008449 -0.068330 -0.051474 -0.026328
-    2 -0.045599  0.002864  0.044451 -0.005671
-    3  0.012191  0.022692 -0.011595 -0.036656
-    4  0.003935 -0.031991 -0.036385  0.021872
+              2         3         4         8
+    0  0.061696  0.021872 -0.044223  0.019907
+    1 -0.051474 -0.026328 -0.008449 -0.068332
+    2  0.044451 -0.005670 -0.045599  0.002861
+    3 -0.011595 -0.036656  0.012191  0.022688
+    4 -0.036385  0.021872  0.003935 -0.031988
 

docs/user_guide/selection/RecursiveFeatureElimination.rst

@@ -62,7 +62,7 @@ First, we load the data:
     # load dataset
     diabetes_X, diabetes_y = load_diabetes(return_X_y=True)
     X = pd.DataFrame(diabetes_X)
-    y = pd.DataFrame(diabetes_y)
+    y = pd.Series(diabetes_y)
 
 Now, we set up :class:`RecursiveFeatureElimination` to select features based on the r2
 returned by a Linear Regression model, using 3 fold cross-validation. In this case,
@@ -139,13 +139,12 @@ If we now print the transformed data, we see that the features above were remove
 
 ..  code:: python
 
-              1         3         5         2         8         4
-    0  0.050680  0.021872 -0.034821  0.061696  0.019908 -0.044223
-    1 -0.044642 -0.026328 -0.019163 -0.051474 -0.068330 -0.008449
-    2  0.050680 -0.005671 -0.034194  0.044451  0.002864 -0.045599
-    3 -0.044642 -0.036656  0.024991 -0.011595  0.022692  0.012191
-    4 -0.044642  0.021872  0.015596 -0.036385 -0.031991  0.003935
-
+              1         2         3         4         5         8
+    0  0.050680  0.061696  0.021872 -0.044223 -0.034821  0.019907
+    1 -0.044642 -0.051474 -0.026328 -0.008449 -0.019163 -0.068332
+    2  0.050680  0.044451 -0.005670 -0.045599 -0.034194  0.002861
+    3 -0.044642 -0.011595 -0.036656  0.012191  0.024991  0.022688
+    4 -0.044642 -0.036385  0.021872  0.003935  0.015596 -0.031988
 
 
 More details

docs/user_guide/selection/SelectByShuffling.rst

@@ -38,7 +38,7 @@ First, we load the data:
     # load dataset
     diabetes_X, diabetes_y = load_diabetes(return_X_y=True)
     X = pd.DataFrame(diabetes_X)
-    y = pd.DataFrame(diabetes_y)
+    y = pd.Series(diabetes_y)
 
 Now, we set up the model for which we want to have the performance drop evaluated:
 
@@ -88,16 +88,16 @@ an idea of where the threshold could be by looking at these values:
 
 .. code:: python
 
-    {0: -0.02368121940502793,
-     1: 0.017909161264480666,
-     2: 0.18565460365508413,
-     3: 0.07655405817715671,
-     4: 0.4327180164470878,
-     5: 0.16394693824418372,
-     6: -0.012876023845921625,
-     7: 0.01048781540981647,
-     8: 0.3921465005640224,
-     9: -0.01427065640301245}
+    {0: -0.0035681361984126747,
+     1: 0.041170843574652394,
+     2: 0.1920054944393057,
+     3: 0.07007527443645178,
+     4: 0.49871458125373913,
+     5: 0.1802858704499694,
+     6: 0.025536233845966705,
+     7: 0.024058931694668884,
+     8: 0.40901959802129045,
+     9: 0.004487448637912506}
 
 :class:`SelectByShuffling()` also stores the features that will be dropped based on the
 threshold indicated.
@@ -110,3 +110,17 @@ threshold indicated.
 
     [0, 1, 3, 6, 7, 9]
 
+If we now print the transformed data, we see that the features above were removed.
+
+..  code:: python
+
+    print(Xt.head())
+
+..  code:: python
+
+              2         4         5         8
+    0  0.061696 -0.044223 -0.034821  0.019907
+    1 -0.051474 -0.008449 -0.019163 -0.068332
+    2  0.044451 -0.045599 -0.034194  0.002861
+    3 -0.011595  0.012191  0.024991  0.022688
+    4 -0.036385  0.003935  0.015596 -0.031988