DEPR: dropping nuisance columns in rolling aggregations #42834
Conversation
jbrockmendel
added
Nuisance Columns
| expected = df.ewm(halflife=1.0, min_periods=min_periods).mean() | ||
| with tm.assert_produces_warning(FutureWarning, match="nuisance columns"): | ||
| # GH#42738 | ||
| result = df.ewm(halflife=halflife, min_periods=min_periods, times=times).mean() |
There was a problem hiding this comment.
Also should probably add a deprecation here too that allows times to be a string of a column label (which is already kinda flawed because a column label isn't always a string). A column of times will always be a nuisance column I think
pandas/pandas/core/window/ewm.py
Line 317 in a5f8c9a
There was a problem hiding this comment.
can this be done separately? this is less obvious to me
pandas/tests/window/test_numba.py
Outdated
| result = ewm.mean(engine="numba", engine_kwargs=engine_kwargs) | ||
| expected = ewm.mean(engine="cython") | ||
|
|
||
| # TODO: why only in these cases? |
There was a problem hiding this comment.
Because column "A" is string dtype and in the non-group case there's an attempt to agg over the column, while in the groupby case, the grouping column should be dropped internally before hand
doc/source/whatsnew/v1.4.0.rst
Outdated
| @@ -159,6 +159,7 @@ Deprecations | |||
| - Deprecated treating ``numpy.datetime64`` objects as UTC times when passed to the :class:`Timestamp` constructor along with a timezone. In a future version, these will be treated as wall-times. To retain the old behavior, use ``Timestamp(dt64).tz_localize("UTC").tz_convert(tz)`` (:issue:`24559`) | |||
| - Deprecated ignoring missing labels when indexing with a sequence of labels on a level of a MultiIndex (:issue:`42351`) | |||
| - Creating an empty Series without a dtype will now raise a more visible ``FutureWarning`` instead of a ``DeprecationWarning`` (:issue:`30017`) | |||
| - Deprecated dropping of nuisance columns in :class:`Rolling` aggregations (:issue:`42738`) | |||
There was a problem hiding this comment.
Good to include Expanding and EWM as well.
|
comments addressed + rebased + green |
| if 0 != len(new_mgr.items) != len(mgr.items): | ||
| # GH#42738 ignore_failures dropped nuisance columns | ||
| warnings.warn( | ||
| "Dropping of nuisance columns in rolling operations " |
There was a problem hiding this comment.
can you list the columns that are problematic in the error message?
… into depr-window-nuisance
we did this in 1.3 for DataFrame reductions and GroupBy reductions. AFAIK this is the last place we use this pattern.
cc @mroeschke