Skip to content

Remove fillna from Resampler #57348

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Feb 12, 2024
Merged

Remove fillna from Resampler #57348

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
62b0b53
Remove fillna from Resampler
phofl Feb 11, 2024
091a60c
Remove fillna from Resampler
phofl Feb 11, 2024
adafd0f
Update docs
phofl Feb 11, 2024
9c5b8cf
Merge remote-tracking branch 'upstream/main' into resample_fillna
phofl Feb 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion doc/redirects.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,6 @@ generated/pandas.core.resample.Resampler.backfill,../reference/api/pandas.core.r
generated/pandas.core.resample.Resampler.bfill,../reference/api/pandas.core.resample.Resampler.bfill
generated/pandas.core.resample.Resampler.count,../reference/api/pandas.core.resample.Resampler.count
generated/pandas.core.resample.Resampler.ffill,../reference/api/pandas.core.resample.Resampler.ffill
generated/pandas.core.resample.Resampler.fillna,../reference/api/pandas.core.resample.Resampler.fillna
generated/pandas.core.resample.Resampler.first,../reference/api/pandas.core.resample.Resampler.first
generated/pandas.core.resample.Resampler.get_group,../reference/api/pandas.core.resample.Resampler.get_group
generated/pandas.core.resample.Resampler.groups,../reference/api/pandas.core.resample.Resampler.groups
Expand Down
1 change: 0 additions & 1 deletion doc/source/reference/resampling.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@ Upsampling
Resampler.ffill
Resampler.bfill
Resampler.nearest
Resampler.fillna
Resampler.asfreq
Resampler.interpolate

Expand Down
3 changes: 2 additions & 1 deletion doc/source/whatsnew/v3.0.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,12 +106,13 @@ Removal of prior version deprecations/changes
- :func:`read_excel`, :func:`read_json`, :func:`read_html`, and :func:`read_xml` no longer accept raw string or byte representation of the data. That type of data must be wrapped in a :py:class:`StringIO` or :py:class:`BytesIO` (:issue:`53767`)
- All arguments except the first ``path``-like argument in IO writers are now keyword only (:issue:`54229`)
- Changed the default value of ``observed`` in :meth:`DataFrame.groupby` and :meth:`Series.groupby` to ``True`` (:issue:`51811`)
- Removed :meth:`DataFrame.applymap`, :meth:`Styler.applymap` and :meth:`Styler.applymap_index` (:issue:`52364`)
- Removed ``DataFrame.applymap``, ``Styler.applymap`` and ``Styler.applymap_index`` (:issue:`52364`)
- Removed ``DataFrame.bool`` and ``Series.bool`` (:issue:`51756`)
- Removed ``DataFrame.first`` and ``DataFrame.last`` (:issue:`53710`)
- Removed ``DataFrameGroupBy.grouper`` and ``SeriesGroupBy.grouper`` (:issue:`56521`)
- Removed ``DataFrameGroupby.fillna`` and ``SeriesGroupBy.fillna``` (:issue:`55719`)
- Removed ``Index.format``, use :meth:`Index.astype` with ``str`` or :meth:`Index.map` with a ``formatter`` function instead (:issue:`55439`)
- Removed ``Resample.fillna`` (:issue:`55719`)
- Removed ``Series.__int__`` and ``Series.__float__``. Call ``int(Series.iloc[0])`` or ``float(Series.iloc[0])`` instead. (:issue:`51131`)
- Removed ``Series.ravel`` (:issue:`56053`)
- Removed ``Series.view`` (:issue:`56054`)
Expand Down
188 changes: 2 additions & 186 deletions pandas/core/resample.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -631,8 +631,8 @@ def nearest(self, limit: int | None = None):

See Also
--------
backfill : Backward fill the new missing values in the resampled data.
pad : Forward fill ``NaN`` values.
bfill : Backward fill the new missing values in the resampled data.
ffill : Forward fill ``NaN`` values.

Examples
--------
Expand Down Expand Up @@ -686,9 +686,6 @@ def bfill(self, limit: int | None = None):

See Also
--------
bfill : Alias of backfill.
fillna : Fill NaN values using the specified method, which can be
'backfill'.
nearest : Fill NaN values with nearest neighbor starting from center.
ffill : Forward fill NaN values.
Series.fillna : Fill NaN values in the Series using the
Expand Down Expand Up @@ -767,177 +764,6 @@ def bfill(self, limit: int | None = None):
"""
return self._upsample("bfill", limit=limit)

@final
def fillna(self, method, limit: int | None = None):
"""
Fill missing values introduced by upsampling.

In statistics, imputation is the process of replacing missing data with
substituted values [1]_. When resampling data, missing values may
appear (e.g., when the resampling frequency is higher than the original
frequency).

Missing values that existed in the original data will
not be modified.

Parameters
----------
method : {'pad', 'backfill', 'ffill', 'bfill', 'nearest'}
Method to use for filling holes in resampled data

* 'pad' or 'ffill': use previous valid observation to fill gap
(forward fill).
* 'backfill' or 'bfill': use next valid observation to fill gap.
* 'nearest': use nearest valid observation to fill gap.

limit : int, optional
Limit of how many consecutive missing values to fill.

Returns
-------
Series or DataFrame
An upsampled Series or DataFrame with missing values filled.

See Also
--------
bfill : Backward fill NaN values in the resampled data.
ffill : Forward fill NaN values in the resampled data.
nearest : Fill NaN values in the resampled data
with nearest neighbor starting from center.
interpolate : Fill NaN values using interpolation.
Series.fillna : Fill NaN values in the Series using the
specified method, which can be 'bfill' and 'ffill'.
DataFrame.fillna : Fill NaN values in the DataFrame using the
specified method, which can be 'bfill' and 'ffill'.

References
----------
.. [1] https://en.wikipedia.org/wiki/Imputation_(statistics)

Examples
--------
Resampling a Series:

>>> s = pd.Series(
... [1, 2, 3], index=pd.date_range("20180101", periods=3, freq="h")
... )
>>> s
2018-01-01 00:00:00 1
2018-01-01 01:00:00 2
2018-01-01 02:00:00 3
Freq: h, dtype: int64

Without filling the missing values you get:

>>> s.resample("30min").asfreq()
2018-01-01 00:00:00 1.0
2018-01-01 00:30:00 NaN
2018-01-01 01:00:00 2.0
2018-01-01 01:30:00 NaN
2018-01-01 02:00:00 3.0
Freq: 30min, dtype: float64

>>> s.resample("30min").fillna("backfill")
2018-01-01 00:00:00 1
2018-01-01 00:30:00 2
2018-01-01 01:00:00 2
2018-01-01 01:30:00 3
2018-01-01 02:00:00 3
Freq: 30min, dtype: int64

>>> s.resample("15min").fillna("backfill", limit=2)
2018-01-01 00:00:00 1.0
2018-01-01 00:15:00 NaN
2018-01-01 00:30:00 2.0
2018-01-01 00:45:00 2.0
2018-01-01 01:00:00 2.0
2018-01-01 01:15:00 NaN
2018-01-01 01:30:00 3.0
2018-01-01 01:45:00 3.0
2018-01-01 02:00:00 3.0
Freq: 15min, dtype: float64

>>> s.resample("30min").fillna("pad")
2018-01-01 00:00:00 1
2018-01-01 00:30:00 1
2018-01-01 01:00:00 2
2018-01-01 01:30:00 2
2018-01-01 02:00:00 3
Freq: 30min, dtype: int64

>>> s.resample("30min").fillna("nearest")
2018-01-01 00:00:00 1
2018-01-01 00:30:00 2
2018-01-01 01:00:00 2
2018-01-01 01:30:00 3
2018-01-01 02:00:00 3
Freq: 30min, dtype: int64

Missing values present before the upsampling are not affected.

>>> sm = pd.Series(
... [1, None, 3], index=pd.date_range("20180101", periods=3, freq="h")
... )
>>> sm
2018-01-01 00:00:00 1.0
2018-01-01 01:00:00 NaN
2018-01-01 02:00:00 3.0
Freq: h, dtype: float64

>>> sm.resample("30min").fillna("backfill")
2018-01-01 00:00:00 1.0
2018-01-01 00:30:00 NaN
2018-01-01 01:00:00 NaN
2018-01-01 01:30:00 3.0
2018-01-01 02:00:00 3.0
Freq: 30min, dtype: float64

>>> sm.resample("30min").fillna("pad")
2018-01-01 00:00:00 1.0
2018-01-01 00:30:00 1.0
2018-01-01 01:00:00 NaN
2018-01-01 01:30:00 NaN
2018-01-01 02:00:00 3.0
Freq: 30min, dtype: float64

>>> sm.resample("30min").fillna("nearest")
2018-01-01 00:00:00 1.0
2018-01-01 00:30:00 NaN
2018-01-01 01:00:00 NaN
2018-01-01 01:30:00 3.0
2018-01-01 02:00:00 3.0
Freq: 30min, dtype: float64

DataFrame resampling is done column-wise. All the same options are
available.

>>> df = pd.DataFrame(
... {"a": [2, np.nan, 6], "b": [1, 3, 5]},
... index=pd.date_range("20180101", periods=3, freq="h"),
... )
>>> df
a b
2018-01-01 00:00:00 2.0 1
2018-01-01 01:00:00 NaN 3
2018-01-01 02:00:00 6.0 5

>>> df.resample("30min").fillna("bfill")
a b
2018-01-01 00:00:00 2.0 1
2018-01-01 00:30:00 NaN 3
2018-01-01 01:00:00 NaN 3
2018-01-01 01:30:00 6.0 5
2018-01-01 02:00:00 6.0 5
"""
warnings.warn(
f"{type(self).__name__}.fillna is deprecated and will be removed "
"in a future version. Use obj.ffill(), obj.bfill(), "
"or obj.nearest() instead.",
FutureWarning,
stacklevel=find_stack_level(),
)
return self._upsample(method, limit=limit)

@final
def interpolate(
self,
Expand Down Expand Up @@ -1808,11 +1634,6 @@ def _upsample(self, method, limit: int | None = None, fill_value=None):
Maximum size gap to fill when reindexing
fill_value : scalar, default None
Value to use for missing values

See Also
--------
.fillna: Fill NA/NaN values using the specified method.

"""
if self._from_selection:
raise ValueError(
Expand Down Expand Up @@ -1961,11 +1782,6 @@ def _upsample(self, method, limit: int | None = None, fill_value=None):
Maximum size gap to fill when reindexing.
fill_value : scalar, default None
Value to use for missing values.

See Also
--------
.fillna: Fill NA/NaN values using the specified method.

"""
# we may need to actually resample as if we are timestamps
if self.kind == "timestamp":
Expand Down
26 changes: 0 additions & 26 deletions pandas/tests/resample/test_resample_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -296,32 +296,6 @@ def test_transform_frame(on):
tm.assert_frame_equal(result, expected)


def test_fillna():
# need to upsample here
rng = date_range("1/1/2012", periods=10, freq="2s")
ts = Series(np.arange(len(rng), dtype="int64"), index=rng)
r = ts.resample("s")

expected = r.ffill()
msg = "DatetimeIndexResampler.fillna is deprecated"
with tm.assert_produces_warning(FutureWarning, match=msg):
result = r.fillna(method="ffill")
tm.assert_series_equal(result, expected)

expected = r.bfill()
with tm.assert_produces_warning(FutureWarning, match=msg):
result = r.fillna(method="bfill")
tm.assert_series_equal(result, expected)

msg2 = (
r"Invalid fill method\. Expecting pad \(ffill\), backfill "
r"\(bfill\) or nearest\. Got 0"
)
with pytest.raises(ValueError, match=msg2):
with tm.assert_produces_warning(FutureWarning, match=msg):
r.fillna(0)


@pytest.mark.parametrize(
"func",
[
Expand Down