Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ENH: Processing of .mask() for pd.NA #56844 #58730

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

ENH: Processing of .mask() for pd.NA #56844 #58730

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9cebdf8
Fix #56844: function mask handling None values as condition
mafaldam Apr 3, 2024
a0369c7
Update pandas/core/generic.py
mafaldam Apr 17, 2024
4f75357
Fix #56844: function mask handling None values as condition
mafaldam Apr 3, 2024
3b20b84
Update pandas/core/generic.py
mafaldam Apr 17, 2024
15c74b5
fix: rebase with master
mafaldam May 8, 2024
d005ecc
Update pandas/tests/frame/indexing/test_mask.py
mafaldam May 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v3.0.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -482,6 +482,7 @@ Other
- Bug in :class:`DataFrame` when passing a ``dict`` with a NA scalar and ``columns`` that would always return ``np.nan`` (:issue:`57205`)
- Bug in :func:`eval` where the names of the :class:`Series` were not preserved when using ``engine="numexpr"``. (:issue:`10239`)
- Bug in :func:`unique` on :class:`Index` not always returning :class:`Index` (:issue:`57043`)
- Bug in :func:`mask` to handle NaN values in condition of function. (:issue:`56844`)
- Bug in :meth:`DataFrame.eval` and :meth:`DataFrame.query` which caused an exception when using NumPy attributes via ``@`` notation, e.g., ``df.eval("@np.floor(a)")``. (:issue:`58041`)
- Bug in :meth:`DataFrame.eval` and :meth:`DataFrame.query` which did not allow to use ``tan`` function. (:issue:`55091`)
- Bug in :meth:`DataFrame.sort_index` when passing ``axis="columns"`` and ``ignore_index=True`` and ``ascending=False`` not returning a :class:`RangeIndex` columns (:issue:`57293`)
Expand Down
3 changes: 3 additions & 0 deletions pandas/core/generic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9999,6 +9999,9 @@ def mask(
cond = common.apply_if_callable(cond, self)
other = common.apply_if_callable(other, self)

if isinstance(cond, (ABCDataFrame, ABCSeries)):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than a fillna like this do you see where the logic is breaking down that causes .mask to treat pd.NA the same was as True? We shouldn't have to mess around with pd.NA equality semantics for this fix

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For indexing this goes through check_array_indexer which essentially also converts NA to False (but lower in the stack):

pandas/pandas/core/indexers/utils.py

Lines 532 to 534 in 2aa155a

if is_bool_dtype(dtype):
if isinstance(dtype, ExtensionDtype):
indexer = indexer.to_numpy(dtype=bool, na_value=False)

cond = cond.fillna(False)

# see gh-21891
if not hasattr(cond, "__invert__"):
cond = np.array(cond)
Expand Down
9 changes: 9 additions & 0 deletions pandas/tests/frame/indexing/test_mask.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,3 +150,12 @@ def test_mask_inplace_no_other():
df.mask(cond, inplace=True)
expected = DataFrame({"a": [np.nan, 2], "b": ["x", np.nan]})
tm.assert_frame_equal(df, expected)


def test_mask_with_NA():
df = DataFrame({"A": [0, 1, 2]})
cond = Series([True, False, pd.NA], dtype=pd.BooleanDtype())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To pass the tests, I think dtype=BooleanDtype() should be used here, and from pandas import BooleanDtype at the top of the file, just like other dtypes.


result = df.mask(cond, other=100)
expected = DataFrame({"A": [100, 1, 2]})
tm.assert_frame_equal(result, expected)
Loading