Skip to content

BUG: improve future warning for boolean operations with missaligned indexes #62367

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: 2.3.x
Choose a base branch
from
+5 −3
Open

BUG: improve future warning for boolean operations with missaligned indexes #62367

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions pandas/core/series.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6161,9 +6161,11 @@ def _align_for_op(self, right, align_asobject: bool = False):
np.bool_,
):
warnings.warn(
"Operation between non boolean Series with different "
"indexes will no longer return a boolean result in "
"a future version. Cast both Series to object type "
"Operation between Series with different indexes "
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just add the nullable books to the check on L6161

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like I am misunderstanding something, should I add BooleanDtype() and "bool[pyarrow]" to the right.dtype not in check?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this would dismiss the warnings for these datatypes, which is not intended. The if block just contains the warning call.

On v2.3.x, boolean operations on misaligned indexes are converted to bool (non-nullable) and this is the main reason for this warning. For example:

import pandas as pd

a = pd.Series([False], index=['a'], dtype=pd.BooleanDtype())
b = pd.Series([True], index=['b'], dtype=pd.BooleanDtype())

# Have warning, convert series to non nullable
print(a & b)
# a    False
# b    False
# dtype: bool

# Have warning, convert series to non nullable
print(a | b)
# a    False
# b    False
# dtype: bool

# Have warning, convert series to non nullable
print(a ^ b)
# a    False
# b    False
# dtype: bool

When indexes are aligned, the data type is kept:

import pandas as pd

a = pd.Series([False, None], index=['a', 'b'], dtype=pd.BooleanDtype())
b = pd.Series([None, True], index=['a', 'b'], dtype=pd.BooleanDtype())

# No warning, keep datatype
print(a & b)
# a    False
# b     <NA>
# dtype: boolean

# No warning, keep datatype
print(a | b)
# a    <NA>
# b    True
# dtype: boolean

# No warning, keep datatype
print(a ^ b)
# a    <NA>
# b    <NA>
# dtype: boolean

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

None of those cases should be converting to object. For me they don't on main.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

None of those cases should be converting to object

The behavior changed on the main branch, and the warning is specific to v2.3.x to alert on the change.

On main branch, they no longer convert to object, but on v2.3.x does.

I really don't think that the behavior should change on a patch or minor release.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Woops, didn't see this was going into 2.3.x, never mind.

"that are not of numpy boolean or object dtype "
"will no longer return a numpy boolean result "
"in a future version. "
"Cast both Series to object type "
"to maintain the prior behavior.",
FutureWarning,
stacklevel=find_stack_level(),
Expand Down
Loading