Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid listifying left/right operand in ExtensionArray ops dispatching #22922

Closed
jorisvandenbossche opened this issue Oct 1, 2018 · 8 comments

Comments

Projects
None yet
3 participants
@jorisvandenbossche
Copy link
Member

commented Oct 1, 2018

We now have the follow code in the ExtensionArray ops dispatching:

pandas/pandas/core/ops.py

Lines 1169 to 1192 in 2f1b842

# TODO(jreback)
# we need to listify to avoid ndarray, or non-same-type extension array
# dispatching
if is_extension_array_dtype(left):
new_left = left.values
if isinstance(right, np.ndarray):
# handle numpy scalars, this is a PITA
# TODO(jreback)
new_right = lib.item_from_zerodim(right)
if is_scalar(new_right):
new_right = [new_right]
new_right = list(new_right)
elif is_extension_array_dtype(right) and type(left) != type(right):
new_right = list(new_right)
else:
new_right = right
else:
new_left = list(left.values)
new_right = right

There are a few places where we convert the values to a list, which should be avoided.

@TomAugspurger

This comment has been minimized.

Copy link
Contributor

commented Oct 1, 2018

This specific case of Series[EA] + EA (where the EAs are the same type) seems very easy to handle.

@jorisvandenbossche

This comment has been minimized.

Copy link
Member Author

commented Oct 1, 2018

Looking at the code, it seems that that case is already working OK, if the type(EA) is equal.

@TomAugspurger

This comment has been minimized.

Copy link
Contributor

commented Oct 1, 2018

Hmm is it? I thought that's what cased the NameError. I think we need something like type(new_left) is type(right), where new_left has unboxed the EA from the Series. Haven't looked closely though.

@jorisvandenbossche

This comment has been minimized.

Copy link
Member Author

commented Oct 1, 2018

Yes, it's what caused the NameError, but this is about the conversion to list?

@TomAugspurger

This comment has been minimized.

Copy link
Contributor

commented Oct 1, 2018

Correct. I'm saying we shouldn't even reach that elif in the case of Series[EA] + EA.

elif is_extension_array_dtype(right) and type(new_left) == type(right):
    # EA + EA. This is new.
    new_right = right
elif is_extension_array_dtype(right) and type(left) != type(right):
    new_right = list(right)
@TomAugspurger

This comment has been minimized.

Copy link
Contributor

commented Oct 14, 2018

Corner case in pandas master with object-dtype

In [1]: import pandas as pd

In [2]: 1 ** pd.Series([None], dtype=object)
Out[2]:
0    NaN
dtype: object

That's supposed to be 1, since 1 ** anything is 1

In [4]: 1 ** pd.Series([np.nan])
Out[4]:
0    1.0
dtype: float64
@jorisvandenbossche

This comment has been minimized.

Copy link
Member Author

commented Oct 15, 2018

@TomAugspurger is that related to the listifying?

@TomAugspurger

This comment has been minimized.

Copy link
Contributor

commented Oct 15, 2018

Whoops, got my issues mixed up. That was for #22022

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.