Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Python] Add is_nan for Python #34154

Closed
Fokko opened this issue Feb 12, 2023 · 4 comments · Fixed by #34184
Closed

[Python] Add is_nan for Python #34154

Fokko opened this issue Feb 12, 2023 · 4 comments · Fixed by #34184
Assignees
@Fokko
Labels
Component: Python Type: enhancement
Milestone
12.0.0

Comments

@Fokko
Copy link
Contributor

Fokko commented Feb 12, 2023

Describe the enhancement requested

We currently use field.is_null(nan_is_null=True) & field.is_valid() for filtering on NaN values, but would be great to fall back to is_nan. The abovementioned expression doesn't seem to catch NaN values.

Component(s)

Python
@jorisvandenbossche
Copy link
Member

I think an is_nan kernel should already exist? (https://arrow.apache.org/docs/python/api/compute.html#categorizations)

@Fokko
Copy link
Contributor Author

Fokko commented Feb 14, 2023

Sorry, I wasn't clear. I meant as part of the expressions API.

@jorisvandenbossche
Copy link
Member

Ah, I see. Now, that actually works as well with the functions (something we probably need to document better):

In [2]: pc.is_nan(pc.field("a"))
Out[2]: <pyarrow.compute.Expression is_nan(a)>

But given that we have a method on the expression for is_null, we could maybe do that for is_nan as well (although is_nan is specific to one data type, while is_null is generic).

Fokko added a commit to Fokko/arrow that referenced this issue Feb 14, 2023
@Fokko
apacheGH-34154: [Python] Add is_nan expression
de3e379 
Closes apache#34154
Fokko added a commit to Fokko/arrow that referenced this issue Feb 14, 2023
@Fokko
apacheGH-34154: [Python] Add is_nan expression
78ec952 
Closes apache#34154
@Fokko Fokko mentioned this issue Feb 14, 2023
Merged
Fokko added a commit to Fokko/arrow that referenced this issue Feb 14, 2023
@Fokko
apacheGH-34154: [Python] Add is_nan expression
36db249 
Closes apache#34154
jorisvandenbossche added a commit that referenced this issue Mar 9, 2023
@Fokko @jorisvandenbossche
GH-34154: [Python] Add is_nan method to Array and Expression (#34184)
2ff4e3a 
* Closes: GH-34154

Lead-authored-by: Fokko Driesprong <fokko@tabular.io>
Co-authored-by: Fokko Driesprong <fokko@apache.org>
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@jorisvandenbossche jorisvandenbossche added this to the 12.0.0 milestone Mar 9, 2023
@jorisvandenbossche
Copy link
Member

Issue resolved by pull request 34184
#34184

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Fokko Fokko

Labels
Component: Python Type: enhancement
Projects
None yet
Milestone
12.0.0
Development

Successfully merging a pull request may close this issue.

GH-34154: [Python] Add is_nan method to Array and Expression Fokko/arrow
2 participants
@jorisvandenbossche @Fokko