-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-37039][PS] Fix Series.astype
to work properly with missing value
#44570
Conversation
@@ -54,10 +54,7 @@ def test_astype(self): | |||
lambda: psser.astype(int_type), | |||
) | |||
|
|||
# TODO(SPARK-37039): the np.nan series.astype(bool) should be True |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should probably add this into migration guide though.
Oh, it seems the the failures are relevant, @itholic .
|
@itholic can you retrigger and/or fix the tests? |
Oh.. seems like we should separately handle the boolean data type. I got some personal errands right now, so let me take a look tomorrow. Thanks! |
Series.astype
to work properly with missing valueSeries.astype
to work properly with missing value
CI passed. @dongjoon-hyun @HyukjinKwon FYI |
Series.astype
to work properly with missing valueSeries.astype
to work properly with missing value
Merged to master. |
### What changes were proposed in this pull request? This PR followup for #44570 to add migration guide for behavior change. ### Why are the changes needed? We should notice user about any behavior change ### Does this PR introduce _any_ user-facing change? No API change. ### How was this patch tested? The existing CI should pass. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44684 from itholic/SPARK-37039-foolowup. Authored-by: Haejoon Lee <haejoon.lee@databricks.com> Signed-off-by: Kent Yao <yao@apache.org>
What changes were proposed in this pull request?
This PR proposes to fix
Series.astype
to work properly with missing value.Why are the changes needed?
To follow the behavior of latest Pandas.
Does this PR introduce any user-facing change?
Yes, the bug is fixed to follow the behavior of Pandas:
Before
After
How was this patch tested?
Enable the existing UTs.
Was this patch authored or co-authored using generative AI tooling?
No.