Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-46553][PS] FutureWarning for interpolate with object dtype #44550

Closed
wants to merge 2 commits into from

Conversation

itholic
Copy link
Contributor

@itholic itholic commented Jan 2, 2024

What changes were proposed in this pull request?

This PR proposes to issue a FutureWarning for (DataFrame|Series).interpolate with object dtype.

Why are the changes needed?

To match the behavior with Pandas. Using object dtype for interpolate is deprecated and will raise exception in the future version, so we should issue the proper warning such as Pandas does.

Does this PR introduce any user-facing change?

Given DataFrame below,

>>> psdf = ps.DataFrame({"A": ['a', 'b', 'c'], "B": [1, 2, 3]})
>>> psdf
   A  B
0  a  1
1  b  2
2  c  3

Before

>>> psdf.interpolate()  # Excluding column with object dtype without any warning unlike pandas
   B
0  1
1  2
2  3

After

>>> psdf.interpolate()  # Issuing a proper warning
FutureWarning: DataFrame.interpolate with object dtype is deprecated and will raise in a future version. Call df.infer_objects(copy=False) before interpolating instead.
  warnings.warn(
   B
0  1
1  2
2  3

How was this patch tested?

No behavior changes, so the existing CI should pass.

Was this patch authored or co-authored using generative AI tooling?

No.

@HyukjinKwon
Copy link
Member

Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants