Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-42593][PS] Deprecate & remove the APIs that will be removed in pandas 2.0. #40216

Closed
wants to merge 2 commits into from

Conversation

itholic
Copy link
Contributor

@itholic itholic commented Feb 28, 2023

What changes were proposed in this pull request?

This PR proposes to mark the APIs as deprecated or remove the APIs that will be deprecated or removed in upcoming pandas 2.0.0 release.

See What's new in 2.0.0 for more detail.

Why are the changes needed?

We should match the behavior to pandas API.

Does this PR introduce any user-facing change?

Yes, some APIs will be removed, so they will be no longer available.

How was this patch tested?

Fixed UTs when necessary case.

@itholic itholic changed the title [WIP][SPARK-42593][PS] Deprecate & remove the APIs that will be removed in pandas 2.0. [SPARK-42593][PS] Deprecate & remove the APIs that will be removed in pandas 2.0. Mar 1, 2023
@itholic itholic marked this pull request as ready for review March 1, 2023 05:31
@HyukjinKwon
Copy link
Member

Merged to master and branch-3.4.

HyukjinKwon pushed a commit that referenced this pull request Mar 1, 2023
… pandas 2.0

### What changes were proposed in this pull request?

This PR proposes to mark the APIs as deprecated or remove the APIs that will be deprecated or removed in upcoming pandas 2.0.0 release.

See [What's new in 2.0.0](https://pandas.pydata.org/pandas-docs/version/2.0/whatsnew/v2.0.0.html#removal-of-prior-version-deprecations-changes) for more detail.

### Why are the changes needed?

We should match the behavior to pandas API.

### Does this PR introduce _any_ user-facing change?

Yes, some APIs will be removed, so they will be no longer available.

### How was this patch tested?

Fixed UTs when necessary case.

Closes #40216 from itholic/SPARK-42593.

Authored-by: itholic <haejoon.lee@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 9d2fe90)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
@gatorsmile
Copy link
Member

Let us mention all the breaking changes and deprecation in both release notes and migration guides

@itholic itholic deleted the SPARK-42593 branch April 22, 2023 05:49
@bjornjorgensen
Copy link
Contributor

hmm..
F05.info()

TypeError                                 Traceback (most recent call last)
Cell In[12], line 1
----> 1 F05.info()

File /opt/spark/python/pyspark/pandas/frame.py:12167, in DataFrame.info(self, verbose, buf, max_cols, null_counts)
  12163     count_func = self.count
  12164     self.count = (  # type: ignore[assignment]
  12165         lambda: count_func()._to_pandas()  # type: ignore[assignment, misc, union-attr]
  12166     )
> 12167     return pd.DataFrame.info(
  12168         self,  # type: ignore[arg-type]
  12169         verbose=verbose,
  12170         buf=buf,
  12171         max_cols=max_cols,
  12172         memory_usage=False,
  12173         null_counts=null_counts,
  12174     )
  12175 finally:
  12176     del self._data

TypeError: DataFrame.info() got an unexpected keyword argument 'null_counts'

Hope we can get a better description as error messages here.

@bjornjorgensen
Copy link
Contributor

#40913 can be a fix for this.

snmvaughan pushed a commit to snmvaughan/spark that referenced this pull request Jun 20, 2023
… pandas 2.0

### What changes were proposed in this pull request?

This PR proposes to mark the APIs as deprecated or remove the APIs that will be deprecated or removed in upcoming pandas 2.0.0 release.

See [What's new in 2.0.0](https://pandas.pydata.org/pandas-docs/version/2.0/whatsnew/v2.0.0.html#removal-of-prior-version-deprecations-changes) for more detail.

### Why are the changes needed?

We should match the behavior to pandas API.

### Does this PR introduce _any_ user-facing change?

Yes, some APIs will be removed, so they will be no longer available.

### How was this patch tested?

Fixed UTs when necessary case.

Closes apache#40216 from itholic/SPARK-42593.

Authored-by: itholic <haejoon.lee@databricks.com>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
(cherry picked from commit 9d2fe90)
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants