Skip to content

Conversation

@itholic
Copy link
Contributor

@itholic itholic commented Sep 8, 2021

What changes were proposed in this pull request?

This PR proposes cleanup the deprecated APIs in missing/*.py, and raise proper warning message for the deprecated APIs such as pandas does.

Also remove the checking for pandas < 1.0, since now we only focus on following the behavior of latest pandas.

Why are the changes needed?

We should follow the deprecation of APIs of latest pandas.

Does this PR introduce any user-facing change?

Now the some APIs raise proper alternative message for deprecated functions such as pandas does.

How was this patch tested?

Ran dev/lint-python and manually check the pandas API documents one by one.

@SparkQA
Copy link

SparkQA commented Sep 8, 2021

Test build #143064 has finished for PR 33931 at commit de6d1fd.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 8, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47567/

@SparkQA
Copy link

SparkQA commented Sep 8, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47567/

@HyukjinKwon
Copy link
Member

Can you fix tests?

======================================================================
ERROR [0.038s]: test_missing (pyspark.pandas.tests.test_series.SeriesTest)
----------------------------------------------------------------------
pyspark.pandas.exceptions.PandasNotImplementedError: The method `pd.Series.slice_shift()` is deprecated in pandas and will therefore not be supported in pandas-on-Spark. Use `shift` instead.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/__w/spark/spark/python/pyspark/pandas/tests/test_series.py", line 1040, in test_missing
    getattr(psser, name)()
AssertionError: "method.*Series.*tshift.*is deprecated" does not match "The method `pd.Series.slice_shift()` is deprecated in pandas and will therefore not be supported in pandas-on-Spark. Use `shift` instead."

----------------------------------------------------------------------

Otherwise looks good. cc @ueshin too fyi

Copy link
Member

@ueshin ueshin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise, LGTM, pending tests.

@SparkQA
Copy link

SparkQA commented Sep 9, 2021

Test build #143107 has finished for PR 33931 at commit 3550117.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Sep 9, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47611/

@SparkQA
Copy link

SparkQA commented Sep 9, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/47611/

@HyukjinKwon
Copy link
Member

Merged to master.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants