Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-37495][PYTHON] Skip identical index checking of Series.compare when config 'compute.eager_check' is disabled #34750

Closed
wants to merge 2 commits into from
Closed

[SPARK-37495][PYTHON] Skip identical index checking of Series.compare when config 'compute.eager_check' is disabled #34750

wants to merge 2 commits into from

Conversation

dchvn
Copy link
Contributor

@dchvn dchvn commented Nov 30, 2021 •
edited

What changes were proposed in this pull request?

Skip identical index checking of Series.compare when config 'compute.eager_check' is disabled

Why are the changes needed?

Identical index checking is expensive, so we should use config 'compute.eager_check' to skip this one

Does this PR introduce any user-facing change?

Yes

Before this PR

>>> psser1 = ps.Series([1, 2, 3, 4, 5], index=pd.Index([1, 2, 3, 4, 5]))
>>> psser2 = ps.Series([1, 2, 3, 4, 5], index=pd.Index([1, 2, 4, 3, 6]))
>>> psser1.compare(psser2)
Traceback (most recent call last):                                              
  File "<stdin>", line 1, in <module>
  File "/u02/spark/python/pyspark/pandas/series.py", line 5851, in compare
    raise ValueError("Can only compare identically-labeled Series objects")
ValueError: Can only compare identically-labeled Series objects

After this PR, when config 'compute.eager_check' is False, pandas-on-Spark just proceeds and performs by ignoring the identical index checking.

>>> with ps.option_context("compute.eager_check", False):
...     psser1.compare(psser2)
... 
   self  other
3   3.0    4.0
4   4.0    3.0
5   5.0    NaN
6   NaN    5.0

How was this patch tested?

Unit tests

@dchvn dchvn marked this pull request as draft November 30, 2021 05:46
@SparkQA
Copy link

SparkQA commented Nov 30, 2021

Test build #145750 has finished for PR 34750 at commit 2bbb84d.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@dchvn
[SPARK-37495][PYTHON] Skip identical index checking of Series.compare…
dabc3c5 
… when config 'compute.eager_check' is disabled

Add user_guide docs

Add note to docstring
@SparkQA
Copy link

SparkQA commented Nov 30, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50221/

@dchvn dchvn marked this pull request as ready for review November 30, 2021 07:15
@SparkQA
Copy link

SparkQA commented Nov 30, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50221/

@SparkQA
Copy link

SparkQA commented Nov 30, 2021

Test build #145756 has finished for PR 34750 at commit dabc3c5.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 30, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50228/

@SparkQA
Copy link

SparkQA commented Nov 30, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50228/

@dchvn
Copy link
Contributor Author

dchvn commented Nov 30, 2021

CC @HyukjinKwon @itholic @Yikun FYI

HyukjinKwon
HyukjinKwon approved these changes Dec 1, 2021
View reviewed changes
Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good but @itholic and @Yikun FYI to double check.

@SparkQA
Copy link

SparkQA commented Dec 1, 2021

Test build #145785 has finished for PR 34750 at commit 37f22b9.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Dec 1, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50258/

@SparkQA
Copy link

SparkQA commented Dec 1, 2021

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/50258/

Yikun
Yikun approved these changes Dec 1, 2021
View reviewed changes
Copy link
Member

@Yikun Yikun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @dchvn

python/pyspark/pandas/series.py
@@ -5781,6 +5781,25 @@ def compare(
"""
Compare to another Series and show the differences.

.. note:: This API is slightly different from pandas when indexes from both Series
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good notes and doctest

@dchvn
Copy link
Contributor Author

dchvn commented Dec 4, 2021

Ping @HyukjinKwon :-) thanks

@HyukjinKwon
Copy link
Member

Merged to master.

@dchvn dchvn deleted the SPARK-37495 branch December 4, 2021 02:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Yikun Yikun Yikun approved these changes

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

Assignees
No one assigned
Labels
CORE PYTHON
Projects
None yet
Milestone
No milestone
4 participants
@dchvn @SparkQA @HyukjinKwon @Yikun