Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-36333][PYTHON] Reuse isnull where the null check is needed #33562

Closed
wants to merge 2 commits into from
Closed

[SPARK-36333][PYTHON] Reuse isnull where the null check is needed #33562

wants to merge 2 commits into from

Conversation

ueshin
Copy link
Member

@ueshin ueshin commented Jul 28, 2021 •
edited

What changes were proposed in this pull request?

Reuse IndexOpsMixin.isnull() where the null check is needed.

Why are the changes needed?

There are some places where we can reuse IndexOpsMixin.isnull() instead of directly using Spark Column.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing tests.

@ueshin
Copy link
Member Author

ueshin commented Jul 28, 2021

cc @xinrong-databricks @HyukjinKwon @itholic

@ueshin ueshin changed the title [SPARK-36333][PYTHON] Reuse isnull where the nullability check is needed [SPARK-36333][PYTHON] Reuse isnull where the null check is needed Jul 28, 2021
@SparkQA
Copy link

SparkQA commented Jul 29, 2021

Test build #141793 has finished for PR 33562 at commit 21708a0.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jul 29, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46306/

@SparkQA
Copy link

SparkQA commented Jul 29, 2021

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46306/

itholic
itholic approved these changes Jul 29, 2021
View reviewed changes
Copy link
Contributor

@itholic itholic left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise, LGTM.

I'm not sure #33562 (comment) is necessary, though.

@SparkQA
Copy link

SparkQA commented Jul 29, 2021

Test build #141857 has finished for PR 33562 at commit 3ea75f4.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jul 29, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46368/

@ueshin
Copy link
Member Author

ueshin commented Jul 29, 2021

Thanks! merging to master/3.2

@ueshin ueshin closed this in 07ed82b Jul 29, 2021
ueshin added a commit that referenced this pull request Jul 29, 2021
@ueshin
[SPARK-36333][PYTHON] Reuse isnull where the null check is needed
359dfd7 
### What changes were proposed in this pull request?

Reuse `IndexOpsMixin.isnull()` where the null check is needed.

### Why are the changes needed?

There are some places where we can reuse `IndexOpsMixin.isnull()` instead of directly using Spark `Column`.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Existing tests.

Closes #33562 from ueshin/issues/SPARK-36333/reuse_isnull.

Authored-by: Takuya UESHIN <ueshin@databricks.com>
Signed-off-by: Takuya UESHIN <ueshin@databricks.com>
(cherry picked from commit 07ed82b)
Signed-off-by: Takuya UESHIN <ueshin@databricks.com>
@SparkQA
Copy link

SparkQA commented Jul 29, 2021

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46368/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

@itholic itholic itholic approved these changes

@xinrong-meng xinrong-meng xinrong-meng approved these changes

Assignees
No one assigned
Labels
CORE PYTHON
Projects
None yet
Milestone
No milestone
5 participants
@ueshin @SparkQA @HyukjinKwon @itholic @xinrong-meng