Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-36273][Shuffle] Fix identical values comparison #33493

Closed
wants to merge 1 commit into from
Closed

[SPARK-36273][Shuffle] Fix identical values comparison #33493

wants to merge 1 commit into from

Conversation

almogtavor
Copy link
Contributor

This commit fixes the use of the "o.appAttemptId" variable instead of the mistaken "appAttemptId" variable. The current situation is a comparison of identical values. Jira issue report SPARK-36273.

What changes were proposed in this pull request?

This is a patch for SPARK-35546 which is needed for push-based shuffle.

Why are the changes needed?

A very minor fix of adding the reference from the other "FinalizeShuffleMerge".

Does this PR introduce any user-facing change?

No.

How was this patch tested?

No unit tests were added. It's a pretty logical change.

@almogtavor
Fix identical values comparison
fb64458 
This commit fixes the use of the "o.appAttemptId" variable instead of the mistaken "appAttemptId" variable. The current situation is a comparison of identical values
@github-actions github-actions bot added the CORE label Jul 23, 2021
@srowen
Copy link
Member

srowen commented Jul 23, 2021

Good catch - looks like a correct fix.

@srowen
Copy link
Member

srowen commented Jul 23, 2021

Jenkins test this please

@SparkQA
Copy link

SparkQA commented Jul 23, 2021

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46076/

@SparkQA
Copy link

SparkQA commented Jul 23, 2021

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46076/

@SparkQA
Copy link

SparkQA commented Jul 23, 2021

Test build #141558 has finished for PR 33493 at commit fb64458.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen srowen closed this in 530c8ad Jul 23, 2021
srowen pushed a commit that referenced this pull request Jul 23, 2021
@almogtavor @srowen
[SPARK-36273][SHUFFLE] Fix identical values comparison
3a0184d 
This commit fixes the use of the "o.appAttemptId" variable instead of the mistaken "appAttemptId" variable. The current situation is a comparison of identical values. Jira issue report SPARK-36273.

### What changes were proposed in this pull request?
This is a patch for SPARK-35546 which is needed for push-based shuffle.

### Why are the changes needed?
A very minor fix of adding the reference from the other "FinalizeShuffleMerge".

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
No unit tests were added. It's a pretty logical change.

Closes #33493 from almogtavor/patch-1.

Authored-by: Almog Tavor <70065337+almogtavor@users.noreply.github.com>
Signed-off-by: Sean Owen <srowen@gmail.com>
(cherry picked from commit 530c8ad)
Signed-off-by: Sean Owen <srowen@gmail.com>
@srowen
Copy link
Member

srowen commented Jul 23, 2021

Merged to master/3.2. It doesn't cherry-pick cleanly to 3.1 (may not be applicable, haven't looked). @almogtavor if you're interested, and find this can backport to 3.1, open another PR vs 3.1 and I'll merge that. Thanks!

@dongjoon-hyun dongjoon-hyun mentioned this pull request Jul 23, 2021
Closed
dongjoon-hyun
dongjoon-hyun reviewed Jul 23, 2021
View reviewed changes
...rk-shuffle/src/main/java/org/apache/spark/network/shuffle/protocol/FinalizeShuffleMerge.java
@@ -68,7 +68,7 @@ public boolean equals(Object other) {
if (other != null && other instanceof FinalizeShuffleMerge) {
FinalizeShuffleMerge o = (FinalizeShuffleMerge) other;
return Objects.equal(appId, o.appId)
&& appAttemptId == appAttemptId
&& appAttemptId == o.appAttemptId
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this was introduced via https://github.com/apache/spark/pull/33078/files#diff-a4d872ac22f3cbdfb587125ee9dc4c70fe9efca58e3de5b31dad227f689046f9R71 which landed at master/3.2 4 days ago.

So, we don't need this at branch-3.1.

@dongjoon-hyun
Copy link
Member

+1, late LGTM.
Thank you, @almogtavor and @srowen .

cc @zhouyejoe , @mridulm , @Ngone51

@srowen
Copy link
Member

srowen commented Jul 23, 2021

Ah right, should've done my homework. All set

@Ngone51
Copy link
Member

Ngone51 commented Jul 24, 2021

Late LGTM!

Good catch @almogtavor

@mridulm
Copy link
Contributor

mridulm commented Jul 25, 2021

Late LGTM, thanks for fixing this @almogtavor !
And thanks for the ping @dongjoon-hyun

@zhouyejoe
Copy link
Contributor

Thanks for fixing this @almogtavor

domybest11 pushed a commit to domybest11/spark that referenced this pull request Jun 15, 2022
@almogtavor @JQ-Cao
[SPARK-36273][SHUFFLE] Fix identical values comparison
cc954f8 
This commit fixes the use of the "o.appAttemptId" variable instead of the mistaken "appAttemptId" variable. The current situation is a comparison of identical values. Jira issue report SPARK-36273.

### What changes were proposed in this pull request?
This is a patch for SPARK-35546 which is needed for push-based shuffle.

### Why are the changes needed?
A very minor fix of adding the reference from the other "FinalizeShuffleMerge".

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
No unit tests were added. It's a pretty logical change.

Closes apache#33493 from almogtavor/patch-1.

Authored-by: Almog Tavor <70065337+almogtavor@users.noreply.github.com>
Signed-off-by: Sean Owen <srowen@gmail.com>
(cherry picked from commit 530c8ad)
Signed-off-by: Sean Owen <srowen@gmail.com>
wangyum pushed a commit that referenced this pull request May 26, 2023
@almogtavor
[SPARK-36273][SHUFFLE] Fix identical values comparison
09db3a0 
This commit fixes the use of the "o.appAttemptId" variable instead of the mistaken "appAttemptId" variable. The current situation is a comparison of identical values. Jira issue report SPARK-36273.

### What changes were proposed in this pull request?
This is a patch for SPARK-35546 which is needed for push-based shuffle.

### Why are the changes needed?
A very minor fix of adding the reference from the other "FinalizeShuffleMerge".

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
No unit tests were added. It's a pretty logical change.

Closes #33493 from almogtavor/patch-1.

Authored-by: Almog Tavor <70065337+almogtavor@users.noreply.github.com>
Signed-off-by: Sean Owen <srowen@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dongjoon-hyun dongjoon-hyun dongjoon-hyun left review comments

Assignees
No one assigned
Labels
CORE
Projects
None yet
Milestone
No milestone
7 participants
@almogtavor @srowen @SparkQA @dongjoon-hyun @Ngone51 @mridulm @zhouyejoe