Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-1955][GraphX]: VertexRDD can incorrectly assume index sharing #4705

Closed
wants to merge 1 commit into from
Closed

[SPARK-1955][GraphX]: VertexRDD can incorrectly assume index sharing #4705

wants to merge 1 commit into from

Conversation

brennonyork
Copy link

Fixes the issue whereby when VertexRDD's are diffed, innerJoined, or leftJoined and have different partition sizes they fail under the zipPartitions method. This fix tests whether the partitions are equal or not and, if not, will repartition the other to match the partition size of the calling VertexRDD.

@AmplabJenkins
Copy link

Can one of the admins verify this patch?

@brennonyork
Copy link
Author

/cc @ankurdave could you review, thanks!

@ankurdave
Copy link
Contributor

ok to test

@SparkQA
Copy link

SparkQA commented Feb 21, 2015

Test build #27804 has started for PR 4705 at commit 0882590.

  • This patch merges cleanly.

@SparkQA
Copy link

SparkQA commented Feb 21, 2015

Test build #27804 has finished for PR 4705 at commit 0882590.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27804/
Test PASSed.

@srowen
Copy link
Member

srowen commented Feb 21, 2015

LGTM although @ankurdave should probably have the final say.

@brennonyork
Copy link
Author

@ankurdave any thoughts? This is blocking progress on a few other GraphX JIRA's (SPARK-4600, SPARK-5790), so whenever you get a chance that'd be great, thanks!

@brennonyork
Copy link
Author

bump

@maropu
Copy link
Member

maropu commented Feb 25, 2015

@brennonyork I'll add unit tests for your patch before/after the patch merged.

@brennonyork
Copy link
Author

@maropu thanks!

asfgit pushed a commit that referenced this pull request Feb 25, 2015
@ankurdave
[SPARK-1955][GraphX]: VertexRDD can incorrectly assume index sharing
8073767 
Fixes the issue whereby when VertexRDD's are `diff`ed, `innerJoin`ed, or `leftJoin`ed and have different partition sizes they fail under the `zipPartitions` method. This fix tests whether the partitions are equal or not and, if not, will repartition the other to match the partition size of the calling VertexRDD.

Author: Brennon York <brennon.york@capitalone.com>

Closes #4705 from brennonyork/SPARK-1955 and squashes the following commits:

0882590 [Brennon York] updated to properly handle differently-partitioned vertexRDDs

(cherry picked from commit 9f603fc)
Signed-off-by: Ankur Dave <ankurdave@gmail.com>
@asfgit asfgit closed this in 9f603fc Feb 25, 2015
asfgit pushed a commit that referenced this pull request Feb 25, 2015
@ankurdave
[SPARK-1955][GraphX]: VertexRDD can incorrectly assume index sharing
00112ba 
Fixes the issue whereby when VertexRDD's are `diff`ed, `innerJoin`ed, or `leftJoin`ed and have different partition sizes they fail under the `zipPartitions` method. This fix tests whether the partitions are equal or not and, if not, will repartition the other to match the partition size of the calling VertexRDD.

Author: Brennon York <brennon.york@capitalone.com>

Closes #4705 from brennonyork/SPARK-1955 and squashes the following commits:

0882590 [Brennon York] updated to properly handle differently-partitioned vertexRDDs

(cherry picked from commit 9f603fc)
Signed-off-by: Ankur Dave <ankurdave@gmail.com>
@ankurdave
Copy link
Contributor

Thanks! Merged into master, branch-1.3, and branch-1.2.

@ankurdave ankurdave mentioned this pull request Feb 25, 2015
Closed
@brennonyork brennonyork mentioned this pull request Mar 14, 2015
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
6 participants
@brennonyork @AmplabJenkins @ankurdave @SparkQA @srowen @maropu