Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix spatial join when left and right rdd have different PartitionerIndex #3175

Open
wants to merge 3 commits into
base: master
from

Conversation

@soxofaan
Copy link
Contributor

soxofaan commented Jan 9, 2020

Overview

Fixes issue described at #3168
The original implementation blindly assumed same partition indexing, causing empty join when using custom partitioning with one of the RDDs being joined.
This PR adds tests for this use case and fixes the issue by adding an additional check that same PartitionIndexer is used, so that reshuffle is triggered when that is not the case.
PR also fixed some inactive asserts in same file.
It also addresses partitioning mismatch in ReorderedSpaceRDD

Checklist

  • docs/CHANGELOG.rst updated, if necessary
  • Module Hierarcy updated, if necessary
  • docs guides update, if necessary
  • New user API has useful Scaladoc strings
  • Unit tests added for bug-fix or new feature

Closes #3168

@soxofaan soxofaan force-pushed the soxofaan:issue3168-spatial-join branch 5 times, most recently from 9920fac to df7be80 Jan 9, 2020
soxofaan added 3 commits Jan 9, 2020
Signed-off-by: Stefaan Lippens <stefaan.lippens@vito.be>
…eredRDD and SpacePartitioner

Signed-off-by: Stefaan Lippens <stefaan.lippens@vito.be>
Signed-off-by: Stefaan Lippens <stefaan.lippens@vito.be>
@soxofaan soxofaan force-pushed the soxofaan:issue3168-spatial-join branch from df7be80 to d77637f Jan 10, 2020
@echeipesh echeipesh self-assigned this Jan 10, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.