-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-2967][SQL] Fix sort based shuffle for spark sql. #2066
Conversation
QA tests have started for PR 2066 at commit
|
LGTM |
QA tests have finished for PR 2066 at commit
|
Thanks for looking this over! Merged to master and 1.1 |
Add explicit row copies when sort based shuffle is on. Author: Michael Armbrust <michael@databricks.com> Closes #2066 from marmbrus/sortShuffle and squashes the following commits: fcd7bb2 [Michael Armbrust] Fix sort based shuffle for spark sql. (cherry picked from commit a2e658d) Signed-off-by: Michael Armbrust <michael@databricks.com>
val mutablePair = new MutablePair[Row, Row]() | ||
iter.map(r => mutablePair.update(hashExpressions(r), r)) | ||
if (sortBasedShuffleOn) { | ||
iter.map(r => (hashExpressions(r), r.copy())) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi, don't we need to copy hashExpressions(r)
here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch! Fixed here: #2072
Add explicit row copies when sort based shuffle is on. Author: Michael Armbrust <michael@databricks.com> Closes apache#2066 from marmbrus/sortShuffle and squashes the following commits: fcd7bb2 [Michael Armbrust] Fix sort based shuffle for spark sql.
… shuffle fix. Follow-up to apache#2066 Author: Michael Armbrust <michael@databricks.com> Closes apache#2072 from marmbrus/sortShuffle and squashes the following commits: 2ff8114 [Michael Armbrust] Fix bug
Add explicit row copies when sort based shuffle is on.