-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-16369][MLlib] tallSkinnyQR of RowMatrix should aware of empty partition #14049
Conversation
Test build #61735 has finished for PR 14049 at commit
|
partRows.foreach { row => | ||
bdm(i, ::) := row.asBreeze.t | ||
i += 1 | ||
if (partRows.length == 0) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you just flatMap this? so that you don't have to handle None below? or filter then map?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! I changed it to filter then map.
Test build #61763 has finished for PR 14049 at commit
|
@yinxusen if you resolve the conflicts I'll merge. |
Test build #61932 has finished for PR 14049 at commit
|
Merged to master/2.0 |
…partition ## What changes were proposed in this pull request? tallSkinnyQR of RowMatrix should aware of empty partition, which could cause exception from Breeze qr decomposition. See the [archived dev mail](https://mail-archives.apache.org/mod_mbox/spark-dev/201510.mbox/%3CCAF7ADNrycvPL3qX-VZJhq4OYmiUUhoscut_tkOm63Cm18iK1tQmail.gmail.com%3E) for more details. ## How was this patch tested? Scala unit test. Author: Xusen Yin <yinxusen@gmail.com> Closes #14049 from yinxusen/SPARK-16369. (cherry picked from commit 255d74f) Signed-off-by: Sean Owen <sowen@cloudera.com>
What changes were proposed in this pull request?
tallSkinnyQR of RowMatrix should aware of empty partition, which could cause exception from Breeze qr decomposition.
See the archived dev mail for more details.
How was this patch tested?
Scala unit test.