[SPARK-6685][ML]Use DSYRK to compute AtA in ALS #19536
Conversation
set stack size > 128 and comments added
|
Hi @yanboliang , I reopen this PR per your suggestion, thanks. |
|
Wow, thank you for reopening. LOL @mpjlu |
| @@ -589,6 +602,9 @@ class ALS(@Since("1.4.0") override val uid: String) extends Estimator[ALSModel] | |||
| @Since("2.2.0") | |||
| def setColdStartStrategy(value: String): this.type = set(coldStartStrategy, value) | |||
|
|
|||
| @Since("2.3.0") | |||
| def setThreshold(value: Int): this.type = set(threshold, value) | |||
There was a problem hiding this comment.
I'm not sure callers can meaningfully understand and set this. Can't we pick a threshold programmatically?
There was a problem hiding this comment.
Yes, my test result is better than the previous result, especially for native BLAS. I will update my test results here soon, and I will change this set.
| @@ -1043,6 +1043,9 @@ object MimaExcludes { | |||
| // [SPARK-21680][ML][MLLIB]optimzie Vector coompress | |||
| ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.mllib.linalg.Vector.toSparseWithSize"), | |||
| ProblemFilters.exclude[ReversedMissingMethodProblem]("org.apache.spark.ml.linalg.Vector.toSparseWithSize") | |||
| ) ++ Seq( | |||
| // [SPARK-6685][ML]Use DSYRK to compute AtA in ALS | |||
| ProblemFilters.exclude[DirectMissingMethodProblem]("org.apache.spark.ml.recommendation.ALS.train") | |||
There was a problem hiding this comment.
This breaks an API unnecessarily, even though it's a dev API. I think we instead need to remove this as a user-facing param and avoid it altogether.
There was a problem hiding this comment.
I am ok to remove this, and use a loose threshold (e.g. 100), which is helpful for most cases. How about it?
|
Test build #82905 has finished for PR 19536 at commit
|
|
This is the performance data of this PR. |
|
cc @srowen @mengxr @yanboliang |
|
Because I don't have the environment to continue this work, I will close it. Thanks. |
What changes were proposed in this pull request?
This is a reopen of PR: #13891 with mima fix and code optimization.
There is about 30-40% performance improvement comparing with #13891
How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Please review http://spark.apache.org/contributing.html before opening a pull request.