Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-12811] [ML] Estimator for Generalized Linear Models(GLMs) #11136

Closed
wants to merge 17 commits into from
Closed

[SPARK-12811] [ML] Estimator for Generalized Linear Models(GLMs) #11136

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
a37e285
Initial version of Generalized Linear Regression
yanboliang Feb 9, 2016
3082686
fix setParent
yanboliang Feb 9, 2016
378ad6c
clear doc of numFeatures <= 4096, and fix default link
yanboliang Feb 11, 2016
97c3f6a
fix validateParams
yanboliang Feb 11, 2016
cc10147
add constriction to mu for family
yanboliang Feb 14, 2016
4a27970
Add clean for Gaussian, Poisson and Gamma
yanboliang Feb 15, 2016
cc90cb7
Do some refactor of GLM
yanboliang Feb 23, 2016
40bfd09
Better documents
yanboliang Feb 23, 2016
10bef71
Add test with regularization against glmnet.
yanboliang Feb 24, 2016
71e12f1
update doc
yanboliang Feb 24, 2016
8c84052
rename clean() to project()
yanboliang Feb 24, 2016
aa89fdc
Fix typos
yanboliang Feb 24, 2016
2ebcef7
Address comments
yanboliang Feb 25, 2016
c05a948
Rename MaxNumFeatures to MAX_NUM_FEATURES
yanboliang Feb 26, 2016
314b562
Fix test issues
yanboliang Mar 1, 2016
31a912c
Better error message
yanboliang Mar 1, 2016
007a4ec
Use .rdd.map instead of .map
yanboliang Mar 1, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 8 additions & 2 deletions mllib/src/main/scala/org/apache/spark/ml/optim/WeightedLeastSquares.scala
View file
Open in desktop
Expand Up @@ -156,6 +156,12 @@ private[ml] class WeightedLeastSquares(

private[ml] object WeightedLeastSquares {

/**
* In order to take the normal equation approach efficiently, [[WeightedLeastSquares]]
* only supports the number of features is no more than 4096.
*/
val MAX_NUM_FEATURES: Int = 4096

/**
* Aggregator to provide necessary summary statistics for solving [[WeightedLeastSquares]].
*/
Expand All @@ -174,8 +180,8 @@ private[ml] object WeightedLeastSquares {
private var aaSum: DenseVector = _

private def init(k: Int): Unit = {
require(k <= 4096, "In order to take the normal equation approach efficiently, " +
s"we set the max number of features to 4096 but got $k.")
require(k <= MAX_NUM_FEATURES, "In order to take the normal equation approach efficiently, " +
s"we set the max number of features to $MAX_NUM_FEATURES but got $k.")
this.k = k
triK = k * (k + 1) / 2
count = 0L
Expand Down