[SPARK-6705][MLLIB] Add fit intercept api to ml logisticregression

[oefirouz]

I have the fit intercept enabled by default for logistic regression, I
wonder what others think here. I understand that it enables allocation
by default which is undesirable, but one needs to have a very strong
reason for not having an intercept term enabled so it is the safer
default from a statistical sense.

Explicitly modeling the intercept by adding a column of all 1s does not
work. I believe the reason is that since the API for
LogisticRegressionWithLBFGS forces column normalization, and a column of all
1s has 0 variance so dividing by 0 kills it.

[AmplabJenkins]

Can one of the admins verify this patch?

[jkbradley]

@oefirouz Thanks for the PR! I agree we need to support this option, and that it should be set to True by default.

Can you please make a JIRA and put it in the PR title? "[SPARK-####] [MLLIB] Add fit..."

I believe you're correct about normalization causing problems for an all-ones column; the other issue is that LR needs to know not to regularize the intercept term.

I'll look at the code now

[jkbradley]

"fits the intercept term or not" --> "Indicates whether to fit an intercept term"

[jkbradley]

It looks fine to me other than that one comment. However, could you please add a unit test (or modify an existing one) to test this feature in org/apache/spark/ml/classification/LogisticRegressionSuite.scala? Thanks!

[oefirouz]

@jkbradley Thanks for your comments! I've updated the PR.

[jkbradley]

Thinking more about this, can you please make this default to true? (Add "Some(true)" as a 4th argument, plus update doc.) Thanks!

[jkbradley]

@oefirouz Just a few more comments, and then that should be it. Thanks!

[srowen]

ok to test

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29707/
Test FAILed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29718/
Test PASSed.

[SparkQA]

Test build #636 has started for PR 5301 at commit 9f1286b.

[jkbradley]

LGTM once the full tests pass

[SparkQA]

Test build #636 has finished for PR 5301 at commit 9f1286b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.
• This patch does not change any dependencies.

[mengxr]

One comment before we merge this: Is it useful to specify the bias constant? In LIBLINEAR, there is an option called -B bias. If bias == 0, we don't add intercept, otherwise we add intercept with the provided bias. If we take this approach, the param name should be bias.

[jkbradley]

I'm going to say no; that sounds like an extra option we could add later. More details on that API:

• Our current API: Either fit the intercept, or fix it at 0. Do not regularize it if fitting.
• "bias" API in liblinear: Fit the intercept, but adjust regularization by adjusting "bias."
  • My issues with this:
    • It's not intuitive (if I understand their API correctly). To avoid regularizing the intercept, you set the "bias" to be large.
    • R glmnet uses our current API (fit intercept or don't), and it seems like a more authoritative codebase to follow.

Later on, I could imagine us adding the option to regularize the intercept.

[jkbradley]

I'll go ahead and merge this with master. Thanks @oefirouz !