[SPARK-9671] [MLLIB] re-org user guide and add migration guide

[mengxr]

This PR updates the MLlib user guide and adds migration guide for 1.4->1.5.

• merge migration guide for spark.mllib and spark.ml packages
• remove dependency section from spark.ml guide
• move the paragraph about spark.mllib and spark.ml to the top and recommend spark.ml
• move Sam's talk to footnote to make the section focus on dependencies

Minor changes to code examples and other wording will be in a separate PR.

@jkbradley @srowen @feynmanliang

[mengxr]

No content change here. Just moved the paragraphs from mllib-guide and ml-guide.

[SparkQA]

Test build #41734 has finished for PR 8498 at commit 2790270.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • * *(Breaking change)* Theapplyandcopymethods for the case class [BoostingStrategy](api/scala/index.html#org.apache.spark.mllib.tree.configuration.BoostingStrategy) have been changed because of a modification to the case class fields. This could be an issue for users who useBoostingStrategyto set GBT parameters.
  • * *(Breaking change)* The return value of [LDA.run](api/scala/index.html#org.apache.spark.mllib.clustering.LDA) has changed. It now returns an abstract classLDAModelinstead of the concrete classDistributedLDAModel. The object of typeLDAModelcan still be cast to the appropriate concrete type, which depends on the optimization algorithm.
  • * ThescoreColoutput column (with default value "score") was renamed to beprobabilityCol(with default value "probability"). The type was originallyDouble(for the probability of class 1.0), but it is nowVector(for the probability of each class, to support multiclass classification in the future).

[feynmanliang]

I know it is already mentioned in mllib-guide but should we mention DataFrames here in ml-guide's introduction since it is the key distinction in comparison to spark.ml?

[mengxr]

okay

[SparkQA]

Test build #41757 has finished for PR 8498 at commit f8efdcc.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • * *(Breaking change)* Theapplyandcopymethods for the case class [BoostingStrategy](api/scala/index.html#org.apache.spark.mllib.tree.configuration.BoostingStrategy) have been changed because of a modification to the case class fields. This could be an issue for users who useBoostingStrategyto set GBT parameters.
  • * *(Breaking change)* The return value of [LDA.run](api/scala/index.html#org.apache.spark.mllib.clustering.LDA) has changed. It now returns an abstract classLDAModelinstead of the concrete classDistributedLDAModel. The object of typeLDAModelcan still be cast to the appropriate concrete type, which depends on the optimization algorithm.
  • * ThescoreColoutput column (with default value "score") was renamed to beprobabilityCol(with default value "probability"). The type was originallyDouble(for the probability of class 1.0), but it is nowVector(for the probability of each class, to support multiclass classification in the future).

[feynmanliang]

LGTM, the ml-ann.md filename is inconsistent with all referencing text (which refers to it as MLP) but that's unrelated to this PR

[mengxr]

Merged into master and branch-1.5.