[SPARK-14549][ML] Copy the Vector and Matrix classes from mllib to ml in mllib-local

[dbtsai]

What changes were proposed in this pull request?

This task will copy the Vector and Matrix classes from mllib to ml package in mllib-local jar. The UDTs and since annotation in ml vector and matrix will be removed from now. UDTs will be achieved by #SPARK-14487, and since will be replaced by /* @ since 1.2.0 */

The BLAS implementation will be copied, and some of the test utilities will be copies as well.

Summary of changes:

1. In mllib-local/src/main/scala/org/apache/spark/ml/linalg/BLAS.scala
   • Copied from mllib/src/main/scala/org/apache/spark/mllib/linalg/BLAS.scala
   • logDebug("gemm: alpha is equal to 0 and beta is equal to 1. Returning C.") is removed in ml version.
2. In mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala
   • Copied from mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala
   • @Since was removed, and we'll use standard /* @Since /* Java doc. Will be in another PR.
   • UDT related code was removed, and will use SPARK-13944 [SPARK-14487][SQL] User Defined Type registration without SQLUserDefinedType annotation #12259 to replace the annotation.
3. In mllib-local/src/main/scala/org/apache/spark/ml/linalg/Vectors.scala
   • Copied from mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala
   • @Since was removed.
   • UDT related code was removed.
   • In def parseNumeric, it was throwing throw new SparkException(s"Cannot parse $other."), and now it's throwing throw new IllegalArgumentException(s"Cannot parse $other.")
4. In mllib/src/main/scala/org/apache/spark/mllib/linalg/Vectors.scala
   • For consistency with ML version of vector, def parseNumeric is now throwing throw new IllegalArgumentException(s"Cannot parse $other.")
5. mllib/src/main/scala/org/apache/spark/mllib/util/NumericParser.scala is moved to mllib-local/src/main/scala/org/apache/spark/ml/util/NumericParser.scala
   • All the throw new SparkException were replaced by throw new IllegalArgumentException

How was this patch tested?

unit tests

[SparkQA]

Test build #55571 has finished for PR 12317 at commit 5795d54.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • class DenseMatrix (
  • class SparseMatrix (
  • class DenseVector (val values: Array[Double]) extends Vector
  • class SparseVector (

[SparkQA]

Test build #55574 has finished for PR 12317 at commit 1fba791.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #55589 has finished for PR 12317 at commit f9473f5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

@dbtsai Please remove WIP from title when this is ready for review:)

[mengxr]

Core also depends on json4s (See https://github.com/apache/spark/blob/master/core/pom.xml#L195.). Shall we move the version definition to a common place?

[dbtsai]

I have another JIRA to do so. We also want to consolidate the version of breeze. https://issues.apache.org/jira/browse/SPARK-14612

If you want, I can do it in this PR.

[SparkQA]

Test build #55752 has finished for PR 12317 at commit 69f6982.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #55755 has finished for PR 12317 at commit 38914b2.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #55760 has finished for PR 12317 at commit 45a0ee8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #55766 has finished for PR 12317 at commit 150a60b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #55827 has finished for PR 12317 at commit 35e2da5.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dbtsai]

@mengxr The versions of the dependencies have been moved to the parent POM.

[dbtsai]

+cc @srowen @MLnick

[mengxr]

I'm making another pass.

[mengxr]

I created https://issues.apache.org/jira/browse/SPARK-14653 as a follow-up item.

[SparkQA]

Test build #55902 has finished for PR 12317 at commit e626d1d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

LGTM. Merged into master. Thanks!