[SPARK-29751][ML] Scalers use Summarizer instead of MultivariateOnlineSummarizer

[zhengruifeng]

What changes were proposed in this pull request?

use ml.Summarizer instead of mllib.MultivariateOnlineSummarizer

Why are the changes needed?

1, I found that using ml.Summarizer is faster than current impl;
2, mllib.MultivariateOnlineSummarizer maintain all arrays, while ml.Summarizer only maintain necessary arrays
3, using ml.Summarizer will avoid vector conversions to mlllib.Vector

Does this PR introduce any user-facing change?

No

How was this patch tested?

existing testsuites

[zhengruifeng]

test code

import org.apache.spark.ml.feature._

scala> var df = spark.read.format("libsvm").load("/data1/Datasets/a9a/a9a")
19/11/05 13:47:02 WARN LibSVMFileFormat: 'numFeatures' option not specified, determining the number of features by going though the input. If you know the number in advance, please specify it via 'numFeatures' option to avoid the extra scan.
df: org.apache.spark.sql.DataFrame = [label: double, features: vector]          

scala> df.persist()
res0: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [label: double, features: vector]


scala> df.count
res1: Long = 32561

scala> (0 until 8).foreach(_ => df = df.union(df))

scala> df.count
res3: Long = 8335616 

val durations1 = (0 until 50).map{i => val tic = System.currentTimeMillis; val scaler = new MaxAbsScaler().setInputCol("features"); val model = scaler.fit(df); val toc = System.currentTimeMillis; toc - tic}

durations1.takeRight(30).sum.toDouble / 30



val durations2 = (0 until 50).map{i => val tic = System.currentTimeMillis; val scaler = new MinMaxScaler().setInputCol("features"); val model = scaler.fit(df); val toc = System.currentTimeMillis; toc - tic}

durations2.takeRight(30).sum.toDouble / 30



val durations3 = (0 until 50).map{i => val tic = System.currentTimeMillis; val scaler = new StandardScaler().setInputCol("features"); val model = scaler.fit(df); val toc = System.currentTimeMillis; toc - tic}

durations3.takeRight(30).sum.toDouble / 30

Results: （the last 30 fitting are taken into account）

MaxAbsScaler(Old)	MinMaxScaler(Old)	StandardScaler(Old)	MaxAbsScaler(New)	MinMaxScaler(New)	StandardScaler(New)
6768.1	6875.2	6899.9	5862.1	5880.3	5889.7

[zhengruifeng]

IIRC, when Summarizer was created, I had done similar tests, and at that time Summarizer was much slower than MultivariateOnlineSummarizer.
I am surprised by the results and will look into whether it works in other places.

[zhengruifeng]

friendly ping @srowen @WeichenXu123

[SparkQA]

Test build #113244 has finished for PR 26393 at commit a2955d8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Also looks OK as a refactoring and optimization - if it's faster.

[srowen]

Is Statistics.colStats still used after this? just wondering if it goes away

[zhengruifeng]

After this PR, Statistics.colStats is no longer directly used in the .ml side.
It is still used in mllib.PCA, which is the impl of ml.PCA

[zhengruifeng]

Thanks @srowen for reviewing!