[SPARK-31659][ML][DOCS] Add VarianceThresholdSelector examples and doc

### What changes were proposed in this pull request? Add VarianceThresholdSelector examples and doc ### Why are the changes needed? VarianceThresholdSelector is a new feature selector in 3.1.0. We need to add examples and doc ### Does this PR introduce _any_ user-facing change? Yes. add Scala, Python and Java examples for VarianceThresholdSelector. Also add doc <img width="860" alt="Screen Shot 2020-05-07 at 9 20 01 AM" src="https://user-images.githubusercontent.com/13592258/81321791-e3f84d80-9047-11ea-837b-e39c193bd437.png"> <img width="860" alt="Screen Shot 2020-05-07 at 9 20 44 AM" src="https://user-images.githubusercontent.com/13592258/81321806-e8246b00-9047-11ea-8f35-206e330a92ab.png"> <img width="860" alt="Screen Shot 2020-05-07 at 9 21 27 AM" src="https://user-images.githubusercontent.com/13592258/81321822-ea86c500-9047-11ea-8743-99adec7f502b.png"> <img width="860" alt="Screen Shot 2020-05-07 at 9 21 43 AM" src="https://user-images.githubusercontent.com/13592258/81321826-ec508880-9047-11ea-9e7a-22ee5e13f495.png"> ### How was this patch tested? Manually checked Closes #28478 from huaxingao/variance_doc. Authored-by: Huaxin Gao <huaxing@us.ibm.com> Signed-off-by: zhengruifeng <ruifengz@foxmail.com>
apache · May 8, 2020 · 08335b6 · 08335b6
1 parent 97332f2
commit 08335b6
Show file tree

Hide file tree

Showing 4 changed files with 267 additions and 0 deletions.
diff --git a/docs/ml-features.md b/docs/ml-features.md
@@ -1793,6 +1793,70 @@ for more details on the API.
 </div>
 </div>
 
+## VarianceThresholdSelector
+
+`VarianceThresholdSelector` is a selector that removes low-variance features. Features with a
+ variance not greater than the `varianceThreshold` will be removed. If not set, `varianceThreshold`
+ defaults to 0, which means only features with variance 0 (i.e. features that have the same value in all samples)
+ will be removed.
+
+**Examples**
+
+Assume that we have a DataFrame with the columns `id` and `features`, which is used as
+our target to be predicted:
+
+~~~
+id | features
+---|--------------------------------
+ 1 | [6.0, 7.0, 0.0, 7.0, 6.0, 0.0]
+ 2 | [0.0, 9.0, 6.0, 0.0, 5.0, 9.0]
+ 3 | [0.0, 9.0, 3.0, 0.0, 5.0, 5.0]
+ 4 | [0.0, 9.0, 8.0, 5.0, 6.0, 4.0]
+ 5 | [8.0, 9.0, 6.0, 5.0, 4.0, 4.0]
+ 6 | [8.0, 9.0, 6.0, 0.0, 0.0, 0.0]
+~~~
+
+The variance for the 6 features are 16.67, 0.67, 8.17, 10.17,
+5.07, and 11.47 respectively. If we use `VarianceThresholdSelector` with
+`varianceThreshold = 8.0`, then the features with variance <= 8.0 are removed:
+
+~~~
+id | features                       | selectedFeatures
+---|--------------------------------|-------------------
+ 1 | [6.0, 7.0, 0.0, 7.0, 6.0, 0.0] | [6.0,0.0,7.0,0.0]
+ 2 | [0.0, 9.0, 6.0, 0.0, 5.0, 9.0] | [0.0,6.0,0.0,9.0]
+ 3 | [0.0, 9.0, 3.0, 0.0, 5.0, 5.0] | [0.0,3.0,0.0,5.0]
+ 4 | [0.0, 9.0, 8.0, 5.0, 6.0, 4.0] | [0.0,8.0,5.0,4.0]
+ 5 | [8.0, 9.0, 6.0, 5.0, 4.0, 4.0] | [8.0,6.0,5.0,4.0]
+ 6 | [8.0, 9.0, 6.0, 0.0, 0.0, 0.0] | [8.0,6.0,0.0,0.0]
+~~~
+
+<div class="codetabs">
+<div data-lang="scala" markdown="1">
+
+Refer to the [VarianceThresholdSelector Scala docs]((api/python/pyspark.ml.html#pyspark.ml.feature.ChiSqSelector))
+for more details on the API.
+
+{% include_example scala/org/apache/spark/examples/ml/VarianceThresholdSelectorExample.scala %}
+</div>
+
+<div data-lang="java" markdown="1">
+
+Refer to the [VarianceThresholdSelector Java docs](api/java/org/apache/spark/ml/feature/VarianceThresholdSelector.html)
+for more details on the API.
+
+{% include_example java/org/apache/spark/examples/ml/JavaVarianceThresholdSelectorExample.java %}
+</div>
+
+<div data-lang="python" markdown="1">
+
+Refer to the [VarianceThresholdSelector Python docs](api/python/pyspark.ml.html#pyspark.ml.feature.VarianceThresholdSelector)
+for more details on the API.
+
+{% include_example python/ml/variance_threshold_selector_example.py %}
+</div>
+</div>
+
 # Locality Sensitive Hashing
 [Locality Sensitive Hashing (LSH)](https://en.wikipedia.org/wiki/Locality-sensitive_hashing) is an important class of hashing techniques, which is commonly used in clustering, approximate nearest neighbor search and outlier detection with large datasets.
 

diff --git a/...ples/src/main/java/org/apache/spark/examples/ml/JavaVarianceThresholdSelectorExample.java b/...ples/src/main/java/org/apache/spark/examples/ml/JavaVarianceThresholdSelectorExample.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.examples.ml;
+
+import org.apache.spark.sql.Dataset;
+import org.apache.spark.sql.SparkSession;
+
+// $example on$
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.spark.ml.feature.VarianceThresholdSelector;
+import org.apache.spark.ml.linalg.VectorUDT;
+import org.apache.spark.ml.linalg.Vectors;
+import org.apache.spark.sql.Row;
+import org.apache.spark.sql.RowFactory;
+import org.apache.spark.sql.types.*;
+// $example off$
+
+/**
+ * An example for VarianceThresholdSelector.
+ * Run with
+ * <pre>
+ * bin/run-example ml.JavaVarianceThresholdSelectorExample
+ * </pre>
+ */
+public class JavaVarianceThresholdSelectorExample {
+  public static void main(String[] args) {
+    SparkSession spark = SparkSession
+      .builder()
+      .appName("JavaVarianceThresholdSelectorExample")
+      .getOrCreate();
+
+    // $example on$
+    List<Row> data = Arrays.asList(
+      RowFactory.create(1, Vectors.dense(6.0, 7.0, 0.0, 7.0, 6.0, 0.0)),
+      RowFactory.create(2, Vectors.dense(0.0, 9.0, 6.0, 0.0, 5.0, 9.0)),
+      RowFactory.create(3, Vectors.dense(0.0, 9.0, 3.0, 0.0, 5.0, 5.0)),
+      RowFactory.create(4, Vectors.dense(0.0, 9.0, 8.0, 5.0, 6.0, 4.0)),
+      RowFactory.create(5, Vectors.dense(8.0, 9.0, 6.0, 5.0, 4.0, 4.0)),
+      RowFactory.create(6, Vectors.dense(8.0, 9.0, 6.0, 0.0, 0.0, 0.0))
+    );
+    StructType schema = new StructType(new StructField[]{
+      new StructField("id", DataTypes.IntegerType, false, Metadata.empty()),
+      new StructField("features", new VectorUDT(), false, Metadata.empty())
+    });
+
+    Dataset<Row> df = spark.createDataFrame(data, schema);
+
+    VarianceThresholdSelector selector = new VarianceThresholdSelector()
+      .setVarianceThreshold(8.0)
+      .setFeaturesCol("features")
+      .setOutputCol("selectedFeatures");
+
+    Dataset<Row> result = selector.fit(df).transform(df);
+
+    System.out.println("Output: Features with variance lower than "
+        + selector.getVarianceThreshold() + " are removed.");
+    result.show();
+
+    // $example off$
+    spark.stop();
+  }
+}
diff --git a/examples/src/main/python/ml/variance_threshold_selector_example.py b/examples/src/main/python/ml/variance_threshold_selector_example.py
@@ -0,0 +1,55 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""
+An example for VarianceThresholdSelector.
+Run with:
+  bin/spark-submit examples/src/main/python/ml/variance_threshold_selector_example.py
+"""
+from __future__ import print_function
+
+from pyspark.sql import SparkSession
+# $example on$
+from pyspark.ml.feature import VarianceThresholdSelector
+from pyspark.ml.linalg import Vectors
+# $example off$
+
+if __name__ == "__main__":
+    spark = SparkSession\
+        .builder\
+        .appName("VarianceThresholdSelectorExample")\
+        .getOrCreate()
+
+    # $example on$
+    df = spark.createDataFrame([
+        (1, Vectors.dense([6.0, 7.0, 0.0, 7.0, 6.0, 0.0])),
+        (2, Vectors.dense([0.0, 9.0, 6.0, 0.0, 5.0, 9.0])),
+        (3, Vectors.dense([0.0, 9.0, 3.0, 0.0, 5.0, 5.0])),
+        (4, Vectors.dense([0.0, 9.0, 8.0, 5.0, 6.0, 4.0])),
+        (5, Vectors.dense([8.0, 9.0, 6.0, 5.0, 4.0, 4.0])),
+        (6, Vectors.dense([8.0, 9.0, 6.0, 0.0, 0.0, 0.0]))], ["id", "features"])
+
+    selector = VarianceThresholdSelector(varianceThreshold=8.0, outputCol="selectedFeatures")
+
+    result = selector.fit(df).transform(df)
+
+    print("Output: Features with variance lower than %f are removed." %
+          selector.getVarianceThreshold())
+    result.show()
+    # $example off$
+
+    spark.stop()
diff --git a/examples/src/main/scala/org/apache/spark/examples/ml/VarianceThresholdSelectorExample.scala b/examples/src/main/scala/org/apache/spark/examples/ml/VarianceThresholdSelectorExample.scala
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// scalastyle:off println
+package org.apache.spark.examples.ml
+
+// $example on$
+import org.apache.spark.ml.feature.VarianceThresholdSelector
+import org.apache.spark.ml.linalg.Vectors
+// $example off$
+import org.apache.spark.sql.SparkSession
+
+/**
+ * An example for VarianceThresholdSelector.
+ * Run with
+ * {{{
+ * bin/run-example ml.VarianceThresholdSelectorExample
+ * }}}
+ */
+object VarianceThresholdSelectorExample {
+  def main(args: Array[String]): Unit = {
+    val spark = SparkSession
+      .builder
+      .appName("VarianceThresholdSelectorExample")
+      .getOrCreate()
+    import spark.implicits._
+
+    // $example on$
+    val data = Seq(
+      (1, Vectors.dense(6.0, 7.0, 0.0, 7.0, 6.0, 0.0)),
+      (2, Vectors.dense(0.0, 9.0, 6.0, 0.0, 5.0, 9.0)),
+      (3, Vectors.dense(0.0, 9.0, 3.0, 0.0, 5.0, 5.0)),
+      (4, Vectors.dense(0.0, 9.0, 8.0, 5.0, 6.0, 4.0)),
+      (5, Vectors.dense(8.0, 9.0, 6.0, 5.0, 4.0, 4.0)),
+      (6, Vectors.dense(8.0, 9.0, 6.0, 0.0, 0.0, 0.0))
+    )
+
+    val df = spark.createDataset(data).toDF("id", "features")
+
+    val selector = new VarianceThresholdSelector()
+      .setVarianceThreshold(8.0)
+      .setFeaturesCol("features")
+      .setOutputCol("selectedFeatures")
+
+    val result = selector.fit(df).transform(df)
+
+    println(s"Output: Features with variance lower than" +
+      s" ${selector.getVarianceThreshold} are removed.")
+    result.show()
+    // $example off$
+
+    spark.stop()
+  }
+}
+// scalastyle:on println