From 736ccbcb60ee0cf22481bbff2d7cb63d563f783b Mon Sep 17 00:00:00 2001 From: Feynman Liang Date: Mon, 6 Jul 2015 13:47:02 -0700 Subject: [PATCH 1/3] NGram feature transformer documentation --- docs/ml-features.md | 76 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) diff --git a/docs/ml-features.md b/docs/ml-features.md index f88c0248c1a8a..4759cf2fc2e89 100644 --- a/docs/ml-features.md +++ b/docs/ml-features.md @@ -288,6 +288,82 @@ for words_label in wordsDataFrame.select("words", "label").take(3): +## $n$-gram + +An [n-gram](https://en.wikipedia.org/wiki/N-gram) is a sequence of $n$ tokens (typically words) for some integer $n$. The [NGram](api/scala/index.html#org.apache.spark.ml.feature.NGram) class can be used to transform input features into $n$-grams. + +`NGram` takes as input a sequence of strings (e.g. the output of a [Tokenizer](api/scala/index.html#org.apache.spark.ml.feature.Tokenizer)). The parameter `n` is used to determine the number of terms in each $n$-gram. The output will consist of a sequence of $n$-grams where each $n$-gram is represented by a space-delimited string of $n$ consecutive words. If the input sequence contains less than `n` strings, no output is produced. + +
+
+{% highlight scala %} +import org.apache.spark.ml.feature.NGram + +val wordDataFrame = sqlContext.createDataFrame(Seq( + (0, Array("Hi", "I", "heard", "about", "Spark")), + (1, Array("I", "wish", "Java", "could", "use", "case", "classes")), + (2, Array("Logistic", "regression", "models", "are", "neat")) +)).toDF("label", "words") + +val ngram = new NGram().setInputCol("words").setOutputCol("ngrams") +val ngramDataFrame = ngram.transform(wordDataFrame) +ngramDataFrame.select("ngrams", "label").take(3).foreach(println) +{% endhighlight %} +
+ +
+{% highlight java %} +import com.google.common.collect.Lists; + +import org.apache.spark.api.java.JavaRDD; +import org.apache.spark.ml.feature.Tokenizer; +import org.apache.spark.mllib.linalg.Vector; +import org.apache.spark.sql.DataFrame; +import org.apache.spark.sql.Row; +import org.apache.spark.sql.RowFactory; +import org.apache.spark.sql.types.DataTypes; +import org.apache.spark.sql.types.Metadata; +import org.apache.spark.sql.types.StructField; +import org.apache.spark.sql.types.StructType; + +JavaRDD jrdd = jsc.parallelize(Lists.newArrayList( + RowFactory.create(0D, Lists.newArrayList("Hi", "I", "heard", "about", "Spark")), + RowFactory.create(1D, Lists.newArrayList("I", "wish", "Java", "could", "use", "case", "classes")), + RowFactory.create(2D, Lists.newArrayList("Logistic", "regression", "models", "are", "neat")) +)); +StructType schema = new StructType(new StructField[]{ + new StructField("label", DataTypes.DoubleType, false, Metadata.empty()), + new StructField("words", DataTypes.createArrayType(DataTypes.StringType), false, Metadata.empty()) +}); +DataFrame wordDataFrame = sqlContext.createDataFrame(jrdd, schema); +NGram ngramTransformer = new NGram().setInputCol("words").setOutputCol("ngrams"); +DataFrame ngramDataFrame = ngramTransformer.transform(wordDataFrame); +for (Row r : ngramDataFrame.select("ngrams", "label").take(3)) { + java.util.List ngrams = r.getList(0); + for (String ngram : ngrams) System.out.print(ngram + " --- "); + System.out.println(); +} +{% endhighlight %} +
+ +
+{% highlight python %} +from pyspark.ml.feature import NGram + +wordDataFrame = sqlContext.createDataFrame([ + (0, ["Hi", "I", "heard", "about", "Spark"]), + (1, ["I", "wish", "Java", "could", "use", "case", "classes"]), + (2, ["Logistic", "regression", "models", "are", "neat"]) +], ["label", "words"]) +ngram = NGram(inputCol="words", outputCol="ngrams") +ngramDataFrame = ngram.transform(wordDataFrame) +for ngrams_label in ngramDataFrame.select("ngrams", "label").take(3): + print(ngrams_label) +{% endhighlight %} +
+
+ + ## Binarizer Binarization is the process of thresholding numerical features to binary features. As some probabilistic estimators make assumption that the input data is distributed according to [Bernoulli distribution](http://en.wikipedia.org/wiki/Bernoulli_distribution), a binarizer is useful for pre-processing the input data with continuous numerical features. From 60d5ac0e5128af62b1d6c0444636d85959f96c50 Mon Sep 17 00:00:00 2001 From: Feynman Liang Date: Tue, 7 Jul 2015 18:29:36 -0700 Subject: [PATCH 2/3] Inline API doc and fix indentation --- docs/ml-features.md | 14 ++++++++++---- 1 file changed, 10 insertions(+), 4 deletions(-) diff --git a/docs/ml-features.md b/docs/ml-features.md index 4759cf2fc2e89..026e3061159a9 100644 --- a/docs/ml-features.md +++ b/docs/ml-features.md @@ -290,12 +290,18 @@ for words_label in wordsDataFrame.select("words", "label").take(3): ## $n$-gram -An [n-gram](https://en.wikipedia.org/wiki/N-gram) is a sequence of $n$ tokens (typically words) for some integer $n$. The [NGram](api/scala/index.html#org.apache.spark.ml.feature.NGram) class can be used to transform input features into $n$-grams. +An [n-gram](https://en.wikipedia.org/wiki/N-gram) is a sequence of $n$ tokens (typically words) for some integer $n$. The `NGram` class can be used to transform input features into $n$-grams. -`NGram` takes as input a sequence of strings (e.g. the output of a [Tokenizer](api/scala/index.html#org.apache.spark.ml.feature.Tokenizer)). The parameter `n` is used to determine the number of terms in each $n$-gram. The output will consist of a sequence of $n$-grams where each $n$-gram is represented by a space-delimited string of $n$ consecutive words. If the input sequence contains less than `n` strings, no output is produced. +`NGram` takes as input a sequence of strings (e.g. the output of a [Tokenizer](api/scala/index.html#org.apache.spark.ml.feature.Tokenizer)). The parameter `n` is used to determine the number of terms in each $n$-gram. The output will consist of a sequence of $n$-grams where each $n$-gram is represented by a space-delimited string of $n$ consecutive words. If the input sequence contains fewer than `n` strings, no output is produced.
+
+ +
+ +[`NGram`](api/scala/index.html#org.apache.spark.ml.feature.NGram) takes an input column name, an output column name, and an optional length parameter n (n=2 by default). + {% highlight scala %} import org.apache.spark.ml.feature.NGram @@ -316,7 +322,7 @@ ngramDataFrame.select("ngrams", "label").take(3).foreach(println) import com.google.common.collect.Lists; import org.apache.spark.api.java.JavaRDD; -import org.apache.spark.ml.feature.Tokenizer; +import org.apache.spark.ml.feature.NGram; import org.apache.spark.mllib.linalg.Vector; import org.apache.spark.sql.DataFrame; import org.apache.spark.sql.Row; @@ -341,7 +347,7 @@ DataFrame ngramDataFrame = ngramTransformer.transform(wordDataFrame); for (Row r : ngramDataFrame.select("ngrams", "label").take(3)) { java.util.List ngrams = r.getList(0); for (String ngram : ngrams) System.out.print(ngram + " --- "); - System.out.println(); + System.out.println(); } {% endhighlight %}
From 5aface95a9b2c7c59ffec4813e037bb9a826f021 Mon Sep 17 00:00:00 2001 From: Feynman Liang Date: Wed, 8 Jul 2015 13:34:58 -0700 Subject: [PATCH 3/3] Pretty print Scala output and add API doc to each codetab --- docs/ml-features.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/ml-features.md b/docs/ml-features.md index 026e3061159a9..54068debe2159 100644 --- a/docs/ml-features.md +++ b/docs/ml-features.md @@ -292,7 +292,7 @@ for words_label in wordsDataFrame.select("words", "label").take(3): An [n-gram](https://en.wikipedia.org/wiki/N-gram) is a sequence of $n$ tokens (typically words) for some integer $n$. The `NGram` class can be used to transform input features into $n$-grams. -`NGram` takes as input a sequence of strings (e.g. the output of a [Tokenizer](api/scala/index.html#org.apache.spark.ml.feature.Tokenizer)). The parameter `n` is used to determine the number of terms in each $n$-gram. The output will consist of a sequence of $n$-grams where each $n$-gram is represented by a space-delimited string of $n$ consecutive words. If the input sequence contains fewer than `n` strings, no output is produced. +`NGram` takes as input a sequence of strings (e.g. the output of a [Tokenizer](ml-features.html#tokenizer). The parameter `n` is used to determine the number of terms in each $n$-gram. The output will consist of a sequence of $n$-grams where each $n$-gram is represented by a space-delimited string of $n$ consecutive words. If the input sequence contains fewer than `n` strings, no output is produced.
@@ -313,11 +313,14 @@ val wordDataFrame = sqlContext.createDataFrame(Seq( val ngram = new NGram().setInputCol("words").setOutputCol("ngrams") val ngramDataFrame = ngram.transform(wordDataFrame) -ngramDataFrame.select("ngrams", "label").take(3).foreach(println) +ngramDataFrame.take(3).map(_.getAs[Stream[String]]("ngrams").toList).foreach(println) {% endhighlight %}
+ +[`NGram`](api/java/org/apache/spark/ml/feature/NGram.html) takes an input column name, an output column name, and an optional length parameter n (n=2 by default). + {% highlight java %} import com.google.common.collect.Lists; @@ -353,6 +356,9 @@ for (Row r : ngramDataFrame.select("ngrams", "label").take(3)) {
+ +[`NGram`](api/python/pyspark.ml.html#pyspark.ml.feature.NGram) takes an input column name, an output column name, and an optional length parameter n (n=2 by default). + {% highlight python %} from pyspark.ml.feature import NGram