[SPARK-7388][SPARK-7383] wrapper for VectorAssembler in Python #5930

brkyvz · 2015-05-06T02:58:39Z

The wrapper required the implementation of the ArrayParam, because Array[T] is hard to obtain from Python. ArrayParam has an extra function called wCast which is an internal function to obtain Array[T] from Seq[T]

AmplabJenkins · 2015-05-06T03:02:09Z

Merged build triggered.

AmplabJenkins · 2015-05-06T03:02:17Z

Merged build started.

SparkQA · 2015-05-06T03:02:51Z

Test build #31938 has started for PR 5930 at commit 7f7ea2a.

SparkQA · 2015-05-06T03:04:28Z

Test build #31938 has finished for PR 5930 at commit 7f7ea2a.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ArrayParam[T : ClassTag](parent: Params, name: String, doc: String, isValid: Array[T] => Boolean)
- class VectorAssembler(JavaTransformer, HasInputCols, HasOutputCol):
- class HasInputCols(Params):

AmplabJenkins · 2015-05-06T03:04:29Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-06T03:04:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31938/
Test FAILed.

AmplabJenkins · 2015-05-06T03:12:10Z

Merged build triggered.

AmplabJenkins · 2015-05-06T03:12:17Z

Merged build started.

SparkQA · 2015-05-06T03:12:45Z

Test build #31943 has started for PR 5930 at commit 39ecb07.

AmplabJenkins · 2015-05-06T03:17:10Z

Merged build triggered.

AmplabJenkins · 2015-05-06T03:17:17Z

Merged build started.

SparkQA · 2015-05-06T03:17:40Z

Test build #31944 has started for PR 5930 at commit 99c2ebf.

SparkQA · 2015-05-06T04:27:04Z

Test build #31943 has finished for PR 5930 at commit 39ecb07.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class ArrayParam[T : ClassTag](
- class VectorAssembler(JavaTransformer, HasInputCols, HasOutputCol):
- class HasInputCols(Params):

AmplabJenkins · 2015-05-06T04:27:09Z

Merged build finished. Test FAILed.

AmplabJenkins · 2015-05-06T04:27:10Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31943/
Test FAILed.

SparkQA · 2015-05-06T04:59:37Z

Test build #31944 has finished for PR 5930 at commit 99c2ebf.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-05-06T04:59:42Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-06T04:59:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31944/
Test PASSed.

mengxr · 2015-05-06T06:21:35Z

mllib/src/main/scala/org/apache/spark/ml/param/params.scala

@@ -218,6 +219,22 @@ class BooleanParam(parent: Params, name: String, doc: String) // No need for isV
  override def w(value: Boolean): ParamPair[Boolean] = super.w(value)
 }

+/** Specialized version of [[Param[Array[T]]]] for Java. */
+class ArrayParam[T : ClassTag](


Having ClassTag is not Java friendly. Array[T] will be translated into Object in Java to handle both primitive arrays and object arrays. How about adding StringArrayParam instead of ArrayParam[T] in this PR?

AmplabJenkins · 2015-05-06T17:07:16Z

Merged build triggered.

AmplabJenkins · 2015-05-06T17:07:24Z

Merged build started.

SparkQA · 2015-05-06T17:08:01Z

Test build #31995 has started for PR 5930 at commit c81072d.

mengxr · 2015-05-06T18:47:37Z

mllib/src/main/scala/org/apache/spark/ml/param/params.scala

@@ -22,6 +22,7 @@ import java.util.NoSuchElementException

 import scala.annotation.varargs
 import scala.collection.mutable
+import scala.reflect.ClassTag


Remove this line.

SparkQA · 2015-05-06T18:51:56Z

Test build #31995 has finished for PR 5930 at commit c81072d.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StringArrayParam(parent: Params, name: String, doc: String, isValid: Array[String] => Boolean)
- class VectorAssembler(JavaTransformer, HasInputCols, HasOutputCol):
- class HasInputCols(Params):

AmplabJenkins · 2015-05-06T18:52:01Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-06T18:52:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31995/
Test PASSed.

mengxr · 2015-05-07T08:11:29Z

@brkyvz We can overload w and let py4j decide which w to call via reflection. I sent you a PR at brkyvz#3.

overload StringArrayParam.w

AmplabJenkins · 2015-05-07T15:32:13Z

Merged build triggered.

AmplabJenkins · 2015-05-07T15:32:21Z

Merged build started.

SparkQA · 2015-05-07T15:34:55Z

Test build #32118 has started for PR 5930 at commit 73e745f.

SparkQA · 2015-05-07T17:18:43Z

Test build #32118 has finished for PR 5930 at commit 73e745f.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- class StringArrayParam(parent: Params, name: String, doc: String, isValid: Array[String] => Boolean)
- class VectorAssembler(JavaTransformer, HasInputCols, HasOutputCol):
- class HasInputCols(Params):

AmplabJenkins · 2015-05-07T17:18:48Z

Merged build finished. Test PASSed.

AmplabJenkins · 2015-05-07T17:18:48Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/32118/
Test PASSed.

The wrapper required the implementation of the `ArrayParam`, because `Array[T]` is hard to obtain from Python. `ArrayParam` has an extra function called `wCast` which is an internal function to obtain `Array[T]` from `Seq[T]` Author: Burak Yavuz <brkyvz@gmail.com> Author: Xiangrui Meng <meng@databricks.com> Closes #5930 from brkyvz/ml-feat and squashes the following commits: 73e745f [Burak Yavuz] Merge pull request #3 from mengxr/SPARK-7388 c221db9 [Xiangrui Meng] overload StringArrayParam.w c81072d [Burak Yavuz] addressed comments 99c2ebf [Burak Yavuz] add to python_shared_params 39ecb07 [Burak Yavuz] fix scalastyle 7f7ea2a [Burak Yavuz] [SPARK-7388][SPARK-7383] wrapper for VectorAssembler in Python (cherry picked from commit 9e2ffb1) Signed-off-by: Xiangrui Meng <meng@databricks.com>

mengxr · 2015-05-07T17:26:40Z

LGTM. Merged into master and branch-1.4. Thanks! I left SPARK-7388 open.

The wrapper required the implementation of the `ArrayParam`, because `Array[T]` is hard to obtain from Python. `ArrayParam` has an extra function called `wCast` which is an internal function to obtain `Array[T]` from `Seq[T]` Author: Burak Yavuz <brkyvz@gmail.com> Author: Xiangrui Meng <meng@databricks.com> Closes apache#5930 from brkyvz/ml-feat and squashes the following commits: 73e745f [Burak Yavuz] Merge pull request apache#3 from mengxr/SPARK-7388 c221db9 [Xiangrui Meng] overload StringArrayParam.w c81072d [Burak Yavuz] addressed comments 99c2ebf [Burak Yavuz] add to python_shared_params 39ecb07 [Burak Yavuz] fix scalastyle 7f7ea2a [Burak Yavuz] [SPARK-7388][SPARK-7383] wrapper for VectorAssembler in Python

[SPARK-7388][SPARK-7383] wrapper for VectorAssembler in Python

7f7ea2a

fix scalastyle

39ecb07

add to python_shared_params

99c2ebf

mengxr reviewed May 6, 2015
View reviewed changes

addressed comments

c81072d

mengxr reviewed May 6, 2015
View reviewed changes

overload StringArrayParam.w

c221db9

Merge pull request #3 from mengxr/SPARK-7388

73e745f

overload StringArrayParam.w

asfgit closed this in 9e2ffb1 May 7, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-7388][SPARK-7383] wrapper for VectorAssembler in Python #5930

[SPARK-7388][SPARK-7383] wrapper for VectorAssembler in Python #5930

brkyvz commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

SparkQA commented May 6, 2015

SparkQA commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

SparkQA commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

SparkQA commented May 6, 2015

SparkQA commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

SparkQA commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

mengxr May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

SparkQA commented May 6, 2015

mengxr May 6, 2015

SparkQA commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

mengxr commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

mengxr commented May 7, 2015

[SPARK-7388][SPARK-7383] wrapper for VectorAssembler in Python #5930

[SPARK-7388][SPARK-7383] wrapper for VectorAssembler in Python #5930

Conversation

brkyvz commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

SparkQA commented May 6, 2015

SparkQA commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

SparkQA commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

SparkQA commented May 6, 2015

SparkQA commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

SparkQA commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

mengxr May 6, 2015

Choose a reason for hiding this comment

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

SparkQA commented May 6, 2015

mengxr May 6, 2015

Choose a reason for hiding this comment

SparkQA commented May 6, 2015

AmplabJenkins commented May 6, 2015

AmplabJenkins commented May 6, 2015

mengxr commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

SparkQA commented May 7, 2015

SparkQA commented May 7, 2015

AmplabJenkins commented May 7, 2015

AmplabJenkins commented May 7, 2015

mengxr commented May 7, 2015