[SPARK-1636][MLLIB] Move main methods to examples #584

mengxr · 2014-04-28T23:29:00Z

NaiveBayes -> SparseNaiveBayes
KMeans -> DenseKMeans
SVMWithSGD and LogisticRegerssionWithSGD -> BinaryClassification
ALS -> MovieLensALS
LinearRegressionWithSGD, LassoWithSGD, and RidgeRegressionWithSGD -> LinearRegression
DecisionTree -> DecisionTreeRunner

scopt is used for parsing command-line parameters. scopt has MIT license and it only depends on scala-library.

Example help message:

BinaryClassification: an example app for binary classification.
Usage: BinaryClassification [options] <input>

  --numIterations <value>
        number of iterations
  --stepSize <value>
        initial step size, default: 1.0
  --algorithm <value>
        algorithm (SVM,LR), default: LR
  --regType <value>
        regularization type (L1,L2), default: L2
  --regParam <value>
        regularization parameter, default: 0.1
  <input>
        input paths to labeled examples in LIBSVM format

…egression example

AmplabJenkins · 2014-04-28T23:32:57Z

Merged build triggered.

AmplabJenkins · 2014-04-28T23:33:05Z

Merged build started.

AmplabJenkins · 2014-04-29T00:24:00Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-04-29T00:24:00Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14545/

rxin · 2014-04-29T03:23:32Z

examples/src/main/scala/org/apache/spark/examples/mllib/BinaryClassification.scala

+ * }}}
+ * If you use it as a template to create your own app, please use `spark-submit` to submit your app.
+ */
+object BinaryClassification extends App {


I think ti is better to declare a main rather than extending App. It is more familiar to readers from the Java/C land.

rxin · 2014-04-29T03:34:34Z

LGTM. If you can avoid wild card imports for some packages, that'd be great.

AmplabJenkins · 2014-04-29T05:07:57Z

Merged build triggered.

AmplabJenkins · 2014-04-29T05:08:06Z

Merged build started.

mengxr · 2014-04-29T05:25:50Z

Made imports explicit except SparkContext and enums.

AmplabJenkins · 2014-04-29T05:27:57Z

Merged build triggered.

AmplabJenkins · 2014-04-29T05:28:06Z

Merged build started.

AmplabJenkins · 2014-04-29T05:47:57Z

Merged build triggered.

AmplabJenkins · 2014-04-29T05:48:07Z

Merged build started.

AmplabJenkins · 2014-04-29T06:38:57Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-04-29T06:38:57Z

Merged build finished. All automated tests passed.

AmplabJenkins · 2014-04-29T06:38:57Z

Merged build finished.

AmplabJenkins · 2014-04-29T06:38:58Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14554/

AmplabJenkins · 2014-04-29T06:38:58Z

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14555/

AmplabJenkins · 2014-04-29T06:38:58Z

Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/14553/

rxin · 2014-04-29T07:41:07Z

Thanks. Merged!

* `NaiveBayes` -> `SparseNaiveBayes` * `KMeans` -> `DenseKMeans` * `SVMWithSGD` and `LogisticRegerssionWithSGD` -> `BinaryClassification` * `ALS` -> `MovieLensALS` * `LinearRegressionWithSGD`, `LassoWithSGD`, and `RidgeRegressionWithSGD` -> `LinearRegression` * `DecisionTree` -> `DecisionTreeRunner` `scopt` is used for parsing command-line parameters. `scopt` has MIT license and it only depends on `scala-library`. Example help message: ~~~ BinaryClassification: an example app for binary classification. Usage: BinaryClassification [options] <input> --numIterations <value> number of iterations --stepSize <value> initial step size, default: 1.0 --algorithm <value> algorithm (SVM,LR), default: LR --regType <value> regularization type (L1,L2), default: L2 --regParam <value> regularization parameter, default: 0.1 <input> input paths to labeled examples in LIBSVM format ~~~ Author: Xiangrui Meng <meng@databricks.com> Closes #584 from mengxr/mllib-main and squashes the following commits: 7b58c60 [Xiangrui Meng] minor 6e35d7e [Xiangrui Meng] make imports explicit and fix code style c6178c9 [Xiangrui Meng] update TS PCA/SVD to use new spark-submit 6acff75 [Xiangrui Meng] use scopt for DecisionTreeRunner be86069 [Xiangrui Meng] use main instead of extending App b3edf68 [Xiangrui Meng] move DecisionTree's main method to examples 8bfaa5a [Xiangrui Meng] change NaiveBayesParams to Params fe23dcb [Xiangrui Meng] remove main from KMeans and add DenseKMeans as an example 67f4448 [Xiangrui Meng] remove main methods from linear regression algorithms and add LinearRegression example b066bbc [Xiangrui Meng] remove main from ALS and add MovieLensALS example b040f3b [Xiangrui Meng] change BinaryClassificationParams to Params 577945b [Xiangrui Meng] remove unused imports from NB 3d299bc [Xiangrui Meng] remove main from LR/SVM and add an example app for binary classification f70878e [Xiangrui Meng] remove main from NaiveBayes and add an example NaiveBayes app 01ec2cd [Xiangrui Meng] Merge branch 'master' into mllib-main 9420692 [Xiangrui Meng] add scopt to examples dependencies (cherry picked from commit 3f38334) Signed-off-by: Reynold Xin <rxin@apache.org>

* `NaiveBayes` -> `SparseNaiveBayes` * `KMeans` -> `DenseKMeans` * `SVMWithSGD` and `LogisticRegerssionWithSGD` -> `BinaryClassification` * `ALS` -> `MovieLensALS` * `LinearRegressionWithSGD`, `LassoWithSGD`, and `RidgeRegressionWithSGD` -> `LinearRegression` * `DecisionTree` -> `DecisionTreeRunner` `scopt` is used for parsing command-line parameters. `scopt` has MIT license and it only depends on `scala-library`. Example help message: ~~~ BinaryClassification: an example app for binary classification. Usage: BinaryClassification [options] <input> --numIterations <value> number of iterations --stepSize <value> initial step size, default: 1.0 --algorithm <value> algorithm (SVM,LR), default: LR --regType <value> regularization type (L1,L2), default: L2 --regParam <value> regularization parameter, default: 0.1 <input> input paths to labeled examples in LIBSVM format ~~~ Author: Xiangrui Meng <meng@databricks.com> Closes apache#584 from mengxr/mllib-main and squashes the following commits: 7b58c60 [Xiangrui Meng] minor 6e35d7e [Xiangrui Meng] make imports explicit and fix code style c6178c9 [Xiangrui Meng] update TS PCA/SVD to use new spark-submit 6acff75 [Xiangrui Meng] use scopt for DecisionTreeRunner be86069 [Xiangrui Meng] use main instead of extending App b3edf68 [Xiangrui Meng] move DecisionTree's main method to examples 8bfaa5a [Xiangrui Meng] change NaiveBayesParams to Params fe23dcb [Xiangrui Meng] remove main from KMeans and add DenseKMeans as an example 67f4448 [Xiangrui Meng] remove main methods from linear regression algorithms and add LinearRegression example b066bbc [Xiangrui Meng] remove main from ALS and add MovieLensALS example b040f3b [Xiangrui Meng] change BinaryClassificationParams to Params 577945b [Xiangrui Meng] remove unused imports from NB 3d299bc [Xiangrui Meng] remove main from LR/SVM and add an example app for binary classification f70878e [Xiangrui Meng] remove main from NaiveBayes and add an example NaiveBayes app 01ec2cd [Xiangrui Meng] Merge branch 'master' into mllib-main 9420692 [Xiangrui Meng] add scopt to examples dependencies

apache#522 got messed after i rewrote the branch hadoop_jar_name. So created a new one. Author: Bijay Bisht <bijay.bisht@gmail.com> Closes apache#584 from bijaybisht/hadoop_jar_name_on_0.9.0 and squashes the following commits: 1b6fb3c [Bijay Bisht] Ported hadoopClient jar for < 1.0.1 fix

Clean up the pre-built k8s binaries output folder which is generated in `install-k8s` role. Otherwise `kubetest` will fail due to some unknown conflicts.

mengxr added 11 commits April 28, 2014 10:21

add scopt to examples dependencies

9420692

Merge branch 'master' into mllib-main

01ec2cd

remove main from NaiveBayes and add an example NaiveBayes app

f70878e

remove main from LR/SVM and add an example app for binary classification

3d299bc

remove unused imports from NB

577945b

change BinaryClassificationParams to Params

b040f3b

remove main from ALS and add MovieLensALS example

b066bbc

remove main methods from linear regression algorithms and add LinearR…

67f4448

…egression example

remove main from KMeans and add DenseKMeans as an example

fe23dcb

change NaiveBayesParams to Params

8bfaa5a

move DecisionTree's main method to examples

b3edf68

mengxr mentioned this pull request Apr 28, 2014

SPARK-1637: Clean up examples for 1.0 #571

Closed

3 tasks

rxin reviewed Apr 29, 2014
View reviewed changes

use main instead of extending App

be86069

mengxr added 2 commits April 28, 2014 21:44

use scopt for DecisionTreeRunner

6acff75

update TS PCA/SVD to use new spark-submit

c6178c9

make imports explicit and fix code style

6e35d7e

minor

7b58c60

asfgit closed this in 3f38334 Apr 29, 2014

mengxr deleted the mllib-main branch May 7, 2014 00:06

peter-toth mentioned this pull request Mar 25, 2020

[SPARK-29375][SQL] Exchange reuse across all subquery levels #26044

Closed

arjunshroff pushed a commit to arjunshroff/spark that referenced this pull request Nov 24, 2020

MapR [SPARK-517] Can't use ssl via spark beeline (apache#584)

3d00f35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-1636][MLLIB] Move main methods to examples #584

[SPARK-1636][MLLIB] Move main methods to examples #584

mengxr commented Apr 28, 2014

AmplabJenkins commented Apr 28, 2014

AmplabJenkins commented Apr 28, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

rxin Apr 29, 2014

mengxr Apr 29, 2014

rxin commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

mengxr commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

rxin commented Apr 29, 2014

[SPARK-1636][MLLIB] Move main methods to examples #584

[SPARK-1636][MLLIB] Move main methods to examples #584

Conversation

mengxr commented Apr 28, 2014

AmplabJenkins commented Apr 28, 2014

AmplabJenkins commented Apr 28, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

rxin Apr 29, 2014

Choose a reason for hiding this comment

mengxr Apr 29, 2014

Choose a reason for hiding this comment

rxin commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

mengxr commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

AmplabJenkins commented Apr 29, 2014

rxin commented Apr 29, 2014