[SPARK-13672] [ML] Add python examples of BisectingKMeans in ML and MLLIB

[zhengruifeng]

JIRA: https://issues.apache.org/jira/browse/SPARK-13672

What changes were proposed in this pull request?

add two python examples of BisectingKMeans for ml and mllib

How was this patch tested?

manual tests

[SparkQA]

Test build #52455 has finished for PR 11515 at commit 5ed2a47.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #52456 has finished for PR 11515 at commit e6da291.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #52457 has finished for PR 11515 at commit ebce780.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MLnick]

While trying to run this, got an exception:

TypeError: unbound method train() must be called with BisectingKMeans instance as first argument (got PipelinedRDD instance instead)

train is missing a @classmethod annotation here. You can just add that in this PR.

[zhengruifeng]

the annotation was added

[SparkQA]

Test build #52751 has finished for PR 11515 at commit cea8ddf.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MLnick]

Could we make this example more consistent with the style of the other one (and the ML kmeans example):

from pyspark.sql.types import Row
from pyspark.mllib.linalg import Vectors
...
data = sc.textFile("data/mllib/kmeans_data.txt")
parsedData = data.map(lambda line: Row(features=Vectors.dense([float(x) for x in line.split(' ')])))
training = sqlContext.createDataFrame(parsedData)
...

[MLnick]

Please add an include_example for the Python example in mllib-clustering.md

[zhengruifeng]

@MLnick I have add an include_example in mllib-clustering.md. And some changes were make according to your commentations.

[SparkQA]

Test build #52890 has finished for PR 11515 at commit 399290c.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #52892 has finished for PR 11515 at commit d441511.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #52894 has finished for PR 11515 at commit 165a4fe.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[zhengruifeng]

Jenkins test this please

[SparkQA]

Test build #52906 has finished for PR 11515 at commit 165a4fe.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[MLnick]

Thanks! Merging to master.

[rmchurch]

This example doesn't seem to work in Spark 2.0.0, and from the master mllib/clustering.py, I don't expect it to work in the most updated code either. Specifically, the Python BisectingKMeansModel class does not have a save method (the KMeansModel class does), so that the last three lines of the following code do not work:

# Build the model (cluster the data)
model = BisectingKMeans.train(parsedData, 2, maxIterations=5)

# Evaluate clustering
cost = model.computeCost(parsedData)
print("Bisecting K-means Cost = " + str(cost))

# Save and load model
path = "target/org/apache/spark/PythonBisectingKMeansExample/BisectingKMeansModel"
model.save(sc, path)
sameModel = BisectingKMeansModel.load(sc, path)

[zhengruifeng]

@rmchurch I think this bug has been resolved in #16515