Skip to content

Commit

Permalink
[SPARK-13201][SPARK-13200] Deprecation warning cleanups: KMeans & MFD…
Browse files Browse the repository at this point in the history
…ataGenerator

KMeans:
Make a private non-deprecated version of setRuns API so that we can call it from the PythonAPI without deprecation warnings in our own build. Also use it internally when being called from train. Add a logWarning for non-1 values

MFDataGenerator:
Apparently we are calling round on an integer which now in Scala 2.11 results in a warning (it didn't make any sense before either). Figure out if this is a mistake we can just remove or if we got the types wrong somewhere.

I put these two together since they are both deprecation fixes in MLlib and pretty small, but I can split them up if we would prefer it that way.

Author: Holden Karau <holden@us.ibm.com>

Closes #11112 from holdenk/SPARK-13201-non-deprecated-setRuns-SPARK-mathround-integer.
  • Loading branch information
holdenk authored and srowen committed Feb 9, 2016
1 parent 159198e commit ce83fe9
Show file tree
Hide file tree
Showing 3 changed files with 13 additions and 5 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -357,7 +357,7 @@ private[python] class PythonMLLibAPI extends Serializable {
val kMeansAlg = new KMeans()
.setK(k)
.setMaxIterations(maxIterations)
.setRuns(runs)
.internalSetRuns(runs)
.setInitializationMode(initializationMode)
.setInitializationSteps(initializationSteps)
.setEpsilon(epsilon)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,18 @@ class KMeans private (
@Since("0.8.0")
@deprecated("Support for runs is deprecated. This param will have no effect in 2.0.0.", "1.6.0")
def setRuns(runs: Int): this.type = {
internalSetRuns(runs)
}

// Internal version of setRuns for Python API, this should be removed at the same time as setRuns
// this is done to avoid deprecation warnings in our build.
private[mllib] def internalSetRuns(runs: Int): this.type = {
if (runs <= 0) {
throw new IllegalArgumentException("Number of runs must be positive")
}
if (runs != 1) {
logWarning("Setting number of runs is deprecated and will have no effect in 2.0.0")
}
this.runs = runs
this
}
Expand Down Expand Up @@ -502,7 +511,7 @@ object KMeans {
seed: Long): KMeansModel = {
new KMeans().setK(k)
.setMaxIterations(maxIterations)
.setRuns(runs)
.internalSetRuns(runs)
.setInitializationMode(initializationMode)
.setSeed(seed)
.run(data)
Expand All @@ -528,7 +537,7 @@ object KMeans {
initializationMode: String): KMeansModel = {
new KMeans().setK(k)
.setMaxIterations(maxIterations)
.setRuns(runs)
.internalSetRuns(runs)
.setInitializationMode(initializationMode)
.run(data)
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -105,8 +105,7 @@ object MFDataGenerator {

// optionally generate testing data
if (test) {
val testSampSize = math.min(
math.round(sampSize * testSampFact), math.round(mn - sampSize)).toInt
val testSampSize = math.min(math.round(sampSize * testSampFact).toInt, mn - sampSize)
val testOmega = shuffled.slice(sampSize, sampSize + testSampSize)
val testOrdered = testOmega.sortWith(_ < _).toArray
val testData: RDD[(Int, Int, Double)] = sc.parallelize(testOrdered)
Expand Down

0 comments on commit ce83fe9

Please sign in to comment.