Skip to content

Commit

Permalink
[SPARK-22899][ML][STREAMING] Fix OneVsRestModel transform on streamin…
Browse files Browse the repository at this point in the history
…g data failed.

## What changes were proposed in this pull request?

Fix OneVsRestModel transform on streaming data failed.

## How was this patch tested?

UT will be added soon, once #19979 merged. (Need a helper test method there)

Author: WeichenXu <weichen.xu@databricks.com>

Closes #20077 from WeichenXu123/fix_ovs_model_transform.
  • Loading branch information
WeichenXu123 authored and jkbradley committed Dec 28, 2017
1 parent 774715d commit 753793b
Showing 1 changed file with 1 addition and 1 deletion.
Expand Up @@ -165,7 +165,7 @@ final class OneVsRestModel private[ml] (
val newDataset = dataset.withColumn(accColName, initUDF())

// persist if underlying dataset is not persistent.
val handlePersistence = dataset.storageLevel == StorageLevel.NONE
val handlePersistence = !dataset.isStreaming && dataset.storageLevel == StorageLevel.NONE
if (handlePersistence) {
newDataset.persist(StorageLevel.MEMORY_AND_DISK)
}
Expand Down

0 comments on commit 753793b

Please sign in to comment.