[SPARK-16697][ML][MLLib] improve LDA submitMiniBatch method to avoid redundant RDD computation #14335
Conversation
|
Test build #62771 has finished for PR 14335 at commit
|
| @@ -472,12 +473,13 @@ final class OnlineLDAOptimizer extends LDAOptimizer { | |||
| gammaPart = gammad :: gammaPart | |||
| } | |||
| Iterator((stat, gammaPart)) | |||
| } | |||
| }.persist(StorageLevel.MEMORY_AND_DISK) | |||
There was a problem hiding this comment.
It's a little risky to assume this can be persisted, but I think it's good idea unless it can be optimized into one pass
There was a problem hiding this comment.
@srowen The type of the RDD to be persisted here is fixed to RDD[(BDM[Double], List[BDV[Double]])] so what's the risk of cannot be persisted ?
There was a problem hiding this comment.
Oh, I mean, it's not always clear that it's OK to use the memory/disk, but, I think it's justified here. Yes it will serialize.
|
@srowen |
|
Test build #62853 has finished for PR 14335 at commit
|
|
Merged to master |
What changes were proposed in this pull request?
In
LDAOptimizer.submitMiniBatch, do persist onstats: RDD[(BDM[Double], List[BDV[Double]])]and also move the place of unpersisting
expElogbetaBcbroadcast variable,to avoid the
expElogbetaBcbroadcast variable to be unpersisted too early,and update previous
expElogbetaBc.unpersist()intoexpElogbetaBc.destroy(false)How was this patch tested?
Existing test.