[SPARK-9560][MLlib] add lda data generator

[hhbyyh]

jira: https://issues.apache.org/jira/browse/SPARK-9560

Add data generator for LDA.
Hope it can help with performance improvement.

[SparkQA]

Test build #39558 has finished for PR 7898 at commit 7cc6a95.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #40135 has finished for PR 7898 at commit aff4d7f.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[feynmanliang]

Thanks! Have you tried running LDA on the generated data? What were the results?

I think @jkbradley meant for this to be contributed to spark-perf and used for perf testing, but I think it might also be useful for validating LDA correctness...

[hhbyyh]

@feynmanliang I only tried with small data set and result seems to be reasonable.

@jkbradley I can move this to spark-perf if it's not suitable for Spark.

[SparkQA]

Test build #60012 has finished for PR 7898 at commit aff4d7f.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #65382 has finished for PR 7898 at commit aff4d7f.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[hhbyyh]

Close this as we've stopped development for MLlib.