Spark 7998 freq item api

[dwmclary]

Here's "a better frequent item API" which provides a DataFrame with each ArrayBuffer expanded into a column. There's surely some improvement that could be done here, but I think this is in the spirit of what the JIRA was asking for.

[davies]

ok to test

[SparkQA]

Test build #35377 has finished for PR 6919 at commit 8dec609.

• This patch fails Scala style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35379 has finished for PR 6919 at commit d0ce7d5.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35380 has finished for PR 6919 at commit 6f25872.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35382 has finished for PR 6919 at commit ff676c9.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35396 has finished for PR 6919 at commit 7ce1248.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35397 has finished for PR 6919 at commit 8c239c7.

• This patch fails Python style tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #35398 has finished for PR 6919 at commit 80c841b.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dwmclary]

So, I'm wondering if the Scala-specific method actually needs to re-implement, or if it would be cleaner to just call mutable.copyToArray and pass it to the agnostic function. Any thoughts @rxin?

[dwmclary]

@davies any review comments?

[davies]

ping @rxin

[dwmclary]

ping @rxin ?

[rxin]

Sorry need some time to think about this.

[dwmclary]

No problem -- just wanted to make sure it was on your radar.

On Wed, Jul 8, 2015 at 12:55 AM, Reynold Xin notifications@github.com
wrote:

Sorry need some time to think about this.

—
Reply to this email directly or view it on GitHub
#6919 (comment).

[rxin]

@dwmclary do you mind closing this pull request?

Discussed with Xiangrui Meng and Burak Yavuz offline. We are not going to change the API, but just update the documentation to explain more clearly the schema and how to get the frequent values out.
In 1.6, we will likely create a frequent items UDAF.

[dwmclary]

Closed.

On Wed, Aug 5, 2015 at 12:49 PM, Reynold Xin notifications@github.com
wrote:

@dwmclary https://github.com/dwmclary do you mind closing this pull
request?

Discussed with Xiangrui Meng and Burak Yavuz offline. We are not going to
change the API, but just update the documentation to explain more clearly
the schema and how to get the frequent values out.
In 1.6, we will likely create a frequent items UDAF.

—
Reply to this email directly or view it on GitHub
#6919 (comment).