[SPARK-9615][SPARK-9616][SQL][MLLIB] Bugs related to FrequentItems when merging and with Tungsten

[brkyvz]

In short:
1- FrequentItems should not use the InternalRow representation, because the keys in the map get messed up. For example, every key in the Map correspond to the very last element observed in the partition, when the elements are strings.

2- Merging two partitions had a bug:

Existing behavior with size 3
Partition A -> Map(1 -> 3, 2 -> 3, 3 -> 4)
Partition B -> Map(4 -> 25)
Result -> Map()

Correct Behavior:
Partition A -> Map(1 -> 3, 2 -> 3, 3 -> 4)
Partition B -> Map(4 -> 25)
Result -> Map(3 -> 1, 4 -> 22)

cc @mengxr @rxin @JoshRosen

[SparkQA]

Test build #39762 has finished for PR 7945 at commit 506753e.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

Since we still need to recompute minCount after each update, I think the code is simpler without tracking minCount in all cases. We can compute minCount directly in this branch.

if (baseMap.contains(key)) {
  baseMap(key) += count
} else {
  baseMap(key) = count
  if (baseMap.size > size) {
    val minCount = baseMap.values.min
    baseMap.retain((_, v) => v > minCount)
    baseMap.transform((_, v) => v - minCount)
  }
}

[SparkQA]

Test build #39884 has finished for PR 7945 at commit 1dc61a8.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

test this please

[mengxr]

minor, this could happen after transforming exiting counts

[brkyvz]

the reason I had this up here, was so that this gets deleted if count = minCount, and that we don't add key -> 0 to the map

[mengxr]

okay, this is minor

[SparkQA]

Test build #39910 has finished for PR 7945 at commit 1dc61a8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #39956 has finished for PR 7945 at commit 07fa001.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[mengxr]

LGTM. Merged into master and branch-1.5. Thanks!