Bug 1349608 - Improve Dataset handling of small pings

[maurodoglio]

The idea here is to use a greedy algorithm to balance the RDD partition
in terms of number of files (first) and total size of each partition.

[coveralls]

Coverage decreased (-0.1%) to 68.486% when pulling daf5526 on group-by-size-min into d52da12 on master.

[coveralls]

Coverage decreased (-0.1%) to 68.486% when pulling daf5526 on group-by-size-min into d52da12 on master.

[coveralls]

Coverage decreased (-0.1%) to 68.486% when pulling daf5526 on group-by-size-min into d52da12 on master.

[maurodoglio]

I ran a few comparison tests on 2 different queries:

• 10% of every ping with docType = OTHER
  • Time before change: 16min 40s
  • Time after change: 10min 43s (-34.7%)
• 10% of main pings for a given submissionDate
  • Time before change: 35min 20s
  • Time after change: 22min 47s (-35.5%)

As a bonus point the logic in the grouping function is now simpler (and more readable).
I also tried a slightly more sophisticated implentation before this one, which was always selecting the smallest partition as the destination for a file. That didn't perform as well as this one, the execution time was half way between the original implementation and the current one.

[coveralls]

Coverage decreased (-0.1%) to 68.486% when pulling dc2ce0a on group-by-size-min into d52da12 on master.

[mreid-moz]

LGTM

[maurodoglio]

I forgot to mention, to make the comparison more correct, I bypassed the sampling provided by .records and instead I used the same set of pre-sampled summaries for both. I can provide the sampled data as json if anybody is interested.