Bug 1390320 - Allow Dataset api consumers to control the amount of parallelism used

[thomcc]

As it is it often forces a potentially very expensive shuffle in order to increase the parallelism. This was causing performance issues in a zeppelin notebook I was using for bug 1381641, where I'd get around 3 partitions and want to expand them to a much larger size.

It would probably be simpler without the threshold checking, but given it's current value, it seems like it might be intended to prevent something further down the line from hitting some signed integer overflow later, I wasn't sure.

I've made it opt-in, although it seems likely to be a better default than the current (if you ask me).

[codecov-io]

Codecov Report

Merging #6 into master will decrease coverage by 1.37%.
The diff coverage is 84.61%.

@@            Coverage Diff             @@
##           master       #6      +/-   ##
==========================================
- Coverage   99.21%   97.84%   -1.38%     
==========================================
  Files           5        5              
  Lines         127      139      +12     
  Branches       20       23       +3     
==========================================
+ Hits          126      136      +10     
- Misses          1        3       +2

Impacted Files	Coverage Δ
...ain/scala/com/mozilla/telemetry/heka/Dataset.scala	97.36% <66.66%> (-2.64%)	⬇️
...la/com/mozilla/telemetry/utils/ObjectSummary.scala	94.11% <90%> (-5.89%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 96e42c6...f15dd47. Read the comment docs.

[maurodoglio]

Hey @thomcc thanks for the awesome patch.
I did a similar work for the python implementation of Dataset here. In general I like your solution as it solves the problem of having fewer partitions than executors so let's ship it. In the future we may want to change it to give priority to the number of files in a group rather than the total file size, but let's keep it for another day.

[maurodoglio]

Almost forgot, let's keep the current behaviour the default for now. Once we have verified on a real cluster that your code performs better we can remove the old code and remove the switch.

[maurodoglio]

🚢 it

[mreid-moz]

let's keep the current behaviour the default for now. Once we have verified
on a real cluster that your code performs better we can remove the old code
and remove the switch.

@maurodoglio Is there an issue/bug for this? If not, please file a follow-up so it remains on our radar.