[LIVY-667][WIP] Collect a part of partition to the driver by batch to avoid OOM

[runzhiwang]

What changes were proposed in this pull request?

Collecting a part of partition to the driver by batch to avoid OOM

Background:
1.When enable livy.server.thrift.incrementalCollect, thrift use "toLocalIterator" to load one partition at each time instead of the whole rdd to avoid OutOfMemory. However, if the largest partition is too big, the OutOfMemory still occurs.

2.This PR collect a part of partition to the driver by batch at each time to avoid OOM.

How was this patch tested?

create a big size of data into one partition and query them all.

[codecov-io]

Codecov Report

Merging #233 into master will decrease coverage by 0.03%.
The diff coverage is 75%.

@@             Coverage Diff              @@
##             master     #233      +/-   ##
============================================
- Coverage     68.37%   68.33%   -0.04%     
  Complexity      921      921              
============================================
  Files           100      100              
  Lines          5729     5732       +3     
  Branches        871      872       +1     
============================================
  Hits           3917     3917              
- Misses         1247     1249       +2     
- Partials        565      566       +1

Impacted Files	Coverage Δ	Complexity Δ
...rver/src/main/scala/org/apache/livy/LivyConf.scala	95.85% <ø> (-0.03%)	21 <0> (ø)
rsc/src/main/java/org/apache/livy/rsc/RSCConf.java	86.23% <100%> (+0.25%)	7 <0> (ø)	⬇️
...e/livy/server/interactive/InteractiveSession.scala	69.25% <50%> (-0.12%)	46 <0> (ø)
...ain/scala/org/apache/livy/utils/SparkYarnApp.scala	66.01% <0%> (-1.31%)	40% <0%> (ø)
...ain/java/org/apache/livy/rsc/driver/RSCDriver.java	77.96% <0%> (ø)	41% <0%> (ø)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b8251eb...6df4ede. Read the comment docs.