[LIVY-545] Try to clear buffer by judge expired or not, rather than drop Eldest …

[WeiWenda]

What changes were proposed in this pull request?

https://issues.apache.org/jira/browse/LIVY-545

1. Try to cleanup _statements of Session by judge expired or not, rather than drop Eldest statement casually.
2. Reject statement request when buffer is full and none is expired

• The new Statechart Diagram of Statement looks like this: (brown box means inner sate, user unknowable)
  
• so that user can configure these properties to control a statement's lifecycle

Property Name	Default	Meaning
livy.rsc.retained-statements	100	Buffer size of statement
livy.rsc.result-discard.timeout	10m	Expired Timeout after read，When the user retrieves the query result, the state change to Readed from Available, and automatically becomes Expired after 10 minutes. Expired means when the buffer queue space is insufficient, this statement may be moved out of the buffer queue.
livy.rsc.result-retained.timeout	1h	Expired timeout before read，After statement became Available, If user never get the query result within 1 hour , StatementState will change to Expired automatically.

How was this patch tested?

we run a jmeter test of 20 threads each post statement 1/60s, set "spark.scheduler.mode": "FAIR", print debug message in AM stdout, and examine the buffer state.

we modify the "should remove expired statements when reaching threshold" of SessionSpec to fit this new character.

[codecov-io]

Codecov Report

Merging #136 into master will increase coverage by <.01%.
The diff coverage is 67.36%.

@@             Coverage Diff              @@
##             master     #136      +/-   ##
============================================
+ Coverage     68.38%   68.38%   +<.01%     
- Complexity      891      908      +17     
============================================
  Files           100      100              
  Lines          5596     5669      +73     
  Branches        839      856      +17     
============================================
+ Hits           3827     3877      +50     
- Misses         1223     1237      +14     
- Partials        546      555       +9

Impacted Files	Coverage Δ	Complexity Δ
...ava/org/apache/livy/rsc/driver/StatementState.java	0% <0%> (ø)	0 <0> (ø)	⬇️
rsc/src/main/java/org/apache/livy/rsc/RSCConf.java	86.11% <100%> (+0.26%)	7 <0> (ø)	⬇️
...c/main/scala/org/apache/livy/repl/ReplDriver.scala	30.23% <20%> (-0.54%)	7 <0> (ø)
.../src/main/scala/org/apache/livy/repl/Session.scala	67.05% <70.11%> (-0.14%)	54 <21> (+17)
...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala	50% <0%> (-5.89%)	7% <0%> (ø)
...scala/org/apache/livy/repl/PythonInterpreter.scala	48.12% <0%> (+0.62%)	9% <0%> (ø)	⬇️
...ain/scala/org/apache/livy/utils/SparkYarnApp.scala	73.23% <0%> (+0.7%)	33% <0%> (ø)	⬇️
...main/scala/org/apache/livy/server/LivyServer.scala	35.51% <0%> (+1.63%)	9% <0%> (ø)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e2d2189...cbee58e. Read the comment docs.

[alex-the-man]

Would you mind describing how you are using interactive sessions?

[WeiWenda]

Would you mind describing how you are using interactive sessions?

we use livy to provide spark sql restful server as a substitute of hive-thriftserver. Such problem only happen when too many user connect to the same livy session. But as now, we find it maybe unrealistic, as spark FAIR schedule mode's performance is so pool, can only deal with 20 around query statement in the same time.

[mgaido91]

@WeiWenda we just introduced a thrift endpoint in Livy (see https://issues.apache.org/jira/browse/LIVY-489). Do you think that would help your use case?

[WeiWenda]

@WeiWenda we just introduced a thrift endpoint in Livy (see https://issues.apache.org/jira/browse/LIVY-489). Do you think that would help your use case?

Thank you! It's much better than raw STS, I'm glad to use this feature of livy.