[LIVY-729] Fix livy recover the killed session

[runzhiwang]

What changes were proposed in this pull request?

Follows are steps to reproduce the problem:

1. Set livy.server.recovery.mode=recovery, and create an interactive session: session0 in yarn-cluster
2. kill the yarn application of the session
3. restart livy
4. livy try to recover session0, but application has been killed and driver does not exist, so client can not connect to driver, and exception was thrown as the image.
5. If the ip:port of the driver was reused by session1, client of session0 will try to connect to driver of session1, then driver will throw exception: Unexpected client ID.
6. Both the exception threw by livy and driver will confuse the user, and recover a lot of killed sessions will delay the recover of alive session.
   

How to fix:
When session was killed or dead, remove the session from the store

How was this patch tested?

Existed IT and UT.

[runzhiwang]

@jerryshao @yiheng Could you help to review ?

[codecov-io]

Codecov Report

Merging #266 into master will increase coverage by 0.09%.
The diff coverage is 76.47%.

@@             Coverage Diff              @@
##             master     #266      +/-   ##
============================================
+ Coverage     68.09%   68.18%   +0.09%     
- Complexity      939      945       +6     
============================================
  Files           102      102              
  Lines          5883     5897      +14     
  Branches        893      896       +3     
============================================
+ Hits           4006     4021      +15     
  Misses         1302     1302              
+ Partials        575      574       -1

Impacted Files	Coverage Δ	Complexity Δ
...che/livy/server/recovery/ZooKeeperStateStore.scala	77.27% <0%> (-1.8%)	17 <0> (ø)
...e/livy/server/interactive/InteractiveSession.scala	70% <100%> (+0.63%)	51 <0> (+5)	⬆️
...he/livy/server/recovery/FileSystemStateStore.scala	62.5% <66.66%> (-0.55%)	11 <1> (ø)
...la/org/apache/livy/server/batch/BatchSession.scala	86.73% <66.66%> (+0.56%)	14 <0> (ø)	⬇️
...ain/java/org/apache/livy/rsc/driver/RSCDriver.java	80.75% <0%> (+0.83%)	46% <0%> (+1%)	⬆️
...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala	55.88% <0%> (+2.94%)	7% <0%> (ø)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 393c162...1a53b03. Read the comment docs.

[jerryshao]

+1

[yiheng]

+1