Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[KYUUBI-105]add a failover mode for kyuubi ha #106

Merged
merged 5 commits into from
Sep 29, 2018
Merged

[KYUUBI-105]add a failover mode for kyuubi ha #106

merged 5 commits into from
Sep 29, 2018

Conversation

yaooqinn
Copy link
Member

Is your feature request related to a problem? Please describe.

Kyuubi now supports load balance service mode for client to discover from a zookeeper namespace. In this mode, all kyuubi service instance uri in the namespace is randomly picked by the client itself.
The picked server will create a Spark application for this user(client). When this user create another client through zookeeper service discovery, there will be two very different situations. one is that the previous server is picked and the Spark application reused, the other one is that the previous server is not picked with a high probability and another Spark application will be created. This is good for large cluster which has enough resources. But for those cluster that are not well resourced, it is a waste to do so. When there're not enough resources to create a Spark application, the client will fail to create a session, which in fact should not happen.

Describe the solution you'd like

An Active/Standby mode may be able to overcome such a problem and still supply Kyuubi with high availability functions.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

@yaooqinn yaooqinn added this to the v0.4.0 milestone Sep 27, 2018
@yaooqinn yaooqinn self-assigned this Sep 27, 2018
@yaooqinn yaooqinn added this to To do in kyuubi via automation Sep 27, 2018
@codecov
Copy link

codecov bot commented Sep 27, 2018

Codecov Report

Merging #106 into master will increase coverage by 1.14%.
The diff coverage is 82.24%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #106      +/-   ##
============================================
+ Coverage     65.34%   66.48%   +1.14%     
- Complexity      447      475      +28     
============================================
  Files            67       71       +4     
  Lines          3353     3390      +37     
  Branches        487      487              
============================================
+ Hits           2191     2254      +63     
+ Misses          906      878      -28     
- Partials        256      258       +2
Impacted Files Coverage Δ Complexity Δ
...src/main/scala/yaooqinn/kyuubi/service/State.scala 100% <ø> (ø) 0 <0> (ø) ⬇️
...n/scala/yaooqinn/kyuubi/yarn/KyuubiAppMaster.scala 43.29% <0%> (-0.71%) 9 <0> (-1)
...r/src/main/scala/org/apache/spark/KyuubiConf.scala 99.54% <100%> (ø) 0 <0> (ø) ⬇️
...cala/yaooqinn/kyuubi/ha/ZooKeeperACLProvider.scala 100% <100%> (ø) 4 <4> (?)
.../main/scala/org/apache/spark/KyuubiSparkUtil.scala 97.19% <100%> (ø) 0 <0> (ø) ⬇️
...ain/scala/yaooqinn/kyuubi/ha/FailoverService.scala 50% <50%> (ø) 8 <8> (?)
...in/scala/yaooqinn/kyuubi/server/KyuubiServer.scala 94.87% <85.71%> (-2.28%) 10 <0> (+2)
...cala/yaooqinn/kyuubi/ha/HighAvailableService.scala 87.2% <87.2%> (ø) 7 <7> (?)
.../scala/yaooqinn/kyuubi/ha/LoadBalanceService.scala 91.66% <91.66%> (ø) 5 <5> (?)
...n/scala/yaooqinn/kyuubi/ha/JaasConfiguration.scala 94.73% <94.73%> (ø) 3 <3> (?)
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 835259a...b591f05. Read the comment docs.

@yaooqinn yaooqinn merged commit 202e1bd into master Sep 29, 2018
kyuubi automation moved this from To do to Done Sep 29, 2018
@yaooqinn yaooqinn deleted the KYUUBI-105 branch September 29, 2018 09:17
@yaooqinn yaooqinn modified the milestone: v0.4.0 Dec 21, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
kyuubi
  
Done
Development

Successfully merging this pull request may close these issues.

None yet

1 participant