rolling restart

[adyach]

Description

Rolling restart of the cluster:

1. User asks for restart usig cli: bubuku-cli rolling-restart --image-tag 123 --instnce-type t2.nano
2. Bubuku creates restart assignment and triggers an action for rolling restart for the first broker in assignment at the same time checking for cluster state using kafka jmx metrics
3. Restart: broker is stopped, volume is detached, instance terminated, instance launched and broker is started
4. Once broker is being restarted is up, broker is making restart triggers rolling restart action with restart assignment without the broker which was restarted
5. Restart assignment is empty then rolling restart is finished

[rcillo]

@adyach could you please explain this to me later. I don't understand it.

[rcillo]

@adyach why not let AWS choose an available IP address for you?

[rcillo]

                    .__   
  ____  ____   ____ |  |  
_/ ___\/  _ \ /  _ \|  |  
\  \__(  <_> |  <_> )  |__
 \___  >____/ \____/|____/
     \/                   

[rcillo]

This is really cool 👍

Some corner cases that could happen but I'm not sure we need to address them right now:

1. Kafka leader imbalance could make this pause permanently between restarts. We've seen that happening. Let's see how it goes.
2. The action is consumed and posted back on each step. The problem with that is that some problems during the execution of a step could cause an unrecoverable failure. I mean, if at step 3 out of 5 something bad happens and an unrecoverable exception is launched, wouldn't that cause the complete process to stop and we would have to somehow identify where it stopped? Please, correct me if I understood it wrong.

[adyach]

Thank you for review! 1. It is intentionally blocking restart if there is prefered replica imbalance, we observed in the past version of Kafka, it could imfluence behavior. I agree, we need to observe it for sometime. 2. The restart can stop if bubuku performing restart was stopped, in that case the restart should be started again from scratch. This is intentionally done like that, because it is quite bad situation, when broker is down while it is restarting other one. The state is not saved, this is not perfect, we can improve it if required. If one of the steps fails with exception, it will be triggered again.

On Thu 20. Dec 2018 at 12:50, Ricardo de Cillo ***@***.***> wrote: This is really cool 👍 Some corner cases that could happen but I'm not sure we need to address them right now: 1. Kafka leader imbalance could make this pause permanently between restarts. We've seen that happening. Let's see how it goes. 2. The action is consumed and posted back on each step. The problem with that is that some problems during the execution of a step could cause an unrecoverable failure. I mean, if at step 3 out of 5 something bad happens and an unrecoverable exception is launched, wouldn't that cause the complete process to stop and we would have to somehow identify where it stopped? Please, correct me if I understood it wrong. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#124 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEo3QmS5BsvbPbJDZ8S-VBU1fcGkOjGQks5u63mVgaJpZM4ZbtFn> .

-- With great enthusiasm, Andrey

[rcillo]

Ok, now I remember that bubuku has a "catch all" exception and it would retry the action, right? So the only chance of having a partial rolling restart would be to kill bubuku process with -9, for example. If that's the case, then I agree that it's quite dramatic event and maybe we would like to start all over.

[rcillo]

👍

[rcillo]

I checked the lastest change. Was it necessary because it would retry too often?

[adyach]

👍

[antban]

👍