Coordinated Shutdown (improve cluster leaving)

[patriknw]

There are lot of things that can be improved around graceful cluster leaving.

• the process of leaving and finally shutdown actor system and jvm is complicated,
• it becomes even more complicated when cluster sharding and cluster singleton should also be gracefully stopped
• it's rather slow
• the usage of the failure detector unreachability marker for the final steps before removing is questionable

Somewhat related tickets:
#18373
#21298
#21521

[patriknw]

We discussed a solution for this. Introduce some kind of CoordinatedShutdown manager in akka-actor where different modules can register their "shutdown hook" (might need both graceful and forceful hooks). Need dependencies between modules for sequential shutdown steps in right order (probably just with string identifiers in config). Might also need different phases?

[marekzebrowski]

Leaving cluster is indeed a complex process. One issue is to when trigger leaving and second is to what needs to be done and in what order.
Our approach so far:
When - Runtime.getRuntime.addShutdownHook
Oder:

1. akkaHttp server
2. custom long-running flows
3. shardRegions + other singletons
4. leave cluster - awaiting for successful MemberExited(addr) message
5. shutdown actor system
   we also have a special case useful for developement, which can be considered a hack: if there is one or less servers on seeds list, we don't wait for anything and just shut down immediately. Rationale behind it is one-node setup is development, and some test machines, so we don't need for graceful shutdown in such case. As we routinely do rolling restarts, steps like gracefully shutting down shard regions are important - if they fail, we need to restart the whole cluster

unfortunately Runtime.getRuntime.addShutdownHook stopped to work when we tried to use artery remoting in akka 2.4.14 as artery shuts down before shutdown other shutdown hooks are completed

[patriknw]

@marekzebrowski @guidomedina I have been working on PR #21930. Most things are in place (a few things are not done yet). I think it will handle all of the things you have had problems with automatically, and you can hook in your own tasks in the shutdown phases.

It also has support for adding jvm shutdown hooks that are run before the Artery shutdown hook. The coordinated shutdown is started automatically by such shutdown hook, so you probably don't even have to add any such yourself.