Provide support for graceful startup / shutdown

[abuijze]

Currently, the shutdown sequence in the configuration api shuts down all components practically simultaneously. This may cause problems when requests are being handled, while components (like async command buses) are being shut down.
Instead, ‘edge’ components should block incoming calls and wait for running invocations to finish.
Edge components are

• incoming connectors to the distributed command bus
• command bus (an extra method could be made available on the configuration to provide an "edge command bus". This wraps the regular command bus and allows blocking incoming calls on shutdown)
• tracking processors
• connectors to 3rd party messaging systems (e.g. RabbitMQ)

When those components have completed the running requests, the prepare-shutdown
sequence of other components may start. In this phase, async components need to empty their processing queues. They should not block new tasks, as other async processes may still provide tasks to complete.

Finally, components are shut down, rejecting any new requests and closing resources.

[sgrimm-sg]

In case it's helpful: My Axon 2.4 application implements graceful shutdown. Obviously Axon 4 is different, but the sequence I used mixes shutdown of Axon and non-Axon components in an attempt to make the shutdown process finish as quickly as possible without losing any data.

This sequence assumes a distributed command bus but local event delivery.

1. Stop the event scheduler from polling for / delivering events. (But new events can still be scheduled.)
2. Set the distributed command bus load factor to 0, but stay connected to the cluster. Block until the load factor change is reflected in the consistent hash.
3. Shut down the application's HTTP service, waiting until any in-flight HTTP requests are finished. This happens after step 2 because we want any commands issued by in-flight requests to be sent to other nodes in the cluster, if any; if we did it before step 2 then in-flight requests could generate more work to do locally and the shutdown would take longer to finish.
4. Wait for application-managed worker threads to finish.
5. Set a flag in the event bus (we wrap the event bus in a custom class for this purpose) that causes newly-generated events to be deferred using the event scheduler rather than processed immediately. Obviously this won't be needed for tracking event processors, but some kind of deferral mechanism still will be for subscribing processors.
6. Wait for any events that are already in flight or already queued to finish processing.
7. Shut down all the saga managers and the event bus.
8. Wait for the command bus to finish handling all in-flight or queued commands.
9. Shut down the command bus.
10. Shut down the write side of the event scheduler.

To make this work flawlessly, I had to add wrappers for DistributedCommandBus, SimpleCommandBus, and SimpleEventBus, and I had to make a small change to DistributedCommandBus itself.

DistributedCommandBus: Changed it to throw an exception if an asynchronous command can't be sent to the cluster. (This is the behavior described in the original class's Javadoc, but it didn't actually throw.)

DistributedCommandBus wrapper: Normally, delegates to DistributedCommandBus. But during shutdown, if there are no nodes available to handle a command (in other words, if this is the last node in the cluster to be shut down) and the command is asynchronous, the command gets wrapped in an event and sent to the event scheduler for later delivery. If the command has a callback then there's no choice but to dispatch it immediately. We're deferring newly-published events in that case as described above, so this is safe even after the local event bus and saga managers are shut down.

SimpleCommandBus wrapper: Normally, delegates to SimpleCommandBus. But after the load factor has been set to 0 during the shutdown sequence, it instead delegates to the DistributedCommandBus wrapper so commands can be sent to other nodes. This is mostly needed to handle cases where the outgoing load factor update and an incoming command hit the network at the same time; that's not super common but happens reproducibly while the cluster is under heavy load. If we don't do this, and instead just process commands locally during shutdown, it's a correctness issue because the current node will no longer be the correct target for the command, and we thus might end up running commands for the same aggregate concurrently with another node.

SimpleEventBus wrapper: Normally, delegates to SimpleEventBus. During shutdown, implements the "defer event delivery by pushing events to the event scheduler" behavior described above. Event ordering is maintained by bumping the scheduling delay up by 1ms each time an event is deferred.

[srnm]

For reference, Spring Integration has a multi-phase Orderly Shutdown process: https://docs.spring.io/spring-integration/docs/5.1.6.RELEASE/reference/html/#jmx-shutdown

[m1l4n54v1c]

Allow auto-start of components to be configured in case of Spring. Sometimes, applications need to execute upgrades before components should be started.