Can't close Gateway with ongoing requests

[Zelldon]

Scenario:

• Start a Broker with embedded Gateway
• Create a Job
• Create new Worker
• Stop the broker with the Gateway
• -> Deadlock Gateway can't be closed

    clientRule.getJobClient().newCreateCommand().jobType("test").send().join();
    final RecordingJobHandler jobHandler = new RecordingJobHandler();
    clientRule.getJobClient().newWorker().jobType("test").handler(jobHandler).open();
    brokerRule.stopBroker();

If we remove line 2 and 3 it works.

[menski]

Problem

This is not related to job workers but in general if there is an ongoing request during shutdown. The reason for this is the order in which we shutdown our components:

1. Service Container close, which closes all stream processors, i.e. no requests are processed and answered anymore
   https://github.com/zeebe-io/zeebe/blob/1f7f5103b71c32904a8d87ff49828f6889b4ce2b/broker-core/src/main/java/io/zeebe/broker/system/SystemContext.java#L218
2. Close the gateway
   1. First we close the broker client, which drops all broker requests without failing them
      https://github.com/zeebe-io/zeebe/blob/1f7f5103b71c32904a8d87ff49828f6889b4ce2b/gateway/src/main/java/io/zeebe/gateway/Gateway.java#L109
   2. Shutdown the gRPC server and await its termination
      https://github.com/zeebe-io/zeebe/blob/1f7f5103b71c32904a8d87ff49828f6889b4ce2b/gateway/src/main/java/io/zeebe/gateway/Gateway.java#L114-L116

The server.shutdown() should ensure a graceful shutdown with allowing ongoing requests to finish. This will never terminate if there where open requests in our scenario as we already closed the stream processors. And the requests will also never fail as we already closed the broker client.

Possible solution

No matter what solution we choose we have to close the gRPC server before closing the broker client in the gateway. As the client is required to complete the ongoing requests. This will at least ensure that requests timeout and the server can terminate, even if the stream processors are already shutdown.
To allow graceful shutdowns of the embedded gateway we have to make sure to close the gateway as first component before stopping anything else, so we ensure that all in flight requests are completed before shutting down the internal processing.

As a quick dirty workaround we can use server.shutdownNow() to also terminate ongoing gRPC requests.

[Zelldon]

There are still some tests ignored