improve http graceful termination #4957
Conversation
| @@ -156,24 +163,50 @@ trait BasicHttpService extends Directives { | |||
| } | |||
|
|
|||
| object BasicHttpService { | |||
| implicit val tid = TransactionId(systemPrefix + "http_service") | |||
| //start with ready true | |||
| protected[http] var ready = true | |||
There was a problem hiding this comment.
The ready var? It is used to alter the response to /ready endpoint defined in BasicRasService.
I'll try to refactor this so that we're not modifying the object var from different places.
| whisk { | ||
| http { | ||
| shutdown-unready-delay = 15 seconds # /ready will return 500 for this amount of time before connection draining starts |
There was a problem hiding this comment.
It seems it would wait for 15 seconds, and start the termination phase.
But if there are still client connections connected, are they getting connection closed error?
There was a problem hiding this comment.
This only allows an unready stage, which only affects the added /ready endpoint to return 503 (no other endpoints affected). This does not affect how akka http handles existing client connections during shutdown, so if akka allows gracefully terminating in flight requests, then no connection closed errors, but no new connections allowed. The purpose here is to allow external systems (e.g. kubernetes or reverse proxies) to probe the service and stop routing to this instance for some time before akka shutdown initiates.
| if (readyState) { | ||
| complete("ok") | ||
| } else { | ||
| logging.warn(this, "not ready...") |
There was a problem hiding this comment.
can this just be a debug? I think ready route returning 503 should be self explanatory
|
@tysonnorris would you still like to see this pr merged? |
Description
Deploying controller to kubernetes, we see some case where using /ping as livenessProbe and readinessProbe does not provide great resilience when deleting a controller pod while under load. A typical case for controller unexpected shutdown under load is eviction process for cluster autoscaler, or node maintenance.
This PR provides:
Related issue and scope
My changes affect the following components
Types of changes
Checklist: