feat: Improve shutdown Delay default (#785)

* feat: Improve shutdown Delay default

When a baseplate server is shutting down in k8s

* baseplate calls func (srv *Server) Shutdown(ctx context.Context) error which does the following:
* Shutdown works by first closing all open listeners

* k8s is attempting to remove the endpoint for the endpoints group and thus make (new) incoming traffic to the pod 0

This is a race condition because some callers are still trying to connect. We have empirically seen that the 5 second default is good enough for most services.

See also reddit/baseplate.go#615

Signed-off-by: Sotiris Nanopoulos <sotiris.nanopoulos@reddit.com>

* improve comment

Signed-off-by: Sotiris Nanopoulos <sotiris.nanopoulos@reddit.com>

---------

Signed-off-by: Sotiris Nanopoulos <sotiris.nanopoulos@reddit.com>

baseplate/server/__init__.py

@@ -298,9 +298,14 @@ def load_app_and_run_server() -> None:
         SERVER_STATE.state = ServerLifecycle.SHUTTING_DOWN
 
         cfg = parse_config(config.server, {"drain_time": OptionalConfig(Timespan)})
+        # Default drain time across all baseplate language implementations
+        # which allows enough time for systems such as k8s to remove the
+        # server from the endpoints list.
+        drain_time_seconds = 5
         if cfg.drain_time:
-            logger.debug("Draining inbound requests...")
-            time.sleep(cfg.drain_time.total_seconds())
+            drain_time_seconds = cfg.drain_time.total_seconds()
+        logger.debug("Draining inbound requests...")
+        time.sleep(drain_time_seconds)
     finally:
         logger.debug("Gracefully shutting down...")
         server.stop()