Graceful shutdown not working with keep-alive

[jbransen]

I encountered unexpected behaviour with the graceful shutdown method. I have the following code:

{-# LANGUAGE OverloadedStrings #-}

import Control.Monad

import Data.Time

import Network.Wai (responseLBS, Application)
import Network.Wai.Handler.Warp
import Network.HTTP.Types (status200)
import Network.HTTP.Types.Header (hContentType)

import System.Signal

logM :: String -> IO ()
logM s = do
  t <- getCurrentTime
  putStrLn $ show t ++ ": " ++ s

main = do
    let port = 3000
    logM $ "Listening on port " ++ show port
    let shutdownHandler closeSocket =
          installHandler sigINT $ const (logM "Closing webserver" >> closeSocket >> logM "Socket closed")
    let settings =
          setPort port $
          setGracefulShutdownTimeout (Just 5) $
          setInstallShutdownHandler shutdownHandler $
          defaultSettings
    runSettings settings app
    logM "Server exited"

app :: Application
app req f =
    f $ responseLBS status200 [(hContentType, "text/plain")] "Hello world!"

Now when I run this and send a sigint, the program will shut down properly. However, when I request localhost:3000 in my browser, and then ask the server to shutdown, it will always take up the full grace period:

2021-07-13 07:11:06.6979399 UTC: Listening on port 3000
^C2021-07-13 07:11:16.0393217 UTC: Closing webserver
2021-07-13 07:11:16.0398661 UTC: Socket closed
2021-07-13 07:11:21.0430659 UTC: Server exited

However, this is unexpected, as there were no request coming in at this moment at all, so the server could have shut down immediately. As this does not happen when I make requests in other ways, I assume it is related to the fact that the browser sends a Connection: keep-alive header and keeps the connection alive. This alive connection probably stops the graceful shutdown from terminating, even though no requests are being made.

Is this by design, or can this be solved? With this behaviour I think the graceful shutdown has little value, while it could be a quite useful tool.

[robx]

Yup, this looks like a bug. Worse than just not shutting down, requests keep being handled after shutdown (though that's not entirely unexpected all things considered):

2021-11-08 17:46:54.975498 UTC: Listening on port 3000
2021-11-08 17:47:04.407312 UTC: Handling request
^C2021-11-08 17:47:08.194051 UTC: Closing webserver
2021-11-08 17:47:08.194347 UTC: Socket closed
2021-11-08 17:47:10.461856 UTC: Handling request
2021-11-08 17:47:13.197359 UTC: Server exited

while I was talking to the server as follows:

$ nc localhost 3000
GET / HTTP/1.1
connection: keep-alive

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Mon, 08 Nov 2021 17:47:04 GMT
Server: Warp/3.3.17
Content-Type: text/plain

000C
Hello world!
0

GET / HTTP/1.1
connection: keep-alive

HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Mon, 08 Nov 2021 17:47:10 GMT
Server: Warp/3.3.17
Content-Type: text/plain

000C
Hello world!
0

(Sequence was 1. cabal run 2. nc and enter the first http request 3. hit Ctrl-C on the server 4. enter second http request.)

[swamp-agr]

1. Combination of connection: keep-alive and graceful shutdown has not been specified explicitly.
2. According to RFC 2616 section 8.1.4:

When a client or server wishes to time-out it SHOULD issue a graceful
close on the transport connection. Clients and servers SHOULD both
constantly watch for the other side of the transport close, and
respond to it as appropriate. If a client or server does not detect
the other side's close promptly it could cause unnecessary resource
drain on the network.

A client, server, or proxy MAY close the transport connection at any
time. For example, a client might have started to send a new request
at the same time that the server has decided to close the "idle"
connection. From the server's point of view, the connection is being
closed while it was idle, but from the client's point of view, a
request is in progress.

This means that clients, servers, and proxies MUST be able to recover
from asynchronous close events. Client software SHOULD reopen the
transport connection and retransmit the aborted sequence of requests
without user interaction so long as the request sequence is
idempotent (see section 9.1.2). Non-idempotent methods or sequences
MUST NOT be automatically retried, although user agents MAY offer a
human operator the choice of retrying the request(s). Confirmation by
user-agent software with semantic understanding of the application
MAY substitute for user confirmation. The automatic retry SHOULD NOT
be repeated if the second sequence of requests fails.

3. In both examples server waits for 5s graceful shutdown timeout and closes underlying connection.

Thus, I think this behaviour must be explicitly specified as not a bug. Moreover If you check different web servers, then you'll find that it's implemented in a similar way.

[jbransen]

@swamp-agr That is not true, Nginx does close the connection as soon as a response has been sent, e.g. see https://trac.nginx.org/nginx/ticket/1022 and also this is easy to test. Start it, then with nc send a request and keep the connection open, then in another terminal do nginx -s quit (graceful shutdown) and see that the nc tcp connection is dropped immediately and no further requests are served. This in contrast with wai, which does continue to serve requests while the idle connection could have been closed already.