accept(2) error handling #553
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There is no guarantee that running out of file-descriptors is a temporary condition, and simply sleeping for a second is a naive solution. In the general case, it is better to scream loudly and alert an administrator to this case.
ECONNABORTED is a recoverable error, and only applies to a particular socket in the accept(2) queue. It is safe to simply try accept'ing the next socket on the queue.
For servers with long-lived (or infinite lived) connections, waiting for graceful shutdown can block the server indefinitely. A new setting is added, which allows for a time (in seconds) to be applied when commencing a graceful shutdown.
|
The failure is a timeout on the |
| @@ -347,6 +353,13 @@ setLogger :: (Request -> H.Status -> Maybe Integer -> IO ()) | |||
| -> Settings -> Settings | |||
| setLogger lgr y = y { settingsLogger = lgr } | |||
|
|
|||
| -- | Set the graceful shutdown timeout. A timeout of `Nothing' will | |||
| -- wait indefinitely, and a number, if provided, will be treated as seconds | |||
| -- to wait for requests to finish, before shutting down the server entirely. | |||
There was a problem hiding this comment.
It'd be good to add -- @since x.x.x or -- Since x.x.x like the other APIs have. I think the next version would be 3.2.8 (Current is 3.2.7, and this PR adds new APIs. Not sure if the ResourceExhausted thing would be considered a breaking change, though)
There was a problem hiding this comment.
Added in 504fa37.
|
I've force-pushed fce3103 to address the formatting issue in the documentation. |
|
Looks fine to me. @kazu-yamamoto do you want to review before merge? |
|
I support this changes. |
|
Thanks! |
|
Warp 3.2.7 has been released. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains three commits which deal with an issue we've been having with our production Warp services. Currently, Warp treats all exceptions that can be raised by
acceptas unrecoverable (other than its special handling ofResourceExhausted, which I've removed for reasons described in 3011cad). However, some errors from theacceptsyscall are recoverable, and do not imply the entire listening socket is closed or in a bad state. Specifically, accept(2) and accept4(2) will return ECONNABORTED if a connection request is aborted between entering the socket's listen queue, and actually being accepted. In this case, a subsequent call toacceptwill simply return the next socket on the queue.Since this was treated as a fatal error before, when this would happen to us in production, the webserver would begin its graceful shutdown routine, but since we use long-lived connections, graceful shutdown would wait indefinitely for HTTP requests that will never finish (we leave an HTTP connection open and stream data). This manifested itself as the entire application hanging indefinitely. In aebd8d4, I've added an option to the Warp
Settingsfor an optional graceful shutdown timeout, so that if there is a fatal error raised byaccept, the webserver won't simply hang forever, waiting to shut down.I've kept this in three commits, so that it's easier to review, and if you only want some of this functionality, it can be cherry-picked.