Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
Pass queued requests to thread pool on server shutdown #2122
Fixes a concurrency bug during server shutdown when
Expected: s2 receives a successful response
Here is the test implementation, currently failing on
The underlying issue is that in a 'graceful' shutdown, the server stops accepting connections and the thread pool stops accepting new work while it waits up to
To fix, this PR makes a small change to shut down the reactor before the thread pool, and to disable
The logic for determining 'idle' clients that can be closed immediately follows guidance from RFC 2616 section 8.1.4:
The added test depends on the small fix in #2121 to pass properly on Windows, so this branch currently includes that commit as well.
Your checklist for this pull request
schneems left a comment •
Thanks a ton for your PR. It was really hard to review, mostly due to the complexity of the subject matter as well as complexity of Puma internals. I can only imagine that it must have taken you much longer to write this PR.
I want to take a step back from the implementation here and talk about some high level questions and goals
Is this a regression?
My high level question is: Did this used to work? Is this a regression or is this something new we're trying to find/fix? It looks like this might be related to #2200 but...I don't see where something changed in the reactor. Did something else get changed somewhere? If so, can we git bisect and understand what broke the behavior?
If you go back to 4.0.0 and run the same test does it pass or fail?
Regardless of if this was broken or not, another question is: What do we want to happen? While digging in the code I found a few situations, and two of them have semi-undefined behavior:
Scenario A) Client 2 is fully buffered
I think we can all agree on this.
Scenario B) Client 2 CAN be (but is not yet) fully buffered
Is this true? Do we want to do that? The downside is we have no way of knowing if the client will ever send the rest of the request in time. How do we detect the difference between scenario B to scenario C where we need to serve need a default response
Scenario C) Client 2 cannot fully buffered
If we pass a partial response to the threadpool for processing, there's no guarantee that it can do anything with it, it might be a half formed json body, that cannot be decoded. While RFC 2616 says we "should" send at least one response, what should we send? If we pass it to the thread pool in this state it's going to be a 500. I think that's not accurate. A 503 would perhaps be more appropriate.
As far as I can tell, this never used to work since the reactor implementation has existed. For example, on bb2aeb8 (first commit where the
I only encountered this issue as an intermittent unit-test failure while working on #2123, and worked backwards from there to create a minimal reproducible failing test and fix. In any case it seems possible that #2200 is caused by this underlying issue, though I can't be sure.
I wasn't sure how Scenario 'A' differed from 'B' (identical except for the 'desired' part?), so I'll just address B and C.
To clarify the behavior across all combinations of scenarios, I added a few extra test cases to this PR. Let me know if these are clear enough and if the expected behavior is reasonable, or if there are any additional scenarios you think should be covered.
Yes, this is what happens when