Warmup problem when using workers #1757
Labels
Comments
|
I can reproduce this with your test app with just one worker and 2 threads. Send the first request to It turns out this is not Puma's fault, but Rails and boy have I learned a lot 😂. In development you've got the following config: config.cache_classes = false
# Do not eager load code on boot.
config.eager_load = falseBy my rudimentary understanding the very first request is going to attempt to load a bunch of code and to do so will take a lock. It's supposed to release the lock when it's done but it's going to When the second request arrives it believes it needs the lock too, some state that is supposed to be set by the first request isn't set yet. But now the second request's thread is blocked and will wait until the first request's thread completes (or dies) and gives up the lock so it can continue. If you change the order of the requests and visit Knowing this, you can even trigger this blocking behaviour on an already warm server: simply change a class (e.g. increase the I'm not an expert on code unloading/reloading or the relatively new stuff Rails does to sychronise these operations. The Rails Guide on Threading and Code Execution is a decent start place for it and by adding the suggested If you want this to work as expected then you should disable code reloading by setting both the above config variables to |
|
@dannyfallon coming in with the incredible detective work again. Just wondering, does this still lock for quite as long on Rails 6, given that code loading has been completely redone by @fxn? |
|
This happens on Rails 6.0.0.rc1 too 😞 I nerdsniped @eugeneius who helped me a little with this earlier today. It doesn't happen if you do not operate in clustered mode. It doesn't happen if you swap the file watcher from Thanks to rails/rails#25302 if you As you can see from the |
|
Not familiar with the problem, but let me just confirm that the classic autoloader takes an exclusive lock while it is loading code, and another one when it is reloading. Each request must acquire the lock for reading. The new autoloader does not need the one for loading. You still need the one when reloading though (but I guess there is no reloading going on here anyway). |
|
Considering this isn't a puma issue, closing. But great work tracking this down Danny! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Steps to reproduce
rails new /Users/xxx/Development/puma-workers-warmup-bug --api --skiprbenv install && gem install bundler -v 1.17.2 && rbenv rehash && bundle && rbenv rehashrails spuma.rbconf:workers 2andthreads 5, 5. This should allow to process up to 10 (2*5) requests concurrently.curl http://localhost:3000/slow &. This will query in backgroundhttp://localhost:3000/slow, which is basically a controller returning a small JSON but with asleep 10to simulate a (very) long request.curl http://localhost:3000/fast. This will queryhttp://localhost:3000/fast, which is basically the same controller as/slowbut without thesleep 10.Expected behavior
All three requests should be processed concurrently because there are 2 workers with 5 threads each, so up to 10 requests should be able to be processed concurrently.
In other words, when running
curl http://localhost:3000/fast, it should return the response instantly.Actual behavior
The two first requests (
/slow, which take 10 seconds to complete) are processed concurrently but the third one (/fast) hangs and is processed only after one of the first two requests has been processed.In other words, when running
curl http://localhost:3000/fast, it will wait about 10 seconds (the time for at least one of the two first requests to complete) before returning.Note that, from my tests and understanding, this behaviour is only present when the workers have not received
NB_WORKERSrequests yet.Indeed, if you re-run the 3 requests above (2 slow, 1 fast) , you will see that they are processed concurrently and that the fast one returns instantly as expected. However, if you kill the workers or restart them, the problem occurs again.
System configuration
Ruby version: 2.5.1
Rails version: 5.2.3
Puma version: 3.11
The text was updated successfully, but these errors were encountered: