-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
not cleanly closing ssl sockets on phased_restart? #1002
Comments
If it's the same issue as #957, you can configured the persistent_timeout now. Please try that and if it doesn't work, reopen this issue. |
Thanks @evanphx, but adjusting timeouts doesn't affect this. This error occurs without going through the ELB. |
Hm, ok. I wonder if perhaps on a worker exit persistent sockets aren't being closed. That's not an issue for normal HTTP sockets because the process exit will close the socket, but would be an issue for SSL sockets because the close actually sends data back across telling the other side that it's closed. I'll investigate. |
Nope, |
I've reproduced the error with just a puma config file (no rack app) here: https://github.com/mattyb/puma-socket-test |
Thanks so much for the reproduction! I'm working on it now. |
@mattyb That should fix your problem. Do you have a way to try it out easily before I release the fix? |
Thanks for working on this! I no longer see the SSL errors popping up in the console and there are new messages in the verbose logs about connections being closed. Unfortunately, even with the reproduction code updated to the 46416cb commit, I still see failed requests from
It's possible that this is a problem with I could try the new commit on our real app in our staging environment and see if the ELB handles whatever error is happening more cleanly, but it seems like something may still be awry in puma. |
The problem was a few points: * We were not clearing the reactor on a normal stop, which is what is used in a phased restart. * On close, SSL sockets were not sending the shutdown message. * SSL sockets that were completely uninitialized ended up sitting in reactor and could not actually be shutdown because there were not initialized.
We're running apache bench against a running puma server bound to ssl sockets. When we perform a puma phased restart, we get a batch of SSL errors in apache bench each time a worker goes down. With verbose ab flags, a healthy connection includes messages like
Where a failed connection looks like
I believe that this is the reason that our AWS ELB returns 504 errors each time we perform a phased restart. cc @timabdulla, who reported similar symptoms in #957
The text was updated successfully, but these errors were encountered: