New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request Queue Size #1424

Closed
Vikaton opened this Issue Mar 11, 2015 · 13 comments

Comments

Projects
None yet
7 participants
@Vikaton

Vikaton commented Mar 11, 2015

[ 2015-03-11 14:06:32.7761 3690/7f74440c9700 agents/HelperAgent/RequestHandler/CheckoutSession.cpp:236 ]: [Client 1-437] Returning HTTP 503 due to:
Request queue full (configured max. size: 100)

^ thats the error I get benchmarking my Ruby website(500 concurrent), but only the first time, if I benchmark again after the error, its fine.

How can I set the configured max size?

@dmitry

This comment has been minimized.

Show comment
Hide comment
@dmitry

dmitry Mar 11, 2015

@Ap0ph1s you can setup the request queue value through this option (in nginx): https://www.phusionpassenger.com/documentation/Users%20guide%20Nginx.html#passenger_max_request_queue_size

I believe, first time your application was on cold start, and it took just more time to access first requests, that's why many requests was failed and queue was full.

dmitry commented Mar 11, 2015

@Ap0ph1s you can setup the request queue value through this option (in nginx): https://www.phusionpassenger.com/documentation/Users%20guide%20Nginx.html#passenger_max_request_queue_size

I believe, first time your application was on cold start, and it took just more time to access first requests, that's why many requests was failed and queue was full.

@Vikaton

This comment has been minimized.

Show comment
Hide comment
@Vikaton

Vikaton Mar 11, 2015

Thanks, is the setup with nginx any different from standalone?

Vikaton commented Mar 11, 2015

Thanks, is the setup with nginx any different from standalone?

@felixbuenemann

This comment has been minimized.

Show comment
Hide comment
@felixbuenemann

felixbuenemann Mar 11, 2015

You should use passenger-status and passenger-status --show=requests to see what's going on. It's very unlikely that you have 100 outstanding requests per worker unless your workers are stuck and are not accepting new requests. Also check your webserver's error log to see if there are exceptions.

felixbuenemann commented Mar 11, 2015

You should use passenger-status and passenger-status --show=requests to see what's going on. It's very unlikely that you have 100 outstanding requests per worker unless your workers are stuck and are not accepting new requests. Also check your webserver's error log to see if there are exceptions.

@FooBarWidget

This comment has been minimized.

Show comment
Hide comment
@FooBarWidget

FooBarWidget Mar 12, 2015

Member

Hi @Ap0ph1s, thanks for reporting this. Version 5.0.4 has been released. I believe the bug you're reporting has been fixed, but there's also a chance that the issue you're running into is a different one. Could you give 5.0.4 a try? And if you still run into the problem, could you gather some analysis data as @felixbuenemann described?

Here's a more detailed description of the steps involved:

  1. Please save the output of passenger-status --show=requests.
  2. Please save the output of passenger-status (without the --show=requests part).
  3. If in the passenger-status output you saw one or more processes with a session count larger than 0, please send SIGQUIT to one of those processes, and save the backtrace that has been dumped to the web server error log.

If it's not fixed in 5.0.4 we'll do our best to fix it for 5.0.5.

There's also a chance that this is not a bug in Passenger, but rather an issue in the application. The above analysis gathering steps will also tell you whether it's Passenger at fault or the app.

Member

FooBarWidget commented Mar 12, 2015

Hi @Ap0ph1s, thanks for reporting this. Version 5.0.4 has been released. I believe the bug you're reporting has been fixed, but there's also a chance that the issue you're running into is a different one. Could you give 5.0.4 a try? And if you still run into the problem, could you gather some analysis data as @felixbuenemann described?

Here's a more detailed description of the steps involved:

  1. Please save the output of passenger-status --show=requests.
  2. Please save the output of passenger-status (without the --show=requests part).
  3. If in the passenger-status output you saw one or more processes with a session count larger than 0, please send SIGQUIT to one of those processes, and save the backtrace that has been dumped to the web server error log.

If it's not fixed in 5.0.4 we'll do our best to fix it for 5.0.5.

There's also a chance that this is not a bug in Passenger, but rather an issue in the application. The above analysis gathering steps will also tell you whether it's Passenger at fault or the app.

@FooBarWidget FooBarWidget added this to the 5.0.5 milestone Mar 12, 2015

@Vikaton

This comment has been minimized.

Show comment
Hide comment
@Vikaton

Vikaton Mar 12, 2015

Thanks, will do! Also, Using Puma to benchmark my servers, I dont have the initial issue.

Vikaton commented Mar 12, 2015

Thanks, will do! Also, Using Puma to benchmark my servers, I dont have the initial issue.

@FooBarWidget

This comment has been minimized.

Show comment
Hide comment
@FooBarWidget

FooBarWidget Mar 16, 2015

Member

Hi @Ap0ph1s, has the bug been fixed? Can I close this issue now?

Member

FooBarWidget commented Mar 16, 2015

Hi @Ap0ph1s, has the bug been fixed? Can I close this issue now?

@Vikaton

This comment has been minimized.

Show comment
Hide comment
@Vikaton

Vikaton Mar 16, 2015

Been trying to get to it, very soon though

Vikaton commented Mar 16, 2015

Been trying to get to it, very soon though

@Vikaton

This comment has been minimized.

Show comment
Hide comment
@Vikaton

Vikaton commented Mar 17, 2015

Resolved

@Vikaton Vikaton closed this Mar 17, 2015

@paambaati

This comment has been minimized.

Show comment
Hide comment
@paambaati

paambaati Jun 4, 2015

@FooBarWidget I have a similar issue on the standalone version of Passenger 5.0.8, where my Node.js + Express application, when being benchmarked, throws up a lot of 503 errors with the following in the logs -

Returning HTTP 503 due to: Request queue full (configured max. size: 100)

How do I change this parameter in the standalone version? Documentation only includes nginx directives.

paambaati commented Jun 4, 2015

@FooBarWidget I have a similar issue on the standalone version of Passenger 5.0.8, where my Node.js + Express application, when being benchmarked, throws up a lot of 503 errors with the following in the logs -

Returning HTTP 503 due to: Request queue full (configured max. size: 100)

How do I change this parameter in the standalone version? Documentation only includes nginx directives.

@FooBarWidget

This comment has been minimized.

Show comment
Hide comment
Member

FooBarWidget commented Jun 4, 2015

@coin8086

This comment has been minimized.

Show comment
Hide comment
@coin8086

coin8086 Nov 4, 2015

@FooBarWidget I also ran into the problem when I was benchmarking the standalone passenger 5.0.21. I checked the reference you pointed, and I found I need

passenger_max_request_queue_size 1000;

However there's no easy way for standalone passenger to apply that except using the advanced --nginx-config-template option and providing a custom nginx config, in which the above line can be put.

Well, that's it. BTW, I found the the standalone passenger does worse in performance(with passenger_max_request_queue_size 1000 and 20 processes preallocated in a Ubuntu 14.04 with 2 CUP cores and 4GB memory) compared to Apache 2.4 + Passenger Mod. I use a Sinatra app to test them. The standalone version is no more than a half of the Apache version, which is even not optimized yet! So I decide to turn to Apache version.

coin8086 commented Nov 4, 2015

@FooBarWidget I also ran into the problem when I was benchmarking the standalone passenger 5.0.21. I checked the reference you pointed, and I found I need

passenger_max_request_queue_size 1000;

However there's no easy way for standalone passenger to apply that except using the advanced --nginx-config-template option and providing a custom nginx config, in which the above line can be put.

Well, that's it. BTW, I found the the standalone passenger does worse in performance(with passenger_max_request_queue_size 1000 and 20 processes preallocated in a Ubuntu 14.04 with 2 CUP cores and 4GB memory) compared to Apache 2.4 + Passenger Mod. I use a Sinatra app to test them. The standalone version is no more than a half of the Apache version, which is even not optimized yet! So I decide to turn to Apache version.

@dmitry

This comment has been minimized.

Show comment
Hide comment
@dmitry

dmitry Nov 4, 2015

@coin8086 why not to use nginx + passenger?

dmitry commented Nov 4, 2015

@coin8086 why not to use nginx + passenger?

@OnixGH

This comment has been minimized.

Show comment
Hide comment
@OnixGH

OnixGH Nov 4, 2015

Contributor

@dmitry I'm not sure the setup of @coin8086 is a production server setup? Also no idea if the benching is done from a different system to avoid skewing the results.

It would be interesting to see if Apache performance improved vs. Nginx in the latest versions though, so let us know if you decide to do any benching :)

Contributor

OnixGH commented Nov 4, 2015

@dmitry I'm not sure the setup of @coin8086 is a production server setup? Also no idea if the benching is done from a different system to avoid skewing the results.

It would be interesting to see if Apache performance improved vs. Nginx in the latest versions though, so let us know if you decide to do any benching :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment