Connection Timeout - Waited for item but none was pushed. #223
Comments
|
same here. |
|
Getting these errors too. Here is more information: We debugged this deep and found that the the This "hang" originates from # moped/node.rb
def command(database, cmd, options = {})
read(Protocol::Command.new(database, cmd, options))
end |
|
increasing the pool_size in my case helped to fix it temporarily |
|
@manusajith @madwire are you guys having this issue on sidekiq too? or that is not related to sidekiq at all? |
|
@arthurnn i am not using sidekiq in my project |
|
This is NOT related to sidekiq. I see these problems in my nginx + passenger setup too. It's easily reproducible on sidekiq because of a lot of workers. |
|
@arthurnn @manusajith Does the pool_size option really have any effect in mongoid.yml? staging:
sessions:
default:
database: app_staging
hosts:
- localhost:27017
options:
consistency: :strong
options:
identity_map_enabled: true
raise_not_found_error: false
pool_size: 35This does not seem to have any effect at all .. did I set it wrong? I have not seen any documentation about setting pool_size. |
|
@arthurnn no sorry Im not using sidekiq either |
|
@gautamrege think its like this |
|
@madwire An Aha! moment. Thanks 👍 This is undocumented. |
|
INDEED -- the problem reduces as I increase the pool_size. |
|
I'm having this issue as well. My app is a Rack application running in Puma 2.6 using Grape 0.6. The error is raised in Jruby 1.7.5 and Ruby 2.0.0. The first few requests run fine but errors begin when ApacheBench testing more than 5 or so concurrent requests. Upping the pool_size to 20 doesn't help. |
|
This seems to be a bug somewhere on moped connection pool.. I will take a look into that this weekend, for the time being you guys can point mongoid to mongoid/mongoid@f91feef , which should not have this bug. thanks |
|
Can you guys try to increase the sessions:
default:
options:
pool_timeout: 30And let me know if it works or not. |
|
Hello @arthurnn Increasing the pool_timeout isn't helping. Here is the top of the stacktrace (from jruby but I get the same in mri): |
|
@arthurnn dint help me either. |
|
As I cannot simulate the error, could you guys try this attempt of fixing the error? https://github.com/mongoid/moped/tree/conn_pool |
|
Sure! Already on it. On 14-Oct-2013, at 11:34 AM, Arthur Nogueira Neves notifications@github.com wrote:
|
|
Seems like the reap method is only called for dead threads, and not on sleeping threads too. and was able to return connection back to the queue. Dunno its the right way to do it. |
|
@manusajith I see that. Wondering how would you have sleep threads if you are not running on a concurrent environment, thats what tricks me.. thanks |
|
@arthurnn It looks like sleep threads to me. Still dont get why, as Im running on webrick and ruby mri |
|
@manusajith Why do you have so many sleep threads? The implementation of moped connection_pool is to pull one connection from the pool for each thread. So if you have threads pulling the connection, going to sleep without releasing them, that might be an application issue. |
|
@arthurnn I cant see how it can be app related for me, Im not doing anything with threads. Its a very generic app running on MRI and WEBrick in development. Here is my output again: |
|
Same here. |
|
Same issue here Any workaround/patch in the meantime? Thanks! |
|
We have the issue too, had to raise pool_size. |
|
@ralberto @thibaudgg @m-barthelemy is everyone having sleep threads ? |
|
Yes. |
|
for me the error disappeared since i've increased my pool_size + pool_timeout. but i think the root cause, in my case, was an 'app issue': i was updating embedded documents within an after_validation callback. i havent met this error since i changed that to after_save |
|
I got same error. |
|
WEBBrick on default MRI, spins a new thread to handle every request concurrently: https://github.com/ruby/ruby/blob/trunk/lib/webrick/server.rb#L175, and if the request has a keep-alive header the socket will be open until there is a timeout.(https://github.com/ruby/ruby/blob/trunk/lib/webrick/httpserver.rb#L115) So if you have a connection_pool sized to 5 and 5+ connections happening within the timeframe of the timeout you`ll get the problem, because threads would be holding the connection without using it. For sidekiq we need as many connection as the concurrency number, thats by design. I will discuss the design of the thread pool with @durran and we should try to fix this without having to add the middleware. |
|
@arthurnn thanks for the update, that explains a lot. |
|
Yea, I'm getting the same issue with mongoid-head, rails 4 and puma |
|
@pkieltyka if your using puma then all you have todo is increase your pool_size (default 10) |
ghost
assigned
|
Thanks Ill give that a shot On 2013-10-28, at 4:42 PM, Richard Adams notifications@github.com wrote:
|
|
branch https://github.com/mongoid/moped/tree/conn_pool_checkin should fix the problem. Could you guys try that out and let me know. Using that branch, the middleware I wrote is not necessary anymore. Please let me know if it works, so I can merge it to master. |
|
I'm still experiencing the same error. I haven't tweaked connection pool/tmout though. |
|
@blackxored have u tried to use this branch ? |
|
can confirm that a pool size increase did eliminate my Timeout error in sidekiq |
|
@Arthur: nevermind, it does fix it I just somehow assumed it was merged on master, realized that wasn't two minutes from my reply. ;) Best Regards, Sent from my phone. On Wed, Oct 30, 2013 at 5:54 AM, scotthelm notifications@github.com
|
|
Branch working good here too. Thanks! |
|
@arthurnn The conn_pool_checkin branch seems to have solved the issue for us. |
|
@arthurnn same here, all is well :) |
|
Could you please merge this to master? |
|
I'm also getting the same issue with mongoid-head, rails 4 and puma - could you please merge this to master? |
|
We seem to be getting this exact same issue. Rails 4 + Sidekiq 3 + Mongoid 4. |
|
Same here.... any idea since we can't set the pool_size? |
|
I do believe I still have this error. Increasing pool size did not work for me. |
|
Heh. And I'm back here a year later. Surprised to see my own comment above. Note: increasing pool_size in mongoid worked for me. |
Hi
After updating mongoid and moped (2.0.0.beta2), Im occasionally getting this:
Im running in development mode with WEBrick so dont understand why its running out of connections
Versions:
The text was updated successfully, but these errors were encountered: