New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use resolv-replace #1258

Closed
mperham opened this Issue Oct 17, 2013 · 11 comments

Comments

6 participants
@mperham
Copy link
Owner

mperham commented Oct 17, 2013

Sidekiq should pull in resolv-replace so we don't risk locking up MRI.

Relevant blog post.

@halorgium

This comment has been minimized.

Copy link
Contributor

halorgium commented Oct 17, 2013

@mperham do you think this is something which celluloid could also benefit from?
We do actually have an async DNS in celluloid-io, but not everyone will be using that.

@mperham

This comment has been minimized.

Copy link
Owner

mperham commented Oct 17, 2013

I'm not sure. That blog post is ancient so it's unclear if this is still a problem with MRI. The GIL is still there so it's possible that calls into libc's DNS resolution don't release the GIL. I don't want to cargo cult so it'd be nice to have someone knowledgable chime in.

On Oct 16, 2013, at 19:51, Tim Carey-Smith notifications@github.com wrote:

@mperham do you think this is something which celluloid could also benefit from?
We do actually have an async DNS in celluloid-io, but not everyone will be using that.


Reply to this email directly or view it on GitHub.

@mperham

This comment has been minimized.

Copy link
Owner

mperham commented Oct 17, 2013

This comment implies that it's still a problem with MRI resque/resque#1101 (comment)

@mperham

This comment has been minimized.

Copy link
Owner

mperham commented Oct 17, 2013

@tmm1 signed off on it. Good enough for me. efa3915

@mperham mperham closed this Oct 17, 2013

@RaVbaker

This comment has been minimized.

Copy link

RaVbaker commented Oct 30, 2013

I'm using google-adwords-api gem in my workers and since upgrade from previous version (2.11.2) to 2.16.0 I'm getting errors:

ArgumentError: cannot interpret as DNS name: nil

Any idea why? When I downgraded to 2.15.2 everything is working and for now I will stay with it. But I think that you might want to know.

@mperham

This comment has been minimized.

Copy link
Owner

mperham commented Oct 30, 2013

It's hard to diagnose without a backtrace.

@RaVbaker

This comment has been minimized.

Copy link

RaVbaker commented Oct 30, 2013

Backtrace returned by sidekiq in gist.

@tmm1

This comment has been minimized.

Copy link

tmm1 commented Oct 30, 2013

FWIW, I was probably wrong about needing resolv-replace. Looks like rb_getaddrinfo() does resolution outside the GVL in ext/socket/raddrinfo.c.

@thibaudgg

This comment has been minimized.

Copy link
Contributor

thibaudgg commented Oct 31, 2013

Got same issue ArgumentError: cannot interpret as DNS name: nil with Mongoid/Moped, backtrace:

[PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/resolv.rb:1176:in `create`
[PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/resolv.rb:1022:in `generate_candidates`
[PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/resolv.rb:1047:in `resolv`
[PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/resolv.rb:513:in `each_resource`
[PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/resolv.rb:406:in `each_address`
[PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/resolv.rb:115:in `block in each_address`
[PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/resolv.rb:114:in `each`
[PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/resolv.rb:114:in `each_address`
[PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/resolv.rb:92:in `getaddress`
[PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/resolv.rb:43:in `getaddress`
[PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/resolv-replace.rb:10:in `getaddress`
[PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/resolv-replace.rb:23:in `initialize`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/connection/socket/tcp.rb:20:in `block in initialize`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/connection/socket/connectable.rb:119:in `handle_socket_errors`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/connection/socket/tcp.rb:20:in `initialize`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/connection/socket/connectable.rb:153:in `new`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/connection/socket/connectable.rb:153:in `block in connect`
[PROJECT_ROOT]/vendor/ruby-2.0.0/lib/ruby/2.0.0/timeout.rb:66:in `timeout`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/connection/socket/connectable.rb:152:in `connect`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/connection.rb:56:in `connect`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/node.rb:102:in `block in connect`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/node.rb:130:in `block in connection`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/connection/pool.rb:160:in `with_connection`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/node.rb:129:in `connection`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/node.rb:102:in `connect`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/node.rb:191:in `block in ensure_connected`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/executable.rb:25:in `execute`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/node.rb:189:in `ensure_connected`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/node.rb:577:in `block in flush`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/node.rb:608:in `block in logging`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.0/lib/active_support/notifications.rb:159:in `block in instrument`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.0/lib/active_support/notifications/instrumenter.rb:20:in `instrument`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/activesupport-4.0.0/lib/active_support/notifications.rb:159:in `instrument`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/instrumentable.rb:31:in `instrument`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/node.rb:607:in `logging`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/node.rb:576:in `flush`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/node.rb:398:in `process`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/operation/read.rb:48:in `execute`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/node.rb:639:in `read`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/node.rb:87:in `command`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/node.rb:439:in `refresh`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/cluster.rb:166:in `block in refresh`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/cluster.rb:335:in `call`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/cluster.rb:335:in `block in refresh_peers`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/cluster.rb:334:in `each`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/cluster.rb:334:in `refresh_peers`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/cluster.rb:171:in `block in refresh`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/cluster.rb:178:in `each`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/cluster.rb:178:in `refresh`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/cluster.rb:135:in `nodes`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/cluster.rb:224:in `with_primary`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/query.rb:379:in `update`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/moped-2.0.0.beta3/lib/moped/query.rb:420:in `upsert`
[PROJECT_ROOT]/app/models/site_admin_stat.rb:26:in `upsert_stats`
[PROJECT_ROOT]/app/services/site_admin_stat_upserter.rb:16:in `upsert`
[PROJECT_ROOT]/app/services/site_admin_stat_upserter.rb:12:in `upsert`
[PROJECT_ROOT]/app/workers/stats_handler_base_worker.rb:21:in `_handle_al_event`
[PROJECT_ROOT]/app/workers/stats_handler_base_worker.rb:14:in `perform`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/processor.rb:48:in `block (3 levels) in process`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/middleware/chain.rb:119:in `call`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/middleware/chain.rb:119:in `block in invoke`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/librato-sidekiq-0.1.2/lib/librato-sidekiq/middleware.rb:49:in `call`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/middleware/chain.rb:121:in `block in invoke`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm-3.6.8.168/lib/new_relic/agent/instrumentation/sidekiq.rb:25:in `block in call`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm-3.6.8.168/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:324:in `perform_action_with_newrelic_trace`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/newrelic_rpm-3.6.8.168/lib/new_relic/agent/instrumentation/sidekiq.rb:21:in `call`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/middleware/chain.rb:121:in `block in invoke`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/middleware/server/active_record.rb:6:in `call`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/middleware/chain.rb:121:in `block in invoke`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/middleware/server/retry_jobs.rb:62:in `call`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/middleware/chain.rb:121:in `block in invoke`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/middleware/server/logging.rb:11:in `block in call`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/logging.rb:22:in `with_context`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/middleware/server/logging.rb:7:in `call`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/middleware/chain.rb:121:in `block in invoke`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/middleware/chain.rb:124:in `call`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/middleware/chain.rb:124:in `invoke`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/processor.rb:47:in `block (2 levels) in process`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/processor.rb:102:in `stats`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/processor.rb:46:in `block in process`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/processor.rb:83:in `do_defer`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/sidekiq-2.16.0/lib/sidekiq/processor.rb:37:in `process`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `public_send`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:25:in `dispatch`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/calls.rb:122:in `dispatch`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:322:in `block in handle_message`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/actor.rb:416:in `block in task`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/tasks.rb:55:in `block in initialize`
[PROJECT_ROOT]/vendor/bundle/ruby/2.0.0/gems/celluloid-0.15.2/lib/celluloid/tasks/task_fiber.rb:13:in `block in create`
@aackerman

This comment has been minimized.

Copy link
Contributor

aackerman commented Oct 31, 2013

With the release of Sidekiq 2.16.1, the changes to resolve-replace have been reverted. If you are able to upgrade it should solve your issue.

@thibaudgg

This comment has been minimized.

Copy link
Contributor

thibaudgg commented Oct 31, 2013

Yep done, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment