Skip to content

Signals make sidekiq exit on 1.9.2 #232

Closed
betelgeuse opened this Issue Jun 9, 2012 · 7 comments

3 participants

@betelgeuse
Collaborator
$ bundle exec sidekiq -c 1 -r ./worker.rb 
2012-06-09T15:49:52Z 22745 TID-17x9dc INFO: Booting sidekiq 2.0.1 with Redis at redis://localhost:6379/0
2012-06-09T15:49:52Z 22745 TID-17x9dc INFO: Running in ruby 1.9.2p290 (2011-07-09 revision 32553) [i686-linux]
2012-06-09T15:49:52Z 22745 TID-16w8ee INFO: Starting processing, hit Ctrl-C to stop
Thread TID-16w8ee 
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bundler/gems/sidekiq-222442e993d4/lib/sidekiq/cli.rb:21:in `block (2 levels) in <top (required)>'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bundler/gems/sidekiq-222442e993d4/lib/sidekiq/cli.rb:19:in `each'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bundler/gems/sidekiq-222442e993d4/lib/sidekiq/cli.rb:19:in `block in <top (required)>'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bundler/gems/sidekiq-222442e993d4/lib/sidekiq/cli.rb:73:in `call'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bundler/gems/sidekiq-222442e993d4/lib/sidekiq/cli.rb:73:in `sleep'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bundler/gems/sidekiq-222442e993d4/lib/sidekiq/cli.rb:73:in `run'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bundler/gems/sidekiq-222442e993d4/bin/sidekiq:8:in `<top (required)>'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bin/sidekiq:19:in `load'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bin/sidekiq:19:in `<main>'
Thread TID-17x9dc 
/home/betelgeuse/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/thread.rb:71:in `wait'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/mailbox.rb:71:in `receive'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/actor.rb:153:in `run'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/actor.rb:109:in `block in initialize'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/thread_handle.rb:12:in `block in initialize'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/internal_pool.rb:48:in `call'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/internal_pool.rb:48:in `block in create'
Thread TID-17uf52 
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/connection/ruby.rb:52:in `rescue in _read_from_socket'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/connection/ruby.rb:48:in `_read_from_socket'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/connection/ruby.rb:41:in `gets'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/connection/ruby.rb:204:in `read'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/client.rb:205:in `block in read'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/client.rb:196:in `io'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/client.rb:204:in `read'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/client.rb:78:in `block in call'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/client.rb:177:in `block (2 levels) in process'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/client.rb:281:in `ensure_connected'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/client.rb:167:in `block in process'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/client.rb:242:in `logging'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/client.rb:166:in `process'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/client.rb:78:in `call'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/client.rb:159:in `block in call_without_timeout'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/client.rb:220:in `without_socket_timeout'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis/client.rb:158:in `call_without_timeout'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis.rb:946:in `block in _bpop'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis.rb:36:in `block in synchronize'
/home/betelgeuse/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis.rb:36:in `synchronize'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis.rb:945:in `_bpop'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/redis-3.0.1/lib/redis.rb:972:in `blpop'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bundler/gems/sidekiq-222442e993d4/lib/sidekiq/fetch.rb:36:in `block (2 levels) in fetch'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/connection_pool-0.9.1/lib/connection_pool.rb:47:in `with'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bundler/gems/sidekiq-222442e993d4/lib/sidekiq.rb:63:in `redis'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bundler/gems/sidekiq-222442e993d4/lib/sidekiq/fetch.rb:36:in `block in fetch'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bundler/gems/sidekiq-222442e993d4/lib/sidekiq/util.rb:21:in `watchdog'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bundler/gems/sidekiq-222442e993d4/lib/sidekiq/fetch.rb:30:in `fetch'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bundler/gems/sidekiq-222442e993d4/lib/sidekiq/fetch.rb:41:in `block (2 levels) in fetch'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/actor.rb:199:in `block (2 levels) in after'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/task.rb:45:in `block in initialize'
Thread TID-17txvy 
/home/betelgeuse/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/thread.rb:71:in `wait'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/mailbox.rb:71:in `receive'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/actor.rb:153:in `run'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/actor.rb:109:in `block in initialize'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/thread_handle.rb:12:in `block in initialize'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/internal_pool.rb:48:in `call'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/internal_pool.rb:48:in `block in create'
Thread TID-17qdzs 
/home/betelgeuse/.rvm/rubies/ruby-1.9.2-p290/lib/ruby/1.9.1/thread.rb:71:in `wait'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/mailbox.rb:71:in `receive'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/actor.rb:153:in `run'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/actor.rb:109:in `block in initialize'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/thread_handle.rb:12:in `block in initialize'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/internal_pool.rb:48:in `call'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/internal_pool.rb:48:in `block in create'
2012-06-09T15:50:02Z 22745 TID-17x9dc ERROR: Manager#processor_died died
2012-06-09T15:50:02Z 22745 TID-17x9dc ERROR: task was terminated
2012-06-09T15:50:02Z 22745 TID-17x9dc ERROR: /home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/task.rb:23:in `suspend'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/actor.rb:52:in `call'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/actor_proxy.rb:99:in `method_missing'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid.rb:118:in `new_link'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bundler/gems/sidekiq-222442e993d4/lib/sidekiq/manager.rb:91:in `block in processor_died'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bundler/gems/sidekiq-222442e993d4/lib/sidekiq/util.rb:21:in `watchdog'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/bundler/gems/sidekiq-222442e993d4/lib/sidekiq/manager.rb:86:in `processor_died'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/actor.rb:238:in `handle_exit_event'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/actor.rb:155:in `block in run'
/home/betelgeuse/.rvm/gems/ruby-1.9.2-p290/gems/celluloid-0.11.0/lib/celluloid/task.rb:45:in `block in initialize'

Easy fix would be to just tell people to use 1.9.3 for now and document that 1.9.2. If this isn't fixed on 1.9.2 it makes deploys a bit hard because USR1 makes the process exit.

@betelgeuse
Collaborator

Verified that the same happens on 1.9.2-p320

@mperham
Owner
mperham commented Jun 9, 2012

I don't understand. You're saying that USR1 causes Sidekiq to crash on 1.9.2?

@betelgeuse
Collaborator

I am not sure if it crashes or control just flows out. Crashes usually come with a backtrace. But the end result is that the daemon exits with USR1.

@mperham
Owner
mperham commented Jun 11, 2012

Can anyone else reproduce this on 1.9.2? This just seems bizarre.

@mperham
Owner
mperham commented Jun 17, 2012

So bizarre, I can reproduce this but I have no idea what's causing it.

@aosalias

I'm seeing this on one process out of fifteen on a fresh boot of redis and sidekiq with no msgs processed or queued.

2012-12-21T19:13:52Z 53732 TID-oupqwkb0w INFO: Thread TID-ouprlwmmc
2012-12-21T19:13:52Z 53732 TID-oupqwkb0w INFO: /Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/connection/ruby.rb:52:in rescue in _read_from_socket'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/connection/ruby.rb:48:in
_read_from_socket'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/connection/ruby.rb:41:in gets'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/connection/ruby.rb:236:in
read'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/client.rb:211:in block in read'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/client.rb:202:in
io'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/client.rb:210:in read'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/client.rb:84:in
block in call'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/client.rb:183:in block (2 levels) in process'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/client.rb:287:in
ensure_connected'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/client.rb:173:in block in process'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/client.rb:248:in
logging'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/client.rb:172:in process'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/client.rb:84:in
call'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/client.rb:165:in block in call_without_timeout'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/client.rb:226:in
without_socket_timeout'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis/client.rb:164:in call_without_timeout'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis.rb:943:in
block in _bpop'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis.rb:36:in block in synchronize'
/Users/involver/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/monitor.rb:211:in
mon_synchronize'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis.rb:36:in synchronize'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis.rb:942:in
_bpop'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/redis-3.0.2/lib/redis.rb:969:in blpop'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/sidekiq-2.5.4/lib/sidekiq/fetch.rb:35:in
block (2 levels) in fetch'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/connection_pool-0.9.2/lib/connection_pool.rb:49:in with'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/sidekiq-2.5.4/lib/sidekiq.rb:67:in
redis'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/sidekiq-2.5.4/lib/sidekiq/fetch.rb:35:in block in fetch'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/sidekiq-2.5.4/lib/sidekiq/util.rb:15:in
watchdog'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/sidekiq-2.5.4/lib/sidekiq/fetch.rb:31:in fetch'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/sidekiq-2.5.4/lib/sidekiq/fetch.rb:40:in
block (2 levels) in fetch'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/celluloid-0.12.4/lib/celluloid/tasks/task_fiber.rb:24:in block in initialize'
2012-12-21T19:13:52Z 53732 TID-oupqwkb0w INFO: Thread TID-oupqzfwoo
2012-12-21T19:13:52Z 53732 TID-oupqwkb0w INFO: /Users/involver/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/thread.rb:71:in
wait'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/celluloid-0.12.4/lib/celluloid/mailbox.rb:66:in receive'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/celluloid-0.12.4/lib/celluloid/actor.rb:195:in
run'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/celluloid-0.12.4/lib/celluloid/actor.rb:184:in block in initialize'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/celluloid-0.12.4/lib/celluloid/thread_handle.rb:17:in
block in initialize'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/celluloid-0.12.4/lib/celluloid/internal_pool.rb:48:in call'
/Users/involver/.rvm/gems/ruby-1.9.3-p194@amp/gems/celluloid-0.12.4/lib/celluloid/internal_pool.rb:48:in
block in create'

@mperham
Owner
mperham commented Apr 22, 2013

I'm just going to declare 1.9.2 as unsupported. 1.9.2 also has YAML issues.

@mperham mperham closed this Apr 22, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.