Skip to content
Pull request Compare This branch is 1413 commits behind mperham:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
..
Failed to load latest commit information.
app
config
db
lib
log
script
.gitignore
Capfile
Gemfile
Rakefile
bench.rb
config.ru
readme.md

readme.md

Ruby 2.0 slowness with big concurrency

To reproduce:

Install redis locally and checkout the Sidekiq repo:

git clone git://github.com/mperham/sidekiq
cd sidekiq/myapp
bundle

Load lots of jobs into Redis, you can edit Rakefile to change the job count:

> time bundle exec rake load_jobs
2013-04-26T15:32:37Z 66670 TID-ov6pgqt3k INFO: Sidekiq client using redis://localhost:6379/0 with options {:size=>2, :namespace=>"foo"}

real    0m49.206s
user    0m45.037s
sys 0m3.558s

Start Sidekiq with 200 worker threads to process those jobs. You will need to kill Sidekiq with Ctrl-C as soon as log messages stop flying by.

> time bundle exec sidekiq -c 200
...
2013-04-26T15:46:25Z 67059 TID-ovwfo7krk EmptyWorker JID-cce0eed5ea7281b720841ecc INFO: done: 0.224 sec
2013-04-26T15:46:25Z 67059 TID-ovwfojqdg EmptyWorker JID-850c242ec75de77400838fa9 INFO: done: 0.231 sec
2013-04-26T15:46:25Z 67059 TID-ovwfo6u94 EmptyWorker JID-fd2422116a771dc67a848489 INFO: done: 0.223 sec
2013-04-26T15:46:25Z 67059 TID-ovwfo94ag EmptyWorker JID-45ecb59a43e62ebb574ee0da INFO: done: 0.223 sec
2013-04-26T15:46:25Z 67059 TID-ovwfonv7s EmptyWorker JID-c7beabde4c0141b0669c4baf INFO: done: 0.221 sec
^C2013-04-26T15:46:27Z 67059 TID-ovwfalyvw INFO: Shutting down
2013-04-26T15:46:27Z 67059 TID-ovwfefrzw INFO: Shutting down 200 quiet workers

real    3m37.921s
user    3m3.110s
sys 1m4.783s
> ruby -v
ruby 1.9.3p374 (2013-01-15 revision 38858) [x86_64-darwin12.3.0]

Now try with various Rubies to see how your choice in VM performs.

2013-04-26T15:57:04Z 67357 TID-ow0muevo8 EmptyWorker JID-cb03f00310af3590009b19fe INFO: done: 0.189 sec
2013-04-26T15:57:04Z 67357 TID-ow0muqgxc EmptyWorker JID-89710c72bf4a343834833bc4 INFO: done: 0.19 sec
2013-04-26T15:57:04Z 67357 TID-ow0ms6rc4 EmptyWorker JID-bb358d8d8f02af9734b8a3ff INFO: done: 0.186 sec
2013-04-26T15:57:04Z 67357 TID-ow0mwn908 EmptyWorker JID-7c032c55e6ab9a63dc31f175 INFO: done: 0.188 sec
2013-04-26T15:57:04Z 67357 TID-ow0mv6m80 EmptyWorker JID-73dc04f9e431ebf433707d02 INFO: done: 0.188 sec
^C2013-04-26T15:57:11Z 67357 TID-ow0m8kxh4 INFO: Shutting down
2013-04-26T15:57:11Z 67357 TID-ow0mwqr20 INFO: Shutting down 200 quiet workers

real    5m55.033s
user    5m6.156s
sys 1m4.886s
> ruby -v
ruby 2.0.0p0 (2013-02-24 revision 39474) [x86_64-darwin12.3.0]
Something went wrong with that request. Please try again.