Create Sidekiq load tester

[mperham]

The recent Celluloid 0.17 leak fiasco has shown that my rudimentary smoke testing is not sufficient. Create a script which can start up Sidekiq and then push N good/bad jobs to it while also monitoring memory and CPU usage.

Script

The script should:

• Launch Sidekiq server
• Create client thread which fires N jobs as fast as possible
• 1% of jobs should fail deterministically
• Sample process memory and CPU every 10 seconds
• Shutdown system when queue and retries are empty.

Example:

> bundle exec sidekiq_load -n 50000
15:01:00 RSS: 56MB Queue: 12000
15:01:10 RSS: 58MB Queue: 11000
15:01:20 RSS: 59MB Queue: 10000

[mperham]

Clearly visible leak with Celluloid 0.17.1.2:

$ bundle exec bin/sidekiqload 
0.17.1.2
2015-10-01T23:26:36.272Z 14847 TID-owgzfmigw INFO: Booting Sidekiq 3.5.1 with redis options {:db=>13, :url=>nil}
2015-10-01T23:26:38.540Z 14847 TID-owgzg3q6w ERROR: RSS: 70360 Pending: 69965
2015-10-01T23:26:39.095Z 14847 TID-owgzg3qi0 ERROR: Created 100000 jobs
2015-10-01T23:26:40.574Z 14847 TID-owgzg3q6w ERROR: RSS: 72168 Pending: 98879
2015-10-01T23:26:42.606Z 14847 TID-owgzg3q6w ERROR: RSS: 72844 Pending: 97293
2015-10-01T23:26:44.641Z 14847 TID-owgzg3q6w ERROR: RSS: 74020 Pending: 95852
2015-10-01T23:26:46.686Z 14847 TID-owgzg3q6w ERROR: RSS: 74688 Pending: 94258
2015-10-01T23:26:48.719Z 14847 TID-owgzg3q6w ERROR: RSS: 75388 Pending: 92817
2015-10-01T23:26:50.756Z 14847 TID-owgzg3q6w ERROR: RSS: 75916 Pending: 91159
2015-10-01T23:26:52.795Z 14847 TID-owgzg3q6w ERROR: RSS: 76600 Pending: 89600
2015-10-01T23:26:54.860Z 14847 TID-owgzg3q6w ERROR: RSS: 77396 Pending: 88007
2015-10-01T23:26:56.904Z 14847 TID-owgzg3q6w ERROR: RSS: 77700 Pending: 86512
2015-10-01T23:26:58.950Z 14847 TID-owgzg3q6w ERROR: RSS: 78836 Pending: 84892
2015-10-01T23:27:01.006Z 14847 TID-owgzg3q6w ERROR: RSS: 79100 Pending: 83636
2015-10-01T23:27:03.042Z 14847 TID-owgzg3q6w ERROR: RSS: 80152 Pending: 82192
2015-10-01T23:27:05.083Z 14847 TID-owgzg3q6w ERROR: RSS: 81096 Pending: 80603
2015-10-01T23:27:07.122Z 14847 TID-owgzg3q6w ERROR: RSS: 81556 Pending: 79066
2015-10-01T23:27:09.164Z 14847 TID-owgzg3q6w ERROR: RSS: 82896 Pending: 77583
2015-10-01T23:27:11.203Z 14847 TID-owgzg3q6w ERROR: RSS: 83216 Pending: 75988
2015-10-01T23:27:13.307Z 14847 TID-owgzg3q6w ERROR: RSS: 84580 Pending: 74457
2015-10-01T23:27:15.360Z 14847 TID-owgzg3q6w ERROR: RSS: 85060 Pending: 72965
2015-10-01T23:27:17.402Z 14847 TID-owgzg3q6w ERROR: RSS: 86192 Pending: 71378
2015-10-01T23:27:19.468Z 14847 TID-owgzg3q6w ERROR: RSS: 86516 Pending: 69768
2015-10-01T23:27:21.511Z 14847 TID-owgzg3q6w ERROR: RSS: 87144 Pending: 68224

[mperham]

Lookin' mighty fine, Celluloid 0.17.2:

$ bundle exec bin/sidekiqload 
0.17.2
2015-10-01T23:29:47.378Z 15012 TID-ovh6w0bg8 INFO: Booting Sidekiq 3.5.1 with redis options {:db=>13, :url=>nil}
2015-10-01T23:29:49.569Z 15012 TID-ovh741f6k ERROR: RSS: 72572 Pending: 69968
2015-10-01T23:29:50.099Z 15012 TID-ovh741fc4 ERROR: Created 100000 jobs
2015-10-01T23:29:51.595Z 15012 TID-ovh741f6k ERROR: RSS: 73484 Pending: 98885
2015-10-01T23:29:53.626Z 15012 TID-ovh741f6k ERROR: RSS: 74008 Pending: 97211
2015-10-01T23:29:55.661Z 15012 TID-ovh741f6k ERROR: RSS: 71800 Pending: 95505
2015-10-01T23:29:57.695Z 15012 TID-ovh741f6k ERROR: RSS: 71976 Pending: 93811
2015-10-01T23:29:59.726Z 15012 TID-ovh741f6k ERROR: RSS: 71992 Pending: 92135
2015-10-01T23:30:01.757Z 15012 TID-ovh741f6k ERROR: RSS: 72052 Pending: 90451
2015-10-01T23:30:03.783Z 15012 TID-ovh741f6k ERROR: RSS: 72088 Pending: 88805
2015-10-01T23:30:05.808Z 15012 TID-ovh741f6k ERROR: RSS: 72248 Pending: 87151
2015-10-01T23:30:07.844Z 15012 TID-ovh741f6k ERROR: RSS: 72272 Pending: 85461
2015-10-01T23:30:09.872Z 15012 TID-ovh741f6k ERROR: RSS: 72272 Pending: 83761
2015-10-01T23:30:11.897Z 15012 TID-ovh741f6k ERROR: RSS: 72276 Pending: 82095
2015-10-01T23:30:13.929Z 15012 TID-ovh741f6k ERROR: RSS: 72272 Pending: 80410
2015-10-01T23:30:15.958Z 15012 TID-ovh741f6k ERROR: RSS: 72272 Pending: 78735
2015-10-01T23:30:17.987Z 15012 TID-ovh741f6k ERROR: RSS: 72348 Pending: 77070
2015-10-01T23:30:20.021Z 15012 TID-ovh741f6k ERROR: RSS: 72348 Pending: 75396
2015-10-01T23:30:22.049Z 15012 TID-ovh741f6k ERROR: RSS: 72368 Pending: 73753
2015-10-01T23:30:24.082Z 15012 TID-ovh741f6k ERROR: RSS: 72372 Pending: 72072
2015-10-01T23:30:26.109Z 15012 TID-ovh741f6k ERROR: RSS: 72372 Pending: 70414
2015-10-01T23:30:28.143Z 15012 TID-ovh741f6k ERROR: RSS: 72372 Pending: 68737
2015-10-01T23:30:30.164Z 15012 TID-ovh741f6k ERROR: RSS: 72372 Pending: 67065
2015-10-01T23:30:32.196Z 15012 TID-ovh741f6k ERROR: RSS: 72368 Pending: 65400

[aprescott]

Is it worth converting this into some kind of automated test as part of the suite?

[mperham]

I welcome any integration-type tests if someone has an idea and wants to tackle that but it's pretty non-deterministic and it shouldn't run on Travis. I think this is a good manual test to be run occasionally; we don't upgrade Celluloid versions very often.