Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Redesign Sidekiq internals #2593
My recent experiment to remove Celluloid has reaped huge rewards. This branch does several things:
Processing 100,000 no-op jobs with one process and 25 threads on MRI 2.2:
This work would be released as Sidekiq 4.0, with an ETA of Thanksgiving.
Have you considered using a
@mperham What happens if the status updator/heartbeat thread dies? Will it have status info about jobs that finished but it never sends that info back to Redis? It seems like the buffering of stats creates a hole for data to get lost in a crash, resulting in even more double-run or stuck jobs.
referenced this pull request
Oct 20, 2015
I did a quick Resque 1.25.2 benchmark for comparison. Time to process 10,000 noop jobs:
1 process, 96 sec
Extrapolating to 100,000 jobs would be ~300 seconds. Sidekiq does the same in 20 seconds so Sidekiq's job overhead appears to be 15x less.