What is the maximum number of local processes (consumers) for Sidekiq?

[georgeu2000]

I want to run a single instance of Sidekiq on AWS with the largest number of CPU's available, 896.

There would be 894 processes consuming Sidekiq jobs. What is the upper limit for number of consumers? Does this limit change with the license?

[mperham]

There's no hard coded limits. I've heard of customers running Redis with thousands of Sidekiq threads. Sidekiq Enterprise's multi-process support might be essential to save a ton of memory. Can you share anything more about your use case, either publicly here or privately via email?

[georgeu2000]

Thanks for the quick reply.

Sure, it's a data mining project. Each process only runs one thread (for simplicity). Each process only requires appox. 300MB, so memory is not a limiting factor (896 vCPU instances start at 12TB of memory). Currently runs fine with 8 processes on my dev machine. Each job takes about 13 min to complete, so the demand on Sidekiq is actually minimal.

I choose to run on a single high CPU machine for simplicity; I don't need to manage multiple instances, set up virtual networking, a separate instance to run Sidekiq, etc. It's just for a personal project.

[mperham]

It should work fine, especially with a local Redis.

…

On Sat, Nov 2, 2024, at 19:24, George wrote: Thanks for the quick reply. Sure, it's a data mining project. Each process only runs one thread (for simplicity). Each process only requires appox. 300MB, so memory is not a limiting factor (896 vCPU instances start at 12TB of memory). Currently runs fine with 8 processes on my dev machine. Each job takes about 13 min to complete, so the demand on Sidekiq is actually minimal. I choose to run on a single high CPU machine for simplicity; I don't need to manage multiple instances, set up virtual networking, a separate instance to run Sidekiq, etc. It's just for a personal project. — Reply to this email directly, view it on GitHub <#6488 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAWXYVFFH6KAMUNX37LCLZ6WCOVAVCNFSM6AAAAABRCHWBD6VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCMJTGIYTMOI>. You are receiving this because you commented.Message ID: ***@***.***>

[georgeu2000]

Got it. Thanks for all your help.