sidekiq in production stops processing jobs

[Tshamp7]

hello! wasn't too sure where to post this since i dont think its a "bug" but i could certainly use some help!

So, our sidekiq in our production application regularly just stops processing jobs. we'll have something like 95,000 jobs enqueued and the "busy" will crash from 28 (all threads) to 0. with 70 or 80,000 jobs still waiting in the queue. I have noticed, when this happens, these redis metrics will show as "undefined", however, sidekiq apparently can still see how many jobs are waiting, and is just not pulling any?

Our latency goes up to about 100,000 when this occurs.

here is my sidekiq.rb file:

`

frozen_string_literal: true

Sidekiq.configure_server do |config|
config.redis = {
url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1'),
ssl_params: {
verify_mode: OpenSSL::SSL::VERIFY_NONE,
},
}

Sidekiq::Status.configure_server_middleware config, expiration: 30.minutes

Sidekiq::Status.configure_client_middleware config, expiration: 30.minutes
end

Sidekiq.configure_client do |config|
config.redis = {
url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1'),
ssl_params: {
verify_mode: OpenSSL::SSL::VERIFY_NONE,
},
}

config.client_middleware do |chain|
chain.add Sidekiq::Status::ClientMiddleware
end

Sidekiq::Status.configure_client_middleware config, expiration: 30.minutes
end

schedule_file = 'config/schedule.yml'

if File.exist?(schedule_file) && Sidekiq.server?
Rails.application.config.after_initialize do
Sidekiq::Cron::Job.load_from_hash YAML.load_file(schedule_file)
end
end

Sidekiq.logger.level = Logger::WARN if Rails.env.test?

`
any help is much appreciated!

[mperham]

Have you looked at the problems and troubleshooting wiki page, and especially the TTIN signal, which will give you a dump of all the  backtraces in the process? See if that helps you debug the issue.You might also search the closed issues because a number of people have reported this exact same problem over the last few months. On Oct 5, 2022, at 10:27, Tom Shamp ***@***.***> wrote: hello! wasn't too sure where to post this since i dont think its a "bug" but i could certainly use some help! So, our sidekiq in our production application regularly just stops processing jobs. we'll have something like 95,000 jobs enqueued and the "busy" will crash from 28 (all threads) to 0. with 70 or 80,000 jobs still waiting in the queue. I have noticed, when this happens, these redis metrics will show as "undefined", however, sidekiq apparently can still see how many jobs are waiting, and is just not pulling any? Our latency goes up to about 100,000 when this occurs. here is my sidekiq.rb file: ` frozen_string_literal: true Sidekiq.configure_server do |config| config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1'), ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE, }, } Sidekiq::Status.configure_server_middleware config, expiration: 30.minutes Sidekiq::Status.configure_client_middleware config, expiration: 30.minutes end Sidekiq.configure_client do |config| config.redis = { url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/1'), ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE, }, } config.client_middleware do |chain| chain.add Sidekiq::Status::ClientMiddleware end Sidekiq::Status.configure_client_middleware config, expiration: 30.minutes end schedule_file = 'config/schedule.yml' if File.exist?(schedule_file) && Sidekiq.server? Rails.application.config.after_initialize do Sidekiq::Cron::Job.load_from_hash YAML.load_file(schedule_file) end end Sidekiq.logger.level = Logger::WARN if Rails.env.test? ` any help is much appreciated! —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Tshamp7]

ah, sorry, i did not think to look in closed! i will do that now.

[bogn83]

The description matches what we have been experiencing twice in one week since the update to sidekiq 7. Will have a look at the closed issues.

[mperham]

See the Problems and Troubleshooting wiki page.