Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sidekiq-ent loops disappear from web UI #3948

Closed
hanahiroAze opened this Issue Sep 6, 2018 · 4 comments

Comments

Projects
None yet
3 participants
@hanahiroAze
Copy link

hanahiroAze commented Sep 6, 2018

Ruby version: 2.4.3
Sidekiq / Pro / Enterprise version(s): 5.0.4 / 3.7.1 / 1.6.1
redis-rails version: 3.3.5


logs

lv:info 2018-09-06T06:35:39.876Z 12990 TID-gmowx9zdg INFO: Lost leadership of Sidekiq cluster
lv:info 2018-09-06T06:36:39.877Z 12990 TID-gmowx9zdg INFO: Gained leadership of the cluster
lv:info 2018-09-06T06:37:24.878Z 12990 TID-gmowx9zdg INFO: Lost leadership of Sidekiq cluster
lv:info 2018-09-06T06:38:24.879Z 12990 TID-gmowx9zdg INFO: Gained leadership of the cluster

sidekiq.yml

Sidekiq.configure_server do |config|
  config.redis = {
    url: ENV.fetch("SIDEKIQ_REDIS_URI") { "redis://localhost:6379/0" },
  }

  config.periodic do |mgr|
    mgr.register('0 23 * * *', MyWorker)
    mgr.register('*/5 * * * *', TestWorker)
  end
end

Sidekiq.configure_client do |config|
  config.redis = {
    url: ENV.fetch("SIDEKIQ_REDIS_URI") { "redis://localhost:6379/0" },
  }
end

After above log, sidekiq loops's web UI says 'NoLoopsFound'.
However cron(TestWorker) still work.
I wonder why this happens and is related to #3002.

Do you have any idea to fix this?

@mperham

This comment has been minimized.

Copy link
Owner

mperham commented Sep 6, 2018

Thank you, your report uncovered a bug in #3950.

It sounds like there is something very wrong with your Redis, perhaps it is being flushed somehow but it is not normal to repeatedly lose cluster leadership like that. I can't tell any more than that from your report.

@hanahiroAze

This comment has been minimized.

Copy link
Author

hanahiroAze commented Sep 7, 2018

@mperham
Thanks for your reply. It may be happened by Redis's eviction in my case.

@mperham

This comment has been minimized.

Copy link
Owner

mperham commented Sep 7, 2018

@hanahiroAze

This comment has been minimized.

Copy link
Author

hanahiroAze commented Sep 7, 2018

@mperham

sidekiq loops's web UI says 'NoLoopsFound'.

Sorry it was caused by my env var configuration.
As I divided worker's server and web UI server, web's server didn't see the worker's Redis.

lose cluster leadership

After I divide Redis server from our Redis for application cache - I used same sever by dividing db and this Redis eviction had occurred - , it hasn't been seen.

@seuros seuros added the enterprise label Sep 17, 2018

@mperham mperham closed this Oct 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.