Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ERR invalid password using Sentinels on Sidekiq `6.0.4+` #4499

Closed
plribeiro3000 opened this issue Mar 21, 2020 · 8 comments
Closed

ERR invalid password using Sentinels on Sidekiq `6.0.4+` #4499

plribeiro3000 opened this issue Mar 21, 2020 · 8 comments

Comments

@plribeiro3000
Copy link

@plribeiro3000 plribeiro3000 commented Mar 21, 2020

Ruby version:
Ruby 2.6.5 / Rails 5.2.4.2 / Sidekiq 6.04+

Initializer

require 'sidekiq'
require 'sidekiq/middleware/i18n'
require_relative '../../lib/configuration'

Sidekiq.configure_client do |config|
  config.redis = Configuration.sidekiq_client
end

Sidekiq.configure_server do |config|
  config.redis = Configuration.sidekiq_server

  ar_configuration = ActiveRecord::Base.configurations[Rails.env]
  ar_configuration['pool'] = 13

  ActiveRecord::Base.establish_connection(ar_configuration)
end

lib/configuration.rb

class Configuration
  def sidekiq_server
    {
        driver: :hiredis,
        url: "redis://<user>:<password>@<name>",
        sentinels:
          [
            {
              host: "X.XXX.XXX.XX",
              port: XXXXX,
              password: "<password>"
            },
            {
              host: "X.XXX.XXX.XX",
              port: XXXXX,
              password: "<password>"
            },
            {
              host: "X.XXX.XXX.XX",
              port: XXXXX,
              password: "<password>"
          ],
        failover_reconnect_timeout: 20
      }
  end

  def sidekiq_client
    {
        driver: :hiredis,
        url: "redis://<user>:<password>@<name>",
        size: <size>
        sentinels:
          [
            {
              host: "X.XXX.XXX.XX",
              port: XXXXX,
              password: "<password>"
            },
            {
              host: "X.XXX.XXX.XX",
              port: XXXXX,
              password: "<password>"
            },
            {
              host: "X.XXX.XXX.XX",
              port: XXXXX,
              password: "<password>"
          ],
        failover_reconnect_timeout: 20
      }
  end
end

I was using sidekiq 6.0.1 with this code without any issues. Once i've updated to 6.0.5 i've got this error:

2020-03-21T00:39:52.247Z pid=39153 tid=owdxoxoo1 INFO: Booted Rails 5.2.4.2 application in development environment
2020-03-21T00:39:52.248Z pid=39153 tid=owdxoxoo1 INFO: Running in ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin18]
2020-03-21T00:39:52.248Z pid=39153 tid=owdxoxoo1 INFO: See LICENSE and the LGPL-3.0 for licensing details.
2020-03-21T00:39:52.249Z pid=39153 tid=owdxoxoo1 INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
ERR invalid password
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:126:in `call'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:107:in `block in connect'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:306:in `with_reconnect'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:105:in `connect'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:381:in `ensure_connected'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:231:in `block in process'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:319:in `logging'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:230:in `process'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:125:in `call'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:603:in `block in resolve_master'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:585:in `block in sentinel_detect'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:576:in `each'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:576:in `sentinel_detect'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:602:in `resolve_master'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:565:in `resolve'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:344:in `establish_connection'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:106:in `block in connect'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:306:in `with_reconnect'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:105:in `connect'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:381:in `ensure_connected'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:231:in `block in process'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:319:in `logging'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:230:in `process'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis/client.rb:125:in `call'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis.rb:286:in `block in info'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis.rb:52:in `block in synchronize'
/Users/plribeiro3000/.rvm/rubies/ruby-2.6.5/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis.rb:52:in `synchronize'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/redis-4.1.3/lib/redis.rb:285:in `info'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/sidekiq-6.0.5/lib/sidekiq.rb:118:in `block in redis_info'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/sidekiq-6.0.5/lib/sidekiq.rb:97:in `block in redis'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/connection_pool-2.2.2/lib/connection_pool.rb:65:in `block (2 levels) in with'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `handle_interrupt'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `block in with'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `handle_interrupt'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `with'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/sidekiq-6.0.5/lib/sidekiq.rb:94:in `redis'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/sidekiq-6.0.5/lib/sidekiq.rb:112:in `redis_info'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/sidekiq-6.0.5/lib/sidekiq/cli.rb:61:in `run'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/gems/sidekiq-6.0.5/bin/sidekiq:12:in `<top (required)>'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/bin/sidekiq:23:in `load'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/bin/sidekiq:23:in `<main>'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/bin/ruby_executable_hooks:24:in `eval'
/Users/plribeiro3000/.rvm/gems/ruby-2.6.5@four_shark/bin/ruby_executable_hooks:24:in `<main>'

I've tried to use master but it did not solve it.
6.0.4 has the same issue.
It works on 6.0.3

@plribeiro3000
Copy link
Author

@plribeiro3000 plribeiro3000 commented Mar 21, 2020

I've tried to find what changed that could be the cause without success.
v6.0.3...master
What am i missing?

@mperham
Copy link
Owner

@mperham mperham commented Mar 23, 2020

Seems like a Redis config issue, not clear this is a Sidekiq issue.

@mperham mperham closed this Mar 23, 2020
@plribeiro3000
Copy link
Author

@plribeiro3000 plribeiro3000 commented Apr 1, 2020

@mperham I'm using the latest version of hiredis (0.6.3) and latest version of redis (4.13) for a while now and didn't change anything for the code to break besides sidekiq version.

I know this seems like a config issue or either a question, but it is not.
With sidekiq 6.0.3 it just works. If i bump just sidekiq it starts to fail.

Could you at least point me at where i should look?

@mperham
Copy link
Owner

@mperham mperham commented Apr 1, 2020

@plribeiro3000
Copy link
Author

@plribeiro3000 plribeiro3000 commented Apr 1, 2020

This was my first check, but it only affects logging. =s

I will keep it locked to version 6.0.3 for now until i figure out what is happening.

@mperham
Copy link
Owner

@mperham mperham commented Apr 1, 2020

mperham added a commit that referenced this issue Apr 1, 2020
Ensures any password mangling doesn't affect the runtime options
@mperham
Copy link
Owner

@mperham mperham commented Apr 1, 2020

@plribeiro3000 Thank you for following up! I forgot about the change and didn't realize the codebase had special handling for sentinel passwords. You can run master if you need this fix immediately.

@plribeiro3000
Copy link
Author

@plribeiro3000 plribeiro3000 commented Apr 3, 2020

Awesome. Thank you for your time to fix It!
I will check it next week.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.