Using rabbitmq_cluster works only the first time puppet runs

[b-0-b]

Description of problem

• What did you do?

  • I have tried to configure HA RabbitMQ using rabbitmq_cluster and hiera

• What happened?

  • /etc/sensu/conf.d/rabbitmq.json is correctly parsed the first puppet run but the next one will fail with

Error: /Stage[main]/Sensu::Rabbitmq::Config/Sensu_rabbitmq_config[sensu-26-0.foo.bar]: Could not evaluate: no implicit conversion of String into Integer

If I remove /etc/sensu/conf.d/rabbitmq.json the following puppet run will work again but fail on the puppet run after.

• What did you expect to happen?
  • As long as sensu is happy with the config file I would expect the puppet module to be able to cope with it.
• How can someone reproduce the problem?
  • In your puppet profile use include ::sensu
  • In hiera

sensu::rabbitmq_cluster:
  - port: 5671
    host: 192.168.113.9
    user: sensu
    password: REDACTED
    vhost: "sensu"
    ssl: {
      cert_chain_file: "/etc/sensu/ssl/cert.pem",
      private_key_file: "/etc/sensu/ssl/key.pem"
      }
  - port: 5671
    host: 192.168.113.10
    user: sensu
    password: REDACTED
    vhost: "sensu"
    ssl: {
      cert_chain_file: "/etc/sensu/ssl/cert.pem",
      private_key_file: "/etc/sensu/ssl/key.pem"
      }
  - port: 5671
    host: 192.168.113.11
    user: sensu
    password: REDACTED
    vhost: "sensu"
    ssl: {
      cert_chain_file: "/etc/sensu/ssl/cert.pem",
      private_key_file: "/etc/sensu/ssl/key.pem"
      }

Command used and debugging output

• Is this a masterless or master based puppet setup?
  • we have a puppet master

Platform and version information

• Your OS: Debian Jessie
• Your Ruby version:
  Client:

sensu-26-0 /root # ruby -version
ruby 2.1.5p273 (2014-11-13) [x86_64-linux-gnu]

Puppetmaster

root@puppet0:~# ruby --version
ruby 2.1.5p273 (2014-11-13) [x86_64-linux-gnu]
root@puppet0:~# cat /etc/debian_version
8.2

• Your version of Puppet: OpenSource version
  Client

sensu-26-0 /root # puppet --version
4.3.1

Puppetmaster

root@puppet0:~# /opt/puppetlabs/puppet/bin/puppet --version
4.8.1

• Your version of Sensu: latest
• Your version of this module: latest

[csoleimani]

I can confirm that I am having the same issue. I tried using a forked module with johanek's pull request above, but I still get the same error: "Could not evaluate: no implicit conversion of String into Integer " after the rabbitmq.json file has already been generated. I am also on Debian Jesse.

[cwjohnston]

@b-0-b @csoleimani are you setting values for any rabbitmq_* parameters aside from rabbitmq_cluster ? I believe this error results from passing a value for one or more of the rabbitmq parameters which are obviated by the configuration passed to rabbitmq_cluster.

[csoleimani]

@cwjohnston , I checked a previous revision of our code and the only rabbitmq parameter that was being passed was rabbitmq_cluster - I can post what my code looked like if you'd like.

[cwjohnston]

@csoleimani please share your code. Myself and others have used rabbitmq_cluster parameter successfully so whatever code you can share may help us identify the delta.

[csoleimani]

Sure, here is some of the code below. I'm only pasting the important stuff rather than the whole manifest.

# Configure a sensu server
class profile::sensu::server::main {

  $adserver          = hiera('ad::server')
  $adport            = hiera('ad::port')
  $basedn            = hiera('ad::basedn')
  $binduser          = hiera('sensu::ad::binduser')
  $bindpass          = hiera('sensu::ad::bindpass')
  $client_version    = hiera('sensu::client::version')
  $dashboard_name    = upcase($::location)
  $groupbasedn       = hiera('ad::groupbasedn')
  $log_level         = hiera('sensu::log_level')
  $rabbitmq_password = hiera('sensu::rabbitmq_password')
  $rabbitmq_port     = '5672'
  $rabbitmq_vhost    = 'sensu'
  $rabbitmq_vip      = hiera('sensu::rabbitmq::vip')
  $rabbitmq_user     = 'sensu'
  $redis_host        = hiera('sensu::redis::vip')
  $redis_port        = hiera('redis::port')
  $sensu_api_port    = hiera('sensu::api::port')
  $sensu_fqdn        = hiera('sensu::fqdn')
  $sensu_cert        = $sensu_fqdn
  $security          = 'tls'
  $userattribute     = 'mail'
  $userbasedn        = hiera('ad::userbasedn')
  $rabbitmq_cluster  = $::app_tier ? {
    /(production|staging)/ => [
      {
        host               => "rabbitmq-${app_tier}-node01",
        user               => $rabbitmq_user,
        password           => $rabbitmq_password,
        port               => $rabbitmq_port,
        refetch            => '5',
        reconnect_on_error => true,
        vhost              => $rabbitmq_vhost
      },{
        host               => "rabbitmq-${app_tier}-node02",
        user               => $rabbitmq_user,
        password           => $rabbitmq_password,
        port               => $rabbitmq_port,
        prefetch           => '5',
        reconnect_on_error => true,
        vhost              => $rabbitmq_vhost
      },{
        host               => "$rabbitmq-${app_tier}-node03",
        user               => $rabbitmq_user,
        password           => $rabbitmq_password,
        port               => $rabbitmq_port,
        prefetch           => '5',
        reconnect_on_error => true,
        vhost              => $rabbitmq_vhost
      }
    ],
    'test' => [
    {
      host               => $rabbitmq_vip,
      user               => $rabbitmq_user,
      password           => $rabbitmq_password,
      port               => $rabbitmq_port,
      refetch            => '5',
      reconnect_on_error => true,
      vhost              => $rabbitmq_vhost
    }
    ],
    default => undef
  }
  $redis_db          = lookup('sensu::redis::db')

  class { '::sensu':
    enterprise                   => true,
    enterprise_version           => $::sensu::enterprise_version,
    enterprise_dashboard         => true,
    enterprise_dashboard_version => $::sensu::enterprise_dashboard_version,
    enterprise_dashboard_host    => 'localhost',
    enterprise_dashboard_port    => '3000',
    enterprise_dashboard_refresh => '5',
    enterprise_dashboard_ldap    => {
    'basedn'              => $basedn,
    'groupbasedn'         => $groupbasedn,
    'userbasedn'          => $userbasedn,
    'binduser'            => $binduser,
    'bindpass'            => $bindpass,
    'server'              => $adserver,
    'port'                => $adport,
    'roles'               => [{
    'name'    => 'operators',
    'members' => ['smart-people'],
    },
    {
    'name'     => 'guests',
    'members'  => ['look-but-dont- touch'],
    'readonly' => true,
    }
    ],
    'security'            => $security,
    'userattribute'       => $userattribute,
    'disablenestedgroups' => true,
    },
    enterprise_user              => 'REDACTED',
    enterprise_pass              => 'REDACTED',
    install_repo                 => true,
    log_level                    => $log_level,
    purge                        => {
      config => true,
    },
    rabbitmq_cluster             => $rabbitmq_cluster,
    redis_db                     => $redis_db,
    redis_host                   => $redis_host,
    version                      => $client_version,
  }
}

[b-0-b]

Hi,

I tried to do it all in hiera

sensu::rabbitmq_cluster:
  - port: 5671
    host: 192.168.113.9
    user: sensu
    password: password
    vhost: "sensu"
    ssl: {
      cert_chain_file: "/etc/sensu/ssl/cert.pem",
      private_key_file: "/etc/sensu/ssl/key.pem"
      }
  - port: 5671
    host: 192.168.113.10
    user: sensu
    password: password
    vhost: "sensu"
    ssl: {
      cert_chain_file: "/etc/sensu/ssl/cert.pem",
      private_key_file: "/etc/sensu/ssl/key.pem"
      }
  - port: 5671
    host: 192.168.113.11
    user: sensu
    password: password
    vhost: "sensu"
    ssl: {
      cert_chain_file: "/etc/sensu/ssl/cert.pem",
      private_key_file: "/etc/sensu/ssl/key.pem"
      }

This is what I was unsure about how to do. This will create /etc/sensu/conf.d/rabbitmq.json the way I want the first time puppet runs but fails at the next puppet run.

My puppet code is

  class { '::sensu':
    init_stop_max_wait          => 120,
    log_level                   => 'info',
    manage_user                 => false,
    rabbitmq_reconnect_on_error => true,
    redis_reconnect_on_error    => true,
    redis_port                  => 6380,
    install_repo                => true,
    use_embedded_ruby           => true,
    manage_plugins_dir          => false,
    purge                       => {
      config     => true,
      mutators   => true,
      handlers   => true,
      extensions => true,
      plugins    => false,
    },
  }

[kmcfate]

I had the same problem, and confirmed that it was because i had rabbitmq_password set in hiera.

[b-0-b]

Right. Thanks for the info. I now found I have these in hiera as well

sensu::rabbitmq_password: eeCiephahx0kah5hoh4a
sensu::rabbitmq_ssl: true

[csoleimani]

I just tried hardcoding the password into the manifest and not doing a hiera lookup, but I'm still seeing the same issue. Could you please post what your code looks like, @kmcfate ? I just tried taking all of the hiera lookups out of my rabbitmq_cluster variable and it still is showing the same behavior.

[cwjohnston]

@csoleimani for your case i suspect that rabbitmq_reconnect_on_error is a contributing factor. Sensu defaults this to true, so removing the explicit definition from configuration shouldn't change behavior.

[csoleimani]

Thanks for the suggestion, @cwjohnston . I just tried removing it and unfortunately, removing that attribute didn't seem to get rid of the issue.

[b-0-b]

This works for me now after refactoring my puppet an hiera code.

manifest

  class foo::sensu
    class { '::sensu':
    init_stop_max_wait   => 120,
    log_level            => 'info',
    manage_user          => false,
    rabbitmq_cluster     => $rabbitmq_cluster,
    redis_password       => "redis_pass",
    redis_db             => 0,
    redis_auto_reconnect => true,
    redis_sentinels      => $sentinels,
    redis_master         => "sensu",
    install_repo         => true,
    use_embedded_ruby    => true,
    manage_plugins_dir   => false,
    purge                => {
      config     => true,
      mutators   => true,
      handlers   => true,
      extensions => true,
      plugins    => false,
    },
  }

hiera

foo::sensu::rabbitmq_cluster:
  - port: 5671
    host: 192.168.113.9
    user: sensu
    password: rabbitmq_pw
    vhost: "sensu"
    prefetch: 50
    heartbeat: 30
    ssl: {
      cert_chain_file: "/etc/sensu/rabbitmq_ssl_cert.pem",
      private_key_file: "/etc/sensu/rabbitmq_ssl_key.pem"
      }
  - port: 5671
    host: 192.168.113.10
    user: sensu
    password: rabbitmq_pw
    vhost: "sensu"
    prefetch: 50
    heartbeat: 30
    ssl: {
      cert_chain_file: "/etc/sensu/rabbitmq_ssl_cert.pem",
      private_key_file: "/etc/sensu/rabbitmq_ssl_key.pem"
      }
  - port: 5671
    host: 192.168.113.11
    user: sensu
    password: rabbitmq_pw
    vhost: "sensu"
    prefetch: 50
    heartbeat: 30
    ssl: {
      cert_chain_file: "/etc/sensu/rabbitmq_ssl_cert.pem",
      private_key_file: "/etc/sensu/rabbitmq_ssl_key.pem"
      }

and

foo::sensu::sentinels:
  - host: 192.168.113.9
    port: 26379
  - host: 192.168.113.10
    port: 26379
  - host: 192.168.113.11
    port: 26379

Don't use any rabbitmq_foo parameters except for rabbitmq_cluster. Manage the SSL certs for RabbitMQ outside the sensu module if you use SSL.

Hope that helps :)

[ghoneycutt]

Thanks @b-0-b for showing us the working config!

[johanek]

I too had this problem, and undefining any rabbitmq_foo parameters and manually managing the SSL certificates fixed it.

It's a bit of a shame that setting rabbitmq_ssl_cert_chain and rabbitmq_ssl_private_key to the content of the certificates doesn't write the file without disturbing the rabbitmq_cluster config, but such is life.

Thanks for figuring this out @b-0-b!

[csoleimani]

I've tried updating to the latest version of the module and am still facing this issue. Here is what my code looks like:
Hiera:

sensu::rabbitmq_cluster:
  - host: 172.31.32.154
    port: 5672
    user: sensu
    password: "%{lookup('sensu::rabbitmq_password')}"
    vhost: 'sensu'
  - host: 172.31.33.158
    port: 5672
    user: sensu
    password: "%{lookup('sensu::rabbitmq_password')}"
    vhost: 'sensu'
  - host: 172.31.33.70
    port: 5672
    user: sensu
    password: "%{lookup('sensu::rabbitmq_password')}"
    vhost: 'sensu'

Manifest:

$rabbitmq_cluster = lookup('sensu::rabbitmq_cluster')

class { '::sensu':
  enterprise                   => true,
  enterprise_version           => $::sensu::enterprise_version,
  enterprise_dashboard         => true,
  enterprise_dashboard_version => $::sensu::enterprise_dashboard_version,
  enterprise_dashboard_host    => 'localhost',
  enterprise_dashboard_port    => '3000',
  enterprise_dashboard_refresh => '5',
  enterprise_dashboard_ldap    => {
  'basedn'              => $basedn,
  'groupbasedn'         => $groupbasedn,
  'userbasedn'          => $userbasedn,
  'binduser'            => $binduser,
  'bindpass'            => $bindpass,
  'server'              => $adserver,
  'port'                => $adport,
  'roles'               => [{
  'name'    => 'operators',
  'members' => ['operators'],
  },
  {
  'name'     => 'guests',
  'members'  => ['guests'],
  'readonly' => true,
  }
  ],
  'security'            => $security,
  'userattribute'       => $userattribute,
  'disablenestedgroups' => true,
  },
  enterprise_user              => 'enterprise-user',
  enterprise_pass              => 'enterprise-password',
  install_repo                 => true,
  log_level                    => $log_level,
  purge                        => {
    config => true,
  },
  rabbitmq_cluster             => $rabbitmq_cluster,
  redis_db                     => $redis_db,
  redis_host                   => $redis_host,
  version                      => $client_version,
}

Do you see any issues with my code? Could it be possible that I'm facing the issue because I'm using enterprise?

[jeffmccune]

I'm working on this and able to reproduce the error. Thank you so much @b-0-b for the wonderful bug report. I can't express how wonderful it is working on a project like this where the issues are so clearly reported, it's saved me tremendous time trying to understand and reproduce the issue. With this one, and other issues I've recently worked on, it's been downright fun to work them through. Thanks again!

Here's a partial backtrace leading to the issue in the provider, obtained with puppet apply --trace. This is triggered when specifying rabbitmq_password with rabbitmq_cluster as described:

Error: /Stage[main]/Sensu::Rabbitmq::Config/Sensu_rabbitmq_config[sensu-server.example.com]: Could not evaluate: no implicit conversion of String into Integer
/vagrant/lib/puppet/provider/sensu_rabbitmq_config/json.rb:137:in `[]'
/vagrant/lib/puppet/provider/sensu_rabbitmq_config/json.rb:137:in `password'

The provider behavior assumes conf['rabbitmq'] is a hash on line 137. The error occurs when conf['rabbitmq'] is an Array, which lines up with rabbitmq_cluster being an array in the reproduction case.

I'll pick back up tomorrow coming up with a game plan to improve this surprising behavior.

[jeffmccune]

The way the module is currently implemented, setting rabbitmq_* class parameters is mutually exclusive with setting rabbitmq_cluster. #742 changes the behavior to avoid the error by ignoring the sensu::rabbitmq_* class parameters when rabbitmq_cluster is defined.

[jeffmccune]

PR #742 address the original issue reported by @b-0-b . @csoleimani I'm taking a look at the related issue you reported with sensu-enterprise now. Thanks for the example code.

[jeffmccune]

Hi @csoleimani

I've looked into the issue you're facing and you're running into the same issue originally reported. Consider this hiera data you have:

password: "%{lookup('sensu::rabbitmq_password')}"

This means sensu::rabbitmq_password exists, which will have the same effect as specifying the rabbitmq_password sensu class parameter. This, in turn, triggers the issue originally reported.

The change in #742 should fix the issue for you. The change causes sensu::rabbitmq_password to be ignored. With your code the password will take effect in the cluster configuration. Please keep an eye here for the released version, or give the code in #724 a shot and let us know if it's still broken for you.

Thanks for the clear information about how to reproduce your issue.

[ghoneycutt]

Fix implemented in v2.24.0