sensu::client::config keepalives 'change' every run #336

poolski · 2015-03-25T16:12:48Z

Every single time I run puppet, the module seems to change my thresholds for keepalive, flapping detection etc as follows. It's not a big deal but it does add time to puppet runs and it's a bit misleading, given no change takes place

Notice: /Stage[main]/Sensu::Client::Config/Sensu_client_config[hostname.example.com]/custom: custom changed 'handlers => ["default"]' to 'handlers => ["default"], keepalive => {"high_flap_threshold"=>20, "low_flap_threshold"=>5, "refresh"=>14400, "thresholds"=>{"critical"=>120, "warning"=>90}}'
Notice: /Stage[main]/Sensu::Client::Config/Sensu_client_config[hostname.example.com]/keepalive: keepalive changed 'high_flap_threshold => 20, low_flap_threshold => 5, refresh => 14400, thresholds => {"critical"=>120, "warning"=>90}' to ''

The text was updated successfully, but these errors were encountered:

jlambert121 · 2015-03-25T16:31:52Z

What version of the module are you using?

poolski · 2015-03-25T17:00:19Z

1.5.0

poolski · 2015-04-01T11:14:07Z

Any joy with this?

Also, when's the next release of the module out? I'm looking forward to being able to incorporate the fixes from #298 in my environments.

superseb · 2015-04-10T08:54:51Z

@poolski Can you supply the manifest that is causing these messages? I'll try to reproduce.

superseb · 2015-04-13T16:48:57Z

@poolski And please check if the fix in #313 works for you.

poolski · 2015-04-14T09:59:18Z

I will, @superseb

poolski · 2015-04-14T11:04:22Z

With the addition of the port check in #343 my puppet runs now also changes the port every time it runs

Notice: /Stage[main]/Sensu::Client::Config/Sensu_client_config[myhost.mynetwork.net]/port: port changed '' to '3030'
Notice: /Stage[main]/Sensu::Client::Config/Sensu_client_config[myhost.mynetwork.net]/custom: custom changed 'handlers => ["default"]' to 'handlers => ["default"], keepalive => {"high_flap_threshold"=>20, "low_flap_threshold"=>5, "refresh"=>14400, "thresholds"=>{"critical"=>120, "warning"=>90}}'
Notice: /Stage[main]/Sensu::Client::Config/Sensu_client_config[myhost.mynetwork.net]/keepalive: keepalive changed 'high_flap_threshold => 20, low_flap_threshold => 5, refresh => 14400, thresholds => {"critical"=>120, "warning"=>90}' to ''

I'm running v1.5.5 of the module to test if it fixes anything.

superseb · 2015-04-14T11:10:03Z

@poolski That's quite weird, you are using v1.5.5 at the moment? Did you restart the puppetmaster? (puppetmaster/pupperserver/pe-httpd)

poolski · 2015-04-14T12:45:23Z

@superseb yeah I did. It was throwing all sorts of crazy invalid setting errors till I tried that.
We run a puppetmaster with Apache/Passenger.

Also, yes, I'm using v1.5.5.

superseb · 2015-04-14T13:44:18Z

Ok, what Puppet version? Are you using multiple environments?

superseb · 2015-04-14T14:01:38Z

Oh @poolski , please post your manifest aswell so I can try to reproduce

poolski · 2015-04-14T14:01:48Z

We are running

Puppet version 3.7.5

And yes, we have multiple envs, managed with r10k. I'm testing it on our "develop" env before rolling it out to the wider world.

zakuni · 2015-04-15T01:59:59Z

Hi,
It happens to me also.
keepalive, port, and also redis_reconnect_on_error changes from 'true' to 'false' every time puppet runs.

I'm using sensu-puppet v1.5.5 now, but it've been occured before this version. (sorry I can't tell exactly from which version)

Here is part of my manifest

  class { '::sensu':
    version => '0.17.1-1',
    rabbitmq_ssl_cert_chain => '/etc/sensu/ssl/cert.pem',
    rabbitmq_ssl_private_key => '/etc/sensu/ssl/key.pem',
    rabbitmq_host => $server_host,
    rabbitmq_port => 5671,
    rabbitmq_password => $rabbitmq_pass,
    rabbitmq_vhost => '/sensu',
    purge_config => false,
    server    => $server,
    api       => $server,
    subscriptions => $subscriptions,
    client_custom => {
      'keepalive' => {
        'handlers' => ['default', 'slack']
      }
    },
    use_embedded_ruby => $embedded_ruby,
    sensu_plugin_version => latest,
  }

I've specified 'client_port' and 'redis_reconnect_on_error' explicitly, but it didn't fix.

superseb · 2015-04-15T10:16:45Z

Okay. So:

The client_port is fixed after v1.5.5, so you can use master till @jamtur01 releases a new version.
I just submitted Boolean checking/converting on sensu_redis_config #345 for fixing redis_reconnect_on_error
For using the keepalive, you can use client_keepalive like so:

  class { '::sensu':
    version => '0.17.1-1',
    rabbitmq_ssl_cert_chain => '/etc/sensu/ssl/cert.pem',
    rabbitmq_ssl_private_key => '/etc/sensu/ssl/key.pem',
    rabbitmq_host => $server_host,
    rabbitmq_port => 5671,
    rabbitmq_password => $rabbitmq_pass,
    rabbitmq_vhost => '/sensu',
    purge_config => false,
    server    => $server,
    api       => $server,
    subscriptions => $subscriptions,
    client_keepalive => {
        'handlers' => ['default', 'slack']
      }
    },
    use_embedded_ruby => $embedded_ruby,
    sensu_plugin_version => latest,
  }

Let me know if this solves it for you.

zakuni · 2015-04-16T02:17:32Z

@superseb
Thank you, it solved for me!

superseb · 2015-04-16T08:43:29Z

Okay cool, @poolski if you could post your manifest I can take a look.

poolski · 2015-04-16T08:59:13Z

@superseb, alright, here we go. It's a bit broken up because most of the stuff lives in Hiera and is loaded automatically by class-based params. My r10k Sensu block is applied to all hosts (it's in a 'base' profile) and looks like this:

  # Sensu config
  $subs = hiera_array('sensu::subscriptions')
  $plugs = hiera_array('sensu::plugins')
  $checks = hiera_hash('sensu::checks')
  if $checks { create_resources(sensu::check, $checks) }
  class {'::sensu':
    subscriptions => $subs,
    plugins       => $plugs,
  }

The corresponding data is as follows. Checks have been removed because there are lots of them and they aren't complaining.

{
  "sensu::client": true,
  "sensu::client_custom": {
    "handlers": [
      "default"
    ],
    "keepalive": {
      "high_flap_threshold": "20",
      "low_flap_threshold": "5",
      "refresh": 14400,
      "thresholds": {
        "critical": 120,
        "warning": 90
      }
    }
  },
  "sensu::plugins": [
    "puppet:///modules/sensu_site/plugins/system/check-apt.sh",
    "puppet:///modules/sensu_site/plugins/system/check-cpu.rb",
    "puppet:///modules/sensu_site/plugins/system/check-disk.rb",
    "puppet:///modules/sensu_site/plugins/system/check-load.rb",
    "puppet:///modules/sensu_site/plugins/system/cpu-metrics.rb",
    "puppet:///modules/sensu_site/plugins/processes/check-procs.rb",
    "puppet:///modules/sensu_site/plugins/system/check-ram.rb",
    "puppet:///modules/sensu_site/plugins/sendmail/sendmail-mqueue.rb",
    "puppet:///modules/sensu_site/plugins/enabler/check-pidfile.sh",
    "puppet:///modules/sensu_site/plugins/network/check-ports.sh"
  ],
  "sensu::purge_config": true,
  "sensu::rabbitmq_host": "10.10.10.10",
  "sensu::rabbitmq_password": "passwordz",
  "sensu::rabbitmq_port": 5671,
  "sensu::rabbitmq_ssl": true,
  "sensu::rabbitmq_ssl_cert_chain": "puppet:///modules/sensu_site/ssl/cert.pem",
  "sensu::rabbitmq_ssl_private_key": "puppet:///modules/sensu_site/ssl/key.pem",
  "sensu::rabbitmq_vhost": "/sensu",
  "sensu::rabbitmq_reconnect_on_error": true,
  "sensu::redis_reconnect_on_error": true,
  "sensu::sensu_plugin_version": "present",
  "sensu::subscriptions": [
    "common"
  ],
  "sensu::use_embedded_ruby": true,
}

poolski · 2015-04-16T09:07:44Z

Additionally, having installed master over the v1.5.5 release, I've now got a consistent

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter reconnect_on_error on Sensu_rabbitmq_config[myserver.mynetwork.net] at /etc/puppet/environments/develop/modules/sensu/manifests/rabbitmq/config.pp:120 on node myserver.mynetwork.net
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

superseb · 2015-04-16T09:10:49Z

I'll take a look today, did you restart the master after getting master?

poolski · 2015-04-16T09:38:53Z

I did indeed. I killed Apache and waited for the passenger processes to stop before restarting, too.

superseb · 2015-04-17T19:10:13Z

In a clean Vagrant setup with open source Puppet 3.7.5-1 master+agent and the following declaration, it doesn't reset the properties in every run. Could you verify @poolski ? The issue you are experiencing seems like the old bug with custom providers and environments, but should be fixed in 3.7.5 (same version as I tested it on)

  class { '::sensu':
    client => true,
    client_custom => {
      'handlers' => [
        'default'
       ],
    },
    client_keepalive => {
      "high_flap_threshold" => "20",
      "low_flap_threshold" => "5",
      "refresh" => 14400,
      "thresholds" => {
        "critical" => 120,
        "warning" => 90
      }
    },
    purge_config => true,
    rabbitmq_host => '10.10.10.10',
    rabbitmq_password => 'password',
    rabbitmq_port => 5671,
    rabbitmq_ssl => true,
    rabbitmq_vhost => '/sensu',
    rabbitmq_reconnect_on_error => true,
    redis_reconnect_on_error => true,
    subscriptions => [ 'common' ],
    use_embedded_ruby => true,
  }

superseb · 2015-04-29T16:50:31Z

@poolski Any update on this?

jsfrerot · 2015-05-19T17:47:50Z

Hi,
I applied this patch #345 on v1.5.5 and I still get the Sensu_client_config changing every puppet run.

1st run:
Notice: /Stage[main]/Sensu::Client::Config/Sensu_client_config[my-server]/port: port changed '' to '3030'

2nd run:
Notice: /Stage[main]/Sensu::Client::Config/Sensu_client_config[my-server]/custom: custom changed 'check_load => critical1.66,1.25,1.00warning1.25,1.00,0.80, port => 3030' to 'check_load => critical1.66,1.25,1.00warning1.25,1.00,0.80'

The manifest:

$load_warn=inline_template("<%= sprintf('%.2f',(@processorcount.to_i*1.25)) %>,<%= sprintf('%.2f',(@processorcount.to_i*1)) %>,<%= sprintf('%.2f',(@processorcount.to_i*0.8)) %>")
$load_crit=inline_template("<%= sprintf('%.2f',($processorcount.to_i*1.66)) %>,<%= sprintf('%.2f',($processorcount.to_i*1.25)) %>,<%= sprintf('%.2f',($processorcount.to_i*1)) %>")
class { 'sensu':
    rabbitmq_ssl_private_key => "puppet:///data/sensu/certs/client_key.pem",
    rabbitmq_ssl_cert_chain => "puppet:///data/sensu/certs/client_cert.pem",
    rabbitmq_password => 'xxx',
    rabbitmq_host => 'my-sensu-server',
    rabbitmq_port => 5671,
    rabbitmq_vhost => "/sensu",
    plugins => [
        'puppet:///data/sensu/plugins/system/check-ntp.rb',
        'puppet:///data/sensu/plugins/system/check-disk.rb',
        'puppet:///data/sensu/plugins/system/check-load.rb',
    ],  
    use_embedded_ruby => true,
    sensu_plugin_provider => sensu_gem,
    sensu_plugin_version => 'present',
    install_repo => false,
    client_custom => {
        check_load => {
            warning => $load_warn,
            critical => $load_crit,
        },  
    },  
    subscriptions => ['base']
}   
if $is_virtual == "false" {
    sensu::subscription { 'physical': }
}

jsfrerot · 2015-06-01T20:04:32Z

@superseb, since you were the main contact for this issue, would it be possible to look at my last comment and maybe point me where my problem could be?

Thanks.

superseb · 2015-06-02T12:21:27Z

@jsfrerot Did you only apply #345? Because I think you need to apply #343. Let me know if this solves it.

jsfrerot · 2015-06-08T15:36:59Z

@superseb I just applied #343 and it works as expected. Thank you.

poolski · 2015-06-10T11:25:20Z

@superseb I'm now running v1.5.5 which as I understand it has both #343 and #345 rolled in?

Still experiencing the same issue and something new:
Error 400 on SERVER: Invalid parameter reconnect_on_error on Sensu_rabbitmq_config

Restarting the puppetmaster fixes it but for only one run after which it reverts back to erroring.

superseb · 2015-06-10T12:31:50Z

@poolski No, v1.5.5 doesn't contain those fixes. Let me ping @jamtur01 or @jlambert121 to release a new version.

poolski · 2015-06-10T13:21:43Z

Oh, durp!

Thanks @superseb.

poolski · 2015-06-11T10:52:06Z

@superseb - so another fun fact. I'm testing out using master to see if the bleeding edge code fixes any of my issues. It doesn't seem to - in fact, it introduces another one!

Notice: /Stage[main]/Sensu::Repo::Apt/Apt::Source[sensu]/Apt::Key[Add key: 8911D8FF37778F24B4E726A218609E3D7580C77F from Apt::Source sensu]/Exec[d36677d9164a673e5a4b8cdd005afa63c1c67926]/returns: executed successfully
Notice: /Stage[main]/Sensu::Client::Config/Sensu_client_config[server.network.net]/custom: custom changed 'handlers => ["default"]' to 'handlers => ["default"], keepalive => {"high_flap_threshold"=>20, "low_flap_threshold"=>5, "refresh"=>14400, "thresholds"=>{"critical"=>120, "warning"=>90}}'
Info: Class[Sensu::Client::Config]: Scheduling refresh of Service[sensu-client]

Now it "adds" the repo key every time Puppet runs. Also, even though the port changes have been fixed, it's still resetting keepalives and flap thresholds, even though nothing's changed

superseb · 2015-06-22T15:32:07Z

@poolski Did you split your data to client_custom and client_keepalive? I'll see if I can reproduce your apt::key.

poolski · 2015-06-22T15:47:02Z

@superseb - here's my data:

  "sensu::client_custom": {
    "handlers": [
      "default"
    ],
    "keepalive": {
      "high_flap_threshold": "20",
      "low_flap_threshold": "5",
      "refresh": 14400,
      "thresholds": {
        "critical": 120,
        "warning": 90
      }
    }
  }

superseb · 2015-06-22T15:49:03Z

@poolski Please see example like I posted before, and let me know if this helps.

  class { '::sensu':
    client => true,
    client_custom => {
      'handlers' => [
        'default'
       ],
    },
    client_keepalive => {
      "high_flap_threshold" => "20",
      "low_flap_threshold" => "5",
      "refresh" => 14400,
      "thresholds" => {
        "critical" => 120,
        "warning" => 90
      }
    },
    purge_config => true,
    rabbitmq_host => '10.10.10.10',
    rabbitmq_password => 'password',
    rabbitmq_port => 5671,
    rabbitmq_ssl => true,
    rabbitmq_vhost => '/sensu',
    rabbitmq_reconnect_on_error => true,
    redis_reconnect_on_error => true,
    subscriptions => [ 'common' ],
    use_embedded_ruby => true,
  }

poolski · 2015-06-22T15:49:21Z

That's just what I was about to do @superseb :D

superseb · 2015-06-22T15:49:45Z

@poolski 👍 😄

poolski · 2015-06-22T15:52:28Z

@superseb when I first started using the module, I don't recall there being a dedicated keepalive field - it had to be shoehorned in with client_custom.

superseb · 2015-06-22T15:57:25Z

@poolski Correct, was added in 30ddb26 (v1.3.0)

poolski · 2015-06-22T16:00:22Z

Okay, progress - now all I have is it trying to change the port on me...
Notice: /Stage[main]/Sensu::Client::Config/Sensu_client_config[host.network.net]/port: port changed '' to '3030'
or
Notice: /Stage[main]/Sensu::Client::Config/Sensu_client_config[host.network.net]/custom: custom changed 'handlers => ["default"], port => 3030' to 'handlers => ["default"]'

The above happen with no discernible pattern.

superseb · 2015-06-22T16:02:44Z

Awesome. This was fixed in #342, what codebase are you running on now?

poolski · 2015-06-22T16:12:35Z

v1.5.5

superseb · 2015-06-22T16:16:55Z

Yeah, the fix was merged after the v1.5.5 tag.

poolski · 2015-06-22T16:22:23Z

Alright, I'll try master and see what happens

aamerik · 2015-06-23T18:07:38Z

Can someone suggest to me how to fix "Invalid parameter reconnect_on_error on Sensu_rabbitmq_config"?

using master, client ver 3.7.5, puppetserver 1.0.2, r10k environments. restarted both puppetservers - which fixes the issue for the first run, then back to invalid parameter errors.
tx

aamerik · 2015-06-23T18:13:59Z

Looks like the invalid parameter issue is related to r10k environments. If you want the change to stick you need to update this module in all environments simultaneously and restart the puppetserver.

poolski · 2015-06-30T12:55:06Z

@superseb any idea when the next version will be released?

superseb · 2015-06-30T12:56:50Z

@poolski I can't release a new version, let's ping @jamtur01 @jlambert121 again. Does it work as expected now?

jlambert121 · 2015-06-30T13:31:09Z

This is one issue that I think needs to get closed out before a release. Is this still an issue with the latest master?

jcustenborder · 2015-06-30T13:33:00Z

I believe master as of a week or so ago fixed the issue for me. It'd be
great if someone double checked.

On Tue, Jun 30, 2015, 9:31 AM Justin Lambert notifications@github.com
wrote:

This is one issue that I think needs to get closed out before a release.
Is this still an issue with the latest master?

—
Reply to this email directly or view it on GitHub
#336 (comment).

deepakhj · 2015-07-01T23:50:30Z

I had the same issue with the port changing on every run. Switched from 1.5.5 to master and now it's fixed.

jlambert121 · 2015-07-14T14:57:55Z

I'm going to close this, let us know if this is still an issue.

poolski · 2015-07-15T11:35:50Z

@jlambert121 I'm getting the following error if I switch to using master:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter source on Sensu_check[os-disks] at /etc/puppet/environments/staging/modules/sensu/manifests/check.pp:148 on node mynode.mynetwork.net

What broke between 1.5.5 and master? :(

superseb · 2015-07-15T11:50:18Z

Restarted?

#377 seems to contain a fix for source param issues.

poolski · 2015-07-15T11:58:37Z

Yeah I restarted All The Things.

According to #377, it's been merged into master and fixes that error, but something's still not right.

poolski · 2015-07-15T12:02:14Z

Ok, so, stopping the puppetserver actually seemed to fix it.

Issuing an /etc/init.d/puppetserver restart doesn't seem to flush whatever it's caching as effectively.

Might be worth noting in docs somewhere?

devshorts · 2015-08-21T00:02:07Z

Just chiming in here, any ideas when this will get rolled into an official release?

jlambert121 · 2015-08-21T14:28:51Z

@devshorts I keep hoping to get a 2.0 release done real soon. Enterprise support looks like it's pretty much done.

cintiadr · 2015-09-07T07:53:20Z

It took me a while to understand that this issue was closed but not yet released.

I will be using commit cf40de6, as I won't be able to get apt 2.0 any time soon :/, caused by #411

Looking at the commits, I believe that's the first breaking change after 1.5.5.

jlambert121 closed this as completed Jul 14, 2015

rickatnight11 mentioned this issue Jul 16, 2015

Invalid package provider 'sensu_gem' #383

Closed

elfranne mentioned this issue Nov 22, 2022

[question] Keepalive check custom parameters #1329

Open

sensu::client::config keepalives 'change' every run #336

sensu::client::config keepalives 'change' every run #336

Comments

poolski commented Mar 25, 2015

jlambert121 commented Mar 25, 2015

poolski commented Mar 25, 2015

poolski commented Apr 1, 2015

superseb commented Apr 10, 2015

superseb commented Apr 13, 2015

poolski commented Apr 14, 2015

poolski commented Apr 14, 2015

superseb commented Apr 14, 2015

poolski commented Apr 14, 2015

superseb commented Apr 14, 2015

superseb commented Apr 14, 2015

poolski commented Apr 14, 2015

zakuni commented Apr 15, 2015

superseb commented Apr 15, 2015

zakuni commented Apr 16, 2015

superseb commented Apr 16, 2015

poolski commented Apr 16, 2015

poolski commented Apr 16, 2015

superseb commented Apr 16, 2015

poolski commented Apr 16, 2015

superseb commented Apr 17, 2015

superseb commented Apr 29, 2015

jsfrerot commented May 19, 2015

jsfrerot commented Jun 1, 2015

superseb commented Jun 2, 2015

jsfrerot commented Jun 8, 2015

poolski commented Jun 10, 2015

superseb commented Jun 10, 2015

poolski commented Jun 10, 2015

poolski commented Jun 11, 2015

superseb commented Jun 22, 2015

poolski commented Jun 22, 2015

superseb commented Jun 22, 2015

poolski commented Jun 22, 2015

superseb commented Jun 22, 2015

poolski commented Jun 22, 2015

superseb commented Jun 22, 2015

poolski commented Jun 22, 2015

superseb commented Jun 22, 2015

poolski commented Jun 22, 2015

superseb commented Jun 22, 2015

poolski commented Jun 22, 2015

aamerik commented Jun 23, 2015

aamerik commented Jun 23, 2015

poolski commented Jun 30, 2015

superseb commented Jun 30, 2015

jlambert121 commented Jun 30, 2015

jcustenborder commented Jun 30, 2015

deepakhj commented Jul 1, 2015

jlambert121 commented Jul 14, 2015

poolski commented Jul 15, 2015

superseb commented Jul 15, 2015

poolski commented Jul 15, 2015

poolski commented Jul 15, 2015

devshorts commented Aug 21, 2015

jlambert121 commented Aug 21, 2015

cintiadr commented Sep 7, 2015