Skip to content
This repository has been archived by the owner on Jan 1, 2020. It is now read-only.

Sensu 0.23 cannot connect to Redis with same settings as 0.22 #1209

Closed
GhostLyrics opened this issue Apr 6, 2016 · 37 comments
Closed

Sensu 0.23 cannot connect to Redis with same settings as 0.22 #1209

GhostLyrics opened this issue Apr 6, 2016 · 37 comments

Comments

@GhostLyrics
Copy link

Sensu API hangs on reconnect to Redis. Same goes for Sensu-server which also gets stuck reconnecting.

/etc/sensu/conf.d/redis.json:

{
  "redis": {
    "port": 6379,
    "host": "localhost",
    "reconnect_on_error": true,
    "db": 0,
    "auto_reconnect": true
  }
}

Log output with 0.23, on Ubuntu Trusty, from the sensu repository, /var/log/sensu/sensu-api.log:

{"timestamp":"2016-04-06T14:40:27.949838+0200","level":"warn","message":"config file does not exist or is not readable","file":"/etc/sensu/config.json"}
{"timestamp":"2016-04-06T14:40:27.950089+0200","level":"warn","message":"ignoring config file","file":"/etc/sensu/config.json"}
{"timestamp":"2016-04-06T14:40:27.950281+0200","level":"warn","message":"loading config files from directory","directory":"/etc/sensu/conf.d"}
{"timestamp":"2016-04-06T14:40:27.950347+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/client.json"}
{"timestamp":"2016-04-06T14:40:27.950409+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/redis.json"}
{"timestamp":"2016-04-06T14:40:27.950513+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/redis.json","changes":{"redis":[null,{"port":6379,"host":"localhost","reconnect_on_error":true,"db":0,"auto_reconnect":true}]}}
{"timestamp":"2016-04-06T14:40:27.950586+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/handlers/default.json"}
{"timestamp":"2016-04-06T14:40:27.950691+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/handlers/default.json","changes":{"handlers":{"default":[null,{"command":"cat","type":"pipe","filters":[],"severities":["ok","warning","critical","unknown"]}]}}}
{"timestamp":"2016-04-06T14:40:27.950758+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/api.json"}
{"timestamp":"2016-04-06T14:40:27.950846+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/api.json","changes":{"api":[null,{"port":4567,"host":"localhost","bind":"0.0.0.0"}]}}
{"timestamp":"2016-04-06T14:40:27.950912+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/checks/check_reboot_required.json"}
{"timestamp":"2016-04-06T14:40:27.951089+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/checks/check_reboot_required.json","changes":{"checks":{"check_reboot_required":[null,{"command":"check-file-exists.rb -w /var/run/reboot-required","handlers":["default"],"interval":60,"subscribers":["all"],"standalone":true}]}}}
{"timestamp":"2016-04-06T14:40:27.951162+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/checks/check_puppet_last_run.json"}
{"timestamp":"2016-04-06T14:40:27.951263+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/checks/check_puppet_last_run.json","changes":{"checks":{"check_puppet_last_run":[null,{"command":"sudo /opt/sensu/embedded/bin/check-puppet-last-run.rb --warn-age 600 --crit-age 1200 --summary-file /var/lib/puppet/state/last_run_summary.yaml","handlers":["default"],"interval":60,"subscribers":["all"],"standalone":true}]}}}
{"timestamp":"2016-04-06T14:40:27.951400+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/checks/check_disk_usage.json"}
{"timestamp":"2016-04-06T14:40:27.951499+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/checks/check_disk_usage.json","changes":{"checks":{"check_disk_usage":[null,{"command":"check-disk-usage.rb -p '.*gvfs'","handlers":["default"],"interval":60,"subscribers":["all"],"standalone":true}]}}}
{"timestamp":"2016-04-06T14:40:27.951587+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/checks/check_ntp_offset.json"}
{"timestamp":"2016-04-06T14:40:27.951712+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/checks/check_ntp_offset.json","changes":{"checks":{"check_ntp_offset":[null,{"command":"check-ntp.rb","handlers":["default"],"interval":60,"subscribers":["all"],"standalone":true}]}}}
{"timestamp":"2016-04-06T14:40:27.951798+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/rabbitmq.json"}
{"timestamp":"2016-04-06T14:40:27.951941+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/rabbitmq.json","changes":{"rabbitmq":[null,{"port":5672,"host":"localhost","user":"sensu","password":"REDACTED","vhost":"sensu","reconnect_on_error":true,"prefetch":1}]}}
{"timestamp":"2016-04-06T14:40:28.289889+0200","level":"warn","message":"reconnecting to redis"}
{"timestamp":"2016-04-06T14:40:28.296188+0200","level":"info","message":"api listening","bind":"0.0.0.0","port":4567}
{"timestamp":"2016-04-06T14:40:33.302822+0200","level":"info","message":"GET /stashes","remote_address":"127.0.0.1","user_agent":"Go-http-client/1.1","request_method":"GET","request_uri":"/stashes","request_body":""}
{"timestamp":"2016-04-06T14:40:36.213250+0200","level":"info","message":"GET /stashes","remote_address":"127.0.0.1","user_agent":"Go-http-client/1.1","request_method":"GET","request_uri":"/stashes","request_body":""}
// many more GETs

With 0.22:

{"timestamp":"2016-04-06T14:43:56.846987+0200","level":"warn","message":"config file does not exist or is not readable","file":"/etc/sensu/config.json"}
{"timestamp":"2016-04-06T14:43:56.847271+0200","level":"warn","message":"ignoring config file","file":"/etc/sensu/config.json"}
{"timestamp":"2016-04-06T14:43:56.847369+0200","level":"warn","message":"loading config files from directory","directory":"/etc/sensu/conf.d"}
{"timestamp":"2016-04-06T14:43:56.847485+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/client.json"}
{"timestamp":"2016-04-06T14:43:56.847583+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/redis.json"}
{"timestamp":"2016-04-06T14:43:56.847729+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/redis.json","changes":{"redis":[null,{"port":6379,"host":"localhost","reconnect_on_error":true,"db":0,"auto_reconnect":true}]}}
{"timestamp":"2016-04-06T14:43:56.847948+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/handlers/default.json"}
{"timestamp":"2016-04-06T14:43:56.848089+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/handlers/default.json","changes":{"handlers":{"default":[null,{"command":"cat","type":"pipe","filters":[],"severities":["ok","warning","critical","unknown"]}]}}}
{"timestamp":"2016-04-06T14:43:56.848189+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/api.json"}
{"timestamp":"2016-04-06T14:43:56.848301+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/api.json","changes":{"api":[null,{"port":4567,"host":"localhost","bind":"0.0.0.0"}]}}
{"timestamp":"2016-04-06T14:43:56.848418+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/checks/check_reboot_required.json"}
{"timestamp":"2016-04-06T14:43:56.848585+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/checks/check_reboot_required.json","changes":{"checks":{"check_reboot_required":[null,{"command":"check-file-exists.rb -w /var/run/reboot-required","handlers":["default"],"interval":60,"subscribers":["all"],"standalone":true}]}}}
{"timestamp":"2016-04-06T14:43:56.848704+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/checks/check_puppet_last_run.json"}
{"timestamp":"2016-04-06T14:43:56.849012+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/checks/check_puppet_last_run.json","changes":{"checks":{"check_puppet_last_run":[null,{"command":"sudo /opt/sensu/embedded/bin/check-puppet-last-run.rb --warn-age 600 --crit-age 1200 --summary-file /var/lib/puppet/state/last_run_summary.yaml","handlers":["default"],"interval":60,"subscribers":["all"],"standalone":true}]}}}
{"timestamp":"2016-04-06T14:43:56.849168+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/checks/check_disk_usage.json"}
{"timestamp":"2016-04-06T14:43:56.849296+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/checks/check_disk_usage.json","changes":{"checks":{"check_disk_usage":[null,{"command":"check-disk-usage.rb -p '.*gvfs'","handlers":["default"],"interval":60,"subscribers":["all"],"standalone":true}]}}}
{"timestamp":"2016-04-06T14:43:56.849455+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/checks/check_ntp_offset.json"}
{"timestamp":"2016-04-06T14:43:56.849677+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/checks/check_ntp_offset.json","changes":{"checks":{"check_ntp_offset":[null,{"command":"check-ntp.rb","handlers":["default"],"interval":60,"subscribers":["all"],"standalone":true}]}}}
{"timestamp":"2016-04-06T14:43:56.849823+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/rabbitmq.json"}
{"timestamp":"2016-04-06T14:43:56.849964+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/rabbitmq.json","changes":{"rabbitmq":[null,{"port":5672,"host":"localhost","user":"sensu","password":"REDACTED","vhost":"sensu","reconnect_on_error":true,"prefetch":1}]}}
{"timestamp":"2016-04-06T14:43:56.940401+0200","level":"info","message":"api listening","bind":"0.0.0.0","port":4567}
{"timestamp":"2016-04-06T14:44:01.224244+0200","level":"info","message":"GET /stashes","remote_address":"127.0.0.1","user_agent":"Go-http-client/1.1","request_method":"GET","request_uri":"/stashes","request_body":""}
{"timestamp":"2016-04-06T14:44:01.246050+0200","level":"info","message":"GET /checks","remote_address":"127.0.0.1","user_agent":"Go-http-client/1.1","request_method":"GET","request_uri":"/checks","request_body":""}
{"timestamp":"2016-04-06T14:44:01.249213+0200","level":"info","message":"GET /clients","remote_address":"127.0.0.1","user_agent":"Go-http-client/1.1","request_method":"GET","request_uri":"/clients?limit=1000&offset=0","request_body":""}
{"timestamp":"2016-04-06T14:44:01.259487+0200","level":"info","message":"GET /events","remote_address":"127.0.0.1","user_agent":"Go-http-client/1.1","request_method":"GET","request_uri":"/events","request_body":""}
{"timestamp":"2016-04-06T14:44:01.281672+0200","level":"info","message":"GET /info","remote_address":"127.0.0.1","user_agent":"Go-http-client/1.1","request_method":"GET","request_uri":"/info","request_body":""}
{"timestamp":"2016-04-06T14:44:01.286215+0200","level":"info","message":"GET /aggregates","remote_address":"127.0.0.1","user_agent":"Go-http-client/1.1","request_method":"GET","request_uri":"/aggregates","request_body":""}
// more GET blocks (stashes until aggregates)
@mnellemann
Copy link

I'm experiencing the exact same issue.

@spacepants
Copy link

Also seeing the same thing. Running Redhat 6.7 and managing everything via the sensu puppet module.

Manually restarting the service throws notices that it's ignoring certain gems:

Stopping sensu-server [ OK ]
Starting sensu-serverIgnoring json-1.8.2 because its extensions are not built. Try: gem pristine json --version 1.8.2
Ignoring nokogiri-1.6.6.2 because its extensions are not built. Try: gem pristine nokogiri --version 1.6.6.2
Ignoring unf_ext-0.0.7.1 because its extensions are not built. Try: gem pristine unf_ext --version 0.0.7.1
Ignoring vmstat-2.1.0 because its extensions are not built. Try: gem pristine vmstat --version 2.1.0
Ignoring zookeeper-1.4.11 because its extensions are not built. Try: gem pristine zookeeper --version 1.4.11
[ OK ]

@runningman84
Copy link
Contributor

This seems to work for me on Ubuntu 14.04.

@mnellemann
Copy link

I'm using the embedded-ruby, Ubuntu 14.04, Sensu from repo (puppet sensu module).

@portertech
Copy link
Member

@spacepants using the embedded ruby?

@portertech
Copy link
Member

@GhostLyrics so sensu services are able to initially connect to Redis, but they are having troubles reconnecting to Redis?

@spacepants
Copy link

@portertech Yep.

@portertech
Copy link
Member

I'll try to reproduce now.

@portertech
Copy link
Member

@spacepants it seems that you need to reinstall those gems with the embedded Ruby 2.3 to build the extensions.

@portertech
Copy link
Member

@GhostLyrics do you know why the Redis reconnect is triggered? I am unable to reproduce by stopping/kill Redis, the API successfully reconnects.

@GhostLyrics
Copy link
Author

@portertech care to continue in #sensu IRC? Otherwise I'm happy to answer the questions here, as far as I can.

No, I don't know what's causing the reconnect. There are no additional log statements aside from that and Redis is on the same machine.

@mnellemann
Copy link

Hi,

In my case sensu-server did not connect to Redis at all (checked with tcpdump) no matter if redis_reconnect_on_error was true or false. I got this in my log:

{"timestamp":"2016-04-06T06:45:44.570372+0200","level":"warn","message":"loading config file","file":"/etc/sensu/conf.d/redis.json"} {"timestamp":"2016-04-06T06:45:44.570418+0200","level":"warn","message":"config file applied changes","file":"/etc/sensu/conf.d/redis.json","changes":{"redis":[null,{"port":6379,"host":"localhost","reconnect_on_error":false,"db":0,"auto_reconnect":true}]}} {"timestamp":"2016-04-06T06:45:44.585236+0200","level":"warn","message":"loading extension files from directory","directory":"/etc/sensu/extensions"} {"timestamp":"2016-04-06T06:45:44.585321+0200","level":"warn","message":"loaded extension","type":"mutator","name":"ruby_hash","description":"returns ruby hash event data"} {"timestamp":"2016-04-06T06:45:44.585362+0200","level":"warn","message":"loaded extension","type":"mutator","name":"json","description":"returns JSON formatted event data"} {"timestamp":"2016-04-06T06:45:44.585399+0200","level":"warn","message":"loaded extension","type":"mutator","name":"only_check_output","description":"returns check output"} {"timestamp":"2016-04-06T06:45:44.585437+0200","level":"warn","message":"loaded extension","type":"handler","name":"debug","description":"returns raw event data"} {"timestamp":"2016-04-06T06:45:44.695397+0200","level":"error","message":"redis connection error","error":"unable to connect to redis server"}

@dmizelle
Copy link

dmizelle commented Apr 6, 2016

We got bit by this today too. Thankfully it was in a pre-production environment. Practically the same exact error as @mnellemann.

We rolled back to 0.22.x and everything works as expected. A learned lesson to pin to a specific version. If it helps, we are also using puppet-sensu (which I see at least one other commenter is using too)

@portertech
Copy link
Member

After doing some debugging with @GhostLyrics on IRC, it seems that the failure is linked to Redis definition attributes. We're working through the attributes that the puppet module sets by default.

@portertech
Copy link
Member

Looks like the cause was "host": "localhost", changing it to "host": "127.0.0.1" fixed it for @GhostLyrics. This is the default value https://github.com/sensu/sensu-redis/blob/master/lib/sensu/redis.rb#L62

@portertech
Copy link
Member

@mnellemann try changing "host": "localhost" to "host": "127.0.0.1".

@spacepants
Copy link

Fixed the redis issue for me as well.

The only way I was able to clear the gem issue was by manually clearing out those gems. The 2.2.0 gems were still being sourced somehow in the embedded ruby path. Working now, though.

@portertech
Copy link
Member

@spacepants another sensu-build bug, init script.

@portertech
Copy link
Member

portertech commented Apr 6, 2016

Users, please run:

/opt/sensu/embedded/bin/ruby -rresolv -e 'puts Resolv.getaddress "localhost"'

If it resolves to "127.0.0.1", you're OK. If it resolves to "::1", Redis is probably not listening with IPv6.

@mnellemann
Copy link

Hi,

The resolv test suggested returned ::1 for me.
Setting sensu::redis_host: '127.0.0.1' fixed it for me as well.

Thanks everybody for quick fix :)

@portertech
Copy link
Member

@spacepants 0.23.0-3 packages are now available in the repositories, fixing the GEM_PATH issues.

@calebhailey
Copy link

Adding the configuration and documentation labels to indicate that is a configuration issue, which we should document. See: sensu/sensu-docs#347

@asciifaceman
Copy link

If I may, until this is fixed (if it can be in ruby), could a note be put in the documentation to avoid 2 days of frustration? I almost ditched sensu entirely because of this

re: localhost vs 127.0.0.1

@calebhailey
Copy link

@asciifaceman yes, we should have a documentation update up by tomorrow morning at the latest. Our apologies for the frustration. Unfortunately this isn't an issue with Sensu itself as much as a general configuration/operation issue, which we didn't catch in our tests. In this case, Sensu resolving localhost to ::1 isn't actually incorrect (as Sensu does support IPv6), but if Redis isn't configured to listen on IPv6 it obviously won't work.

@calebhailey
Copy link

@asciifaceman I think this should be addressed in sensu/sensu-docs@a9ac5eb, but let me know if that doesn't make sense to you. Barring any further input, this should get shipped as part of sensu/sensu-docs#346 (hopefully tomorrow morning).

@asciifaceman
Copy link

I think it is logical since ruby will likely always default to ::1 for localhost (because screw legacy infrastructures! or something), which I do not agree with but c`est la vie.

@calebhailey
Copy link

@asciifaceman cool, thanks for the quick reply! And sorry again that you got stung by this. #monitoringlove

@AirCombat
Copy link

Please make a note in your documentation as this wasted so much of my time! I never question official documentation. If it wasnt for a godaddy guide incorrectly labeled as a CentOS7 install guide when it actually had a EL6 repo. First impressions... http://serverfault.com/questions/768921

@calebhailey
Copy link

The installation guides have been updated with warnings about IPv6 localhost resolution; e.g.: https://sensuapp.org/docs/latest/install-redis-on-rhel-centos#example-standalone-configuration

@calebhailey
Copy link

@GhostLyrics et al, I'm going to go ahead and close this issue as being resolved via documentation (e.g. https://sensuapp.org/docs/latest/install-redis-on-rhel-centos#example-standalone-configuration).

Please feel free to comment and/or open a new issue if anyone feels like this matter is not resolved. #monitoringlove

@AirCombat
Copy link

I think localhost should be changed back to 127.0.0.1 by default but possibly changed to IPv6 as an option as I imagine there are plenty of guides for Sensu on the net that use localhost and I personally did not find this easy to resolve and it appears people are still encountering this as an issue.

@dzunk
Copy link

dzunk commented May 11, 2016

Just ran in to this problem myself, and it doesn't seem like an issue with Sensu at all. I was able to resolve it in one of two ways:

Update /etc/hosts to resolve localhost to 127.0.0.1 instead of ::1.

_OR_

Update redis.conf to bind to ::1 in addition to 127.0.0.1.

If you're running an IPv6-capable host, the latter option is probably the way to go. Hope this helps!

@agareev
Copy link

agareev commented Jun 5, 2016

Hello. I have a problem with sensu-api. It can't bind to port.
My sensu-redis config
{ "redis": { "host": "127.0.0.1", "port": 6379 },

Му sensu-api config

{ "api": { "host": "127.0.0.1", "bind": "0.0.0.0", "port": 4567 } }

I can't see bind 4567 port.
>ss -lntp|grep 4567

is nothing

I was see problem with strace.
-1 EAGAIN (Resource temporarily unavailable)

But redis was started and redis-cli worked.

My sensu version
ubuntu package 0.23.3-1

@cwjohnston
Copy link
Contributor

Hi @octogeek, sorry to hear you are having trouble with the API. The problem you describe seems like it might be similar but not the same as the condition described by others in this issue.

I suspect that redis is not accepting the connection from the API. Can you please verify that redis-server is either listening on all interfaces (e.g. 0.0.0.0, which would include 127.0.0.1), or listening on 127.0.0.1 explicitly?

@agareev
Copy link

agareev commented Jun 10, 2016

Hi @cwjohnston, I have the problem only with ubuntu linux. I was migrate to centos7 ( It is my real production environment ) and problems were resolve.

But in ubuntu linux the redis was replied on ping command
>redis-cli ping
>PONG

@agareev
Copy link

agareev commented Jun 10, 2016

Oops! I have the same problem after restarting sensu-api on centos linux. I uptated sensu package to the version 0.24, but it did not solve my problem.

Sorry. It was my mistake. I changed linux hostname on the rabbitmq server and it was the problem.

@z-matth
Copy link

z-matth commented Aug 2, 2016

This should not be closed. It is still an issue in Sensu 25. Follow documentation here: https://sensuapp.org/docs/0.25/installation/install-redis-on-ubuntu-debian.html

Does not work. But changing redis host to "localhost" and the issue is resolved. Lost most of a day to this. I wish Sensu had better error logging.

mdavis6890 pushed a commit to datacenter/cloudcenter-content that referenced this issue Sep 15, 2016
…liqrtech.com. Changed config.json to use 127.0.0.1 instead of localhost per this issues page: sensu/sensu#1209
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests