Sensu 0.23 cannot connect to Redis with same settings as 0.22 #1209
Comments
I'm experiencing the exact same issue. |
Also seeing the same thing. Running Redhat 6.7 and managing everything via the sensu puppet module. Manually restarting the service throws notices that it's ignoring certain gems:
|
This seems to work for me on Ubuntu 14.04. |
I'm using the embedded-ruby, Ubuntu 14.04, Sensu from repo (puppet sensu module). |
@spacepants using the embedded ruby? |
@GhostLyrics so sensu services are able to initially connect to Redis, but they are having troubles reconnecting to Redis? |
@portertech Yep. |
I'll try to reproduce now. |
@spacepants it seems that you need to reinstall those gems with the embedded Ruby 2.3 to build the extensions. |
@GhostLyrics do you know why the Redis reconnect is triggered? I am unable to reproduce by stopping/kill Redis, the API successfully reconnects. |
@portertech care to continue in #sensu IRC? Otherwise I'm happy to answer the questions here, as far as I can. No, I don't know what's causing the reconnect. There are no additional log statements aside from that and Redis is on the same machine. |
Hi, In my case sensu-server did not connect to Redis at all (checked with tcpdump) no matter if redis_reconnect_on_error was true or false. I got this in my log:
|
We got bit by this today too. Thankfully it was in a pre-production environment. Practically the same exact error as @mnellemann. We rolled back to 0.22.x and everything works as expected. A learned lesson to pin to a specific version. If it helps, we are also using puppet-sensu (which I see at least one other commenter is using too) |
After doing some debugging with @GhostLyrics on IRC, it seems that the failure is linked to Redis definition attributes. We're working through the attributes that the puppet module sets by default. |
Looks like the cause was |
@mnellemann try changing |
Fixed the redis issue for me as well. The only way I was able to clear the gem issue was by manually clearing out those gems. The 2.2.0 gems were still being sourced somehow in the embedded ruby path. Working now, though. |
@spacepants another sensu-build bug, init script. |
Users, please run:
If it resolves to "127.0.0.1", you're OK. If it resolves to "::1", Redis is probably not listening with IPv6. |
Hi, The resolv test suggested returned Thanks everybody for quick fix :) |
@spacepants 0.23.0-3 packages are now available in the repositories, fixing the GEM_PATH issues. |
Adding the configuration and documentation labels to indicate that is a configuration issue, which we should document. See: sensu/sensu-docs#347 |
If I may, until this is fixed (if it can be in ruby), could a note be put in the documentation to avoid 2 days of frustration? I almost ditched sensu entirely because of this re: localhost vs 127.0.0.1 |
@asciifaceman yes, we should have a documentation update up by tomorrow morning at the latest. Our apologies for the frustration. Unfortunately this isn't an issue with Sensu itself as much as a general configuration/operation issue, which we didn't catch in our tests. In this case, Sensu resolving |
@asciifaceman I think this should be addressed in sensu/sensu-docs@a9ac5eb, but let me know if that doesn't make sense to you. Barring any further input, this should get shipped as part of sensu/sensu-docs#346 (hopefully tomorrow morning). |
I think it is logical since ruby will likely always default to ::1 for localhost (because screw legacy infrastructures! or something), which I do not agree with but c`est la vie. |
@asciifaceman cool, thanks for the quick reply! And sorry again that you got stung by this. #monitoringlove |
Please make a note in your documentation as this wasted so much of my time! I never question official documentation. If it wasnt for a godaddy guide incorrectly labeled as a CentOS7 install guide when it actually had a EL6 repo. First impressions... http://serverfault.com/questions/768921 |
The installation guides have been updated with warnings about IPv6 localhost resolution; e.g.: https://sensuapp.org/docs/latest/install-redis-on-rhel-centos#example-standalone-configuration |
@GhostLyrics et al, I'm going to go ahead and close this issue as being resolved via documentation (e.g. https://sensuapp.org/docs/latest/install-redis-on-rhel-centos#example-standalone-configuration). Please feel free to comment and/or open a new issue if anyone feels like this matter is not resolved. #monitoringlove |
I think localhost should be changed back to 127.0.0.1 by default but possibly changed to IPv6 as an option as I imagine there are plenty of guides for Sensu on the net that use localhost and I personally did not find this easy to resolve and it appears people are still encountering this as an issue. |
Just ran in to this problem myself, and it doesn't seem like an issue with Sensu at all. I was able to resolve it in one of two ways: Update _OR_ Update If you're running an IPv6-capable host, the latter option is probably the way to go. Hope this helps! |
Hello. I have a problem with sensu-api. It can't bind to port. Му sensu-api config
I can't see bind 4567 port.
I was see problem with strace. But redis was started and redis-cli worked. My sensu version |
Hi @octogeek, sorry to hear you are having trouble with the API. The problem you describe seems like it might be similar but not the same as the condition described by others in this issue. I suspect that redis is not accepting the connection from the API. Can you please verify that redis-server is either listening on all interfaces (e.g. 0.0.0.0, which would include 127.0.0.1), or listening on 127.0.0.1 explicitly? |
Hi @cwjohnston, I have the problem only with ubuntu linux. I was migrate to centos7 ( It is my real production environment ) and problems were resolve. But in ubuntu linux the redis was replied on ping command |
Oops! I have the same problem after restarting sensu-api on centos linux. I uptated sensu package to the version 0.24, but it did not solve my problem. Sorry. It was my mistake. I changed linux hostname on the rabbitmq server and it was the problem. |
This should not be closed. It is still an issue in Sensu 25. Follow documentation here: https://sensuapp.org/docs/0.25/installation/install-redis-on-ubuntu-debian.html Does not work. But changing redis host to "localhost" and the issue is resolved. Lost most of a day to this. I wish Sensu had better error logging. |
…liqrtech.com. Changed config.json to use 127.0.0.1 instead of localhost per this issues page: sensu/sensu#1209
Sensu API hangs on reconnect to Redis. Same goes for Sensu-server which also gets stuck reconnecting.
/etc/sensu/conf.d/redis.json
:Log output with
0.23
, on Ubuntu Trusty, from the sensu repository,/var/log/sensu/sensu-api.log
:With
0.22
:The text was updated successfully, but these errors were encountered: