Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

redis cluster doesn't failover automatically #1047

Closed
frafra opened this issue Apr 10, 2013 · 6 comments
Closed

redis cluster doesn't failover automatically #1047

frafra opened this issue Apr 10, 2013 · 6 comments

Comments

@frafra
Copy link

frafra commented Apr 10, 2013

This is what I get when I stop the master.

[8465] 10 Apr 22:05:43.433 * Caching the disconnected master state.
[8465] 10 Apr 22:05:43.769 * Connecting to MASTER...
[8465] 10 Apr 22:05:43.769 * MASTER <-> SLAVE sync started
[8465] 10 Apr 22:05:43.769 # Error condition on socket for SYNC: Connection refused
[8465] 10 Apr 22:05:44.771 * Connecting to MASTER...
[8465] 10 Apr 22:05:44.771 * MASTER <-> SLAVE sync started
[8465] 10 Apr 22:05:44.772 # Error condition on socket for SYNC: Connection refused
[8465] 10 Apr 22:05:45.774 * Connecting to MASTER...
[8465] 10 Apr 22:05:45.775 * MASTER <-> SLAVE sync started
[8465] 10 Apr 22:05:45.775 # Error condition on socket for SYNC: Connection refused
[8465] 10 Apr 22:05:46.777 * Connecting to MASTER...
[8465] 10 Apr 22:05:46.777 * MASTER <-> SLAVE sync started
[8465] 10 Apr 22:05:46.777 # Error condition on socket for SYNC: Connection refused
[8465] 10 Apr 22:05:47.779 * Connecting to MASTER...
[8465] 10 Apr 22:05:47.780 * MASTER <-> SLAVE sync started
[8465] 10 Apr 22:05:47.780 # Cluster state changed: fail
[8465] 10 Apr 22:05:47.780 # Error condition on socket for SYNC: Connection refused
[8465] 10 Apr 22:05:48.782 * Connecting to MASTER...
[8465] 10 Apr 22:05:48.783 * MASTER <-> SLAVE sync started
[8465] 10 Apr 22:05:48.783 # Error condition on socket for SYNC: Connection refused

...and it continues like this.
This is a test using localhost. I've tried to use one maser and one slave and one master and two slaves, but the output is the same.

This is my redis.conf file:

port 1234
cluster-enabled yes
cluster-config-file nodes1.conf
cluster-node-timeout 5
appendonly yes

The other redis.conf files are just like this, but the port and the config-cluster-file are different.

This is the cluster nodes output on the slave before the failure:

redis 127.0.0.1:1235> cluster nodes
ffc6464df68df47864fde0ea9eb4d9d9191dc1b1 127.0.0.1:1234 master - 1365624792 1365624792 connected 0-16383
c785f0d5f4752aab965f39c09d66cb239d51772f 127.0.0.1:1236 slave ffc6464df68df47864fde0ea9eb4d9d9191dc1b1 1365624793 1365624793 connected
b37d87bb080a9021bc265f99c9f114005d8ef0f2 :0 myself,slave ffc6464df68df47864fde0ea9eb4d9d9191dc1b1 0 1365624792 connected

This is the cluster nodes output on the slave after the failure:

redis 127.0.0.1:1235> cluster nodes
ffc6464df68df47864fde0ea9eb4d9d9191dc1b1 127.0.0.1:1234 master,fail? - 1365624843 1365624807 disconnected 0-16383
c785f0d5f4752aab965f39c09d66cb239d51772f 127.0.0.1:1236 slave ffc6464df68df47864fde0ea9eb4d9d9191dc1b1 1365624843 1365624843 connected
b37d87bb080a9021bc265f99c9f114005d8ef0f2 :0 myself,slave ffc6464df68df47864fde0ea9eb4d9d9191dc1b1 0 1365624840 connected

@antirez
Copy link
Contributor

antirez commented Apr 10, 2013

Hello, for the failover to work, you need to have at least three masters. This is the minimum required, because master consensus is needed both for failure detection and promotion authorization. Thank for your experiments with Redis Cluster! That's very appreciated.

@antirez antirez closed this as completed Apr 10, 2013
@frafra
Copy link
Author

frafra commented Apr 10, 2013

I'm so sorry, I didn't got this point :) I hope that this issue could help someone else.
Thank you for your work! :)

@antirez
Copy link
Contributor

antirez commented Apr 10, 2013

It was impossible for you to know that because there is no documentation at all... :-) Sorry for that!

@ghost
Copy link

ghost commented May 14, 2013

Hi, in the future it will be possible to do failover with two master nodes?

@antirez
Copy link
Contributor

antirez commented May 18, 2013

Still not clear, maybe but it's a dangerous setup.

Cheers,
Salvatore

@atul86244
Copy link

I am trying to setup a redis cluster on two centos VMs I created on my windows 7 PC. I keep getting below error:
When I try from node 1:
[root@localhost src]# ./redis-trib.rb create 127.0.0.1:6380 127.0.0.1:6379

Creating cluster
Connecting to node 127.0.0.1:6380: OK
Connecting to node 127.0.0.1:6379: [ERR] Sorry, can't connect to node 127.0.0.1:6379
When I try from node2:
[root@localhost src]# ./redis-trib.rb create 127.0.0.1:6379 127.0.0.1:6380
Creating cluster
Connecting to node 127.0.0.1:6379: OK
Connecting to node 127.0.0.1:6380: [ERR] Sorry, can't connect to node 127.0.0.1:6380
I can confirm redis is running in cluster mode on both the nodes and I have disabled firewalls on both. I cant figure out what's wrong here :(.
Please help.

yossigo added a commit to yossigo/redis that referenced this issue May 30, 2023
b6a052fe0 Helper for setting TCP_USER_TIMEOUT socket option (redis#1188)
3fa9b6944 Add RedisModule adapter (redis#1182)
d13c091e9 Fix wincrypt symbols conflict
5d84c8cfd Add a test ensuring we don't clobber connection error.
3f95fcdae Don't attempt to set a timeout if we are in an error state.
aacb84b8d Fix typo in makefile.
563b062e3 Accept -nan per the RESP3 spec recommendation.
04c1b5b02 Fix colliding option values
4ca8e73f6 Rework searching for openssl
cd208812f Attempt to find the correct path for openssl.
011f7093c Allow specifying the keepalive interval
e9243d4f7 Cmake static or shared (redis#1160)
1cbd5bc76 Write a version file for the CMake package (redis#1165)
6f5bae8c6 fix typo
acd09461d CMakeLists.txt: respect BUILD_SHARED_LIBS
97fcf0fd1 Add sdevent adapter
ccff093bc Bump dev version for the next release cycle.
c14775b4e Prepare for v1.1.0 GA
f0bdf8405 Add support for nan in RESP3 double (redis#1133)
991b0b0b3 Add an example that calls redisCommandArgv (redis#1140)
a36686f84 CI updates (redis#1139)
8ad4985e9 fix flag reference
7583ebb1b Make freeing a NULL redisAsyncContext a no op.
2c53dea7f Update version in dev branch.
f063370ed Prepare for v1.1.0-rc1
2b069573a CI fixes in preparation of release
e1e9eb40d Add author information to release-drafter template.
afc29ee1a Update for mingw cross compile
ceb8a8815 fixed cpp build error with adapters/libhv.h
3b15a04b5 Fixup of PR734: Coverage of hiredis.c (redis#1124)
c245df9fb CMake corrections for building on Windows (redis#1122)
9c338a598 Fix PUSH handler tests for Redis >= 7.0.5
6d5c3ee74 Install on windows fixes (redis#1117)
68b29e1ad Add timeout support to libhv adapter. (redis#1109)
722e3409c Additional include directory given by pkg-config (redis#1118)
bd9ccb8c4 Use __attribute__ when building with clang on windows
5392adc26 set default SSL certificate directory
560e66486 Minor refactor
d756f68a5 Add libhv example to our standard Makefile
a66916719 Add adapters/libhv
855b48a81 Fix pkgconfig for hiredis_ssl
79ae5ffc6 Fix protocol error (redis#1106)
61b5b299f Use a windows specific keepalive function. (redis#1104)
fce8abc1c Introduce .close method for redisContextFuncs
cfb6ca881 Add REDIS_OPT_PREFER_UNSPEC (redis#1101)
cc7c35ce6 Update documentation to explain redisConnectWithOptions.
bc8d837b7 fix heap-buffer-overflow (redis#957)
ca4a0e850 uvadapter: reduce number of uv_poll_start calls
35d398c90 Fix cmake config path on Linux. CMake config files were installed to `/usr/local/share/hiredis`, which is not recognizable by `find_package()`. I'm not sure why it was set that way. Given the commit introducing it is for Windows, I keep that behavior consistent there, but fix the rest.
10c78c6e1 Add possibility to prefer IPv6, IPv4 or unspecified
1abe0c828 fuzzer: No alloc in redisFormatCommand() when fail
329eaf9ba Fix heap-buffer-overflow issue in redisvFormatCommad
eaae7321c Polling adapter requires sockcompat.h
0a5fa3dde Regression test for off-by-one parsing error
9e174e8f7 Add do while(0) protection for macros
4ad99c69a Rework asSleep to be a generic millisleep function.
75cb6c1ea Do store command timeout in the context for redisSetTimeout (redis#593)
c57cad658 CMake: remove dict.c form hiredis_sources
8491a65a9 Add Github Actions CI workflow for hiredis: Arm, Arm64, 386, windows. (redis#943)
77e4f09ea Merge pull request redis#964 from afcidk/fix-createDoubleObject
9219f7e7c Merge pull request redis#901 from devnexen/illumos_test_fix
810cc6104 Merge pull request redis#905 from sundb/master
df8b74d69 Merge pull request redis#1091 from redis/ssl-error-ub-fix
0ed6cdec3 Fix some undefined behaviour
507a6dcaa Merge pull request redis#1090 from Nordix/subscribe-oom-error
b044eaa6a Copy error to redisAsyncContext when finding subscribe cb
e0200b797 Merge pull request redis#1087 from redis/const-and-non-const-callback
6a3e96ad2 Maintain backward compatibiliy withour onConnect callback.
e7afd998f Merge pull request redis#1079 from SukkaW/drop-macos-10.15-runner
17c8fe079 Merge pull request redis#931 from kristjanvalur/pr2
b808c0c20 Merge pull request redis#1083 from chayim/ck-drafter
367a82bf0 Merge pull request redis#1085 from stanhu/ssl-improve-options-setting
71119a71d Make it possible to set SSL verify mode
dd7979ac1 Merge pull request redis#1084 from stanhu/sh-improve-ssl-docs
c71116178 Improve example for SSL initialization in README.md
5c9b6b571 Release drafter
a606ccf2a CI: use recommended `vmactions/freebsd-vm@v0`
0865c115b Merge pull request redis#1080 from Nordix/readme-corrections
f6cee7142 Fix README typos
06be7ff31 Merge pull request redis#1050 from smmir-cent/fix-cmake-version
7dd833d54 CI: bump macos runner version
f69fac769 Drop `const` on redisAsyncContext in redisConnectCallback Since the callback is now re-entrant, it can call apis such as redisAsyncDisconnect()
005d7edeb Support calling redisAsyncDisconnect from the onConnected callback, by deferring context deletion
6ed060920 Add async regression test for issue redis#931
eaa2a7ee7 Merge pull request redis#932 from kristjanvalur/pr3
2ccef30f3 Add regression test for issue redis#945
4b901d44a Initial async tests
31c91408e Polling adapter and example
8a15f4d65 Merge pull request redis#1057 from orgads/static-name
902dd047f Merge pull request redis#1054 from kristjanvalur/pr08
c78d0926b Merge pull request redis#1074 from michael-grunder/kristjanvalur-pr4
2b115d56c Whitespace
1343988ce Fix typos
47b57aa24 Add some documentation on connect/disconnect callbacks and command callbacks
a890d9ce2 Merge pull request redis#1073 from michael-grunder/kristjanvalur-pr1
f246ee433 Whitespace, style
94c1985bd Use correct type for getsockopt()
5e002bc21 Support failed async connects on windows.
5d68ad2f4 Merge pull request redis#1072 from michael-grunder/fix-redis7-unit-tests
f4b6ed289 Fix tests so they work for Redis 7.0
95a0c1283 Merge pull request redis#1058 from orgads/win64
eedb37a65 Fix warnings on Win64
47c3ecefc Merge pull request redis#1062 from yossigo/fix-push-notification-order
e23d91c97 Merge pull request redis#1061 from yossigo/update-redis-apt
34211ad54 Merge pull request redis#1063 from redis/fix-windows-tests
9957af7e3 Whitelist hiredis repo path in cygwin
b455b3381 Handle push notifications before or after reply.
aed9ce446 Use official repository for redis package.
d7683f35a Merge pull request redis#1047 from Nordix/unsubscribe-handling
7c44a9d7e Merge pull request redis#1045 from Nordix/sds-updates
dd4bf9783 Use the same name for static and shared libraries
ff57c18b9 Embed debug information in windows static lib, rather than create a .pdb file
8310ad4f5 fix cmake version
7123b87f6 Handle any pipelined unsubscribe in async
b6fb548fc Ignore pubsub replies without a channel/pattern
00b82683b Handle overflows as errors instead of asserting
64062a1d4 Catch size_t overflows in sds.c
066c6de79 Use size_t/long to avoid truncation
c6657ef65 Merge branch 'redis:master' into master
50cdcab49 Fix potential fault at createDoubleObject
fd033e983 Remove semicolon after do-while in _EL_CLEANUP
664c415e7 Illumos test fixes, error message difference fot bad hostname test.

git-subtree-dir: deps/hiredis
git-subtree-split: b6a052fe0959dae69e16b9d74449faeb1b70dbe1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants