Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[QUESTION] Rename will cause sentinel switching to fail #10328

Open
tanglu86 opened this issue Feb 22, 2022 · 8 comments
Open

[QUESTION] Rename will cause sentinel switching to fail #10328

tanglu86 opened this issue Feb 22, 2022 · 8 comments

Comments

@tanglu86
Copy link

When the primary node fails, sentinel switching will fail, and the log outputs "next failover delay: I will not start a failover before"
How do I fix this problem?

My redis conf :
rename-command FLUSHALL ADMINFLUSHALL
rename-command FLUSHDB ADMINFLUSHDB#
rename-command CONFIG ADMINCONFIG
rename-command KEYS ADMINKEYS

@Jeremy-Run
Copy link

Are you sure that the failover was caused by changing the name? The error log you sent is because of multiple failovers within 6 minutes. If you want to know why the failover failed, you have to look at the previous log. And the log you sent is not complete. The complete log will write the time when the next failover can be performed at the end.

@tanglu86
Copy link
Author

tanglu86 commented Feb 24, 2022

Are you sure that the failover was caused by changing the name? The error log you sent is because of multiple failovers within 6 minutes. If you want to know why the failover failed, you have to look at the previous log. And the log you sent is not complete. The complete log will write the time when the next failover can be performed at the end.

I am sure that RENAME caused the failover failure。
I set up a new cluster,If the RENAME option modifies the CONFIG command, the failover fails
1
2

@tanglu86
Copy link
Author

tanglu86 commented Feb 24, 2022

I've tested many times,When RENAME CONFIG is enabled, sentinel failover will exception
1
2

@Jeremy-Run
Copy link

Jeremy-Run commented Feb 24, 2022

I'm guessing you also have an error of "Unknown command 'flushall' reading the append only file" in your log file, because the AOF doesn't support "flushall". You can try shutting down AOF and restarting the cluster. This is turned off by changing "appendonly" to "no" in the configuration file. You better know what it means to turn off AOF before you turn it off.

@tanglu86
Copy link
Author

I'm guessing you also have an error of "Unknown command 'flushall' reading the append only file" in your log file, because the AOF doesn't support "flushall". You can try shutting down AOF and restarting the cluster. This is turned off by changing "appendonly" to "no" in the configuration file. You better know what it means to turn off AOF before you turn it off.

I don't enabled AOF and RDB ,I have updated the question,RENAME CONFIG ,not RENAME FLUSHALL or RENAME FLUSHDB

@hwware
Copy link
Collaborator

hwware commented Feb 28, 2022

@tanglu86 Hi in order to help solve your issue, could you please provide the following detail to us, then we could try to reproduce it and check why your sentinel can not failover, Thanks

  1. Which version Redis your are using?
  2. How is your cluster node acritude? such as how many master node, how many replica node, and how many sentinel?
  3. Could you please detail config file to us?
  4. As what you say, you do not enable AOF and RDB, do you enable shard or just simple cluster?

Thanks, and looking forward to your feedback.

@tanglu86
Copy link
Author

tanglu86 commented Mar 1, 2022

@tanglu86 Hi in order to help solve your issue, could you please provide the following detail to us, then we could try to reproduce it and check why your sentinel can not failover, Thanks

  1. Which version Redis your are using?
  2. How is your cluster node acritude? such as how many master node, how many replica node, and how many sentinel?
  3. Could you please detail config file to us?
  4. As what you say, you do not enable AOF and RDB, do you enable shard or just simple cluster?

Thanks, and looking forward to your feedback.

  1. The redis version I use is 5.0.9
    2、1 master node, 2 replica nodes, 3 sentinel nodes
    3、My profile:
    redis.conf.txt
    sentinel.conf.txt

@hwware
Copy link
Collaborator

hwware commented Mar 2, 2022

Hi @tanglu86 ,
I tried replicating this by using some combinations of configs and I was able to get the error message which you received. I will take a deeper dive into this to find the exact cause of this and let you know what I find out.

I used the following configs:
master.conf

bind 127.0.0.1
protected-mode yes
port 6379
rename-command FLUSHALL ADMINFLUSHALL
rename-command FLUSHDB ADMINFLUSHDB
rename-command CONFIG ADMINCONFIG
rename-command KEYS ADMINKEYS

replica1.conf

bind 127.0.0.1
protected-mode yes
port 6380
replicaof 127.0.0.1 6379

rename-command FLUSHALL ADMINFLUSHALL
rename-command FLUSHDB ADMINFLUSHDB
rename-command CONFIG ADMINCONFIG
rename-command KEYS ADMINKEYS

replica2.conf

bind 127.0.0.1
protected-mode yes
port 6381
replicaof 127.0.0.1 6379

rename-command FLUSHALL ADMINFLUSHALL
rename-command FLUSHDB ADMINFLUSHDB
rename-command CONFIG ADMINCONFIG
rename-command KEYS ADMINKEYS

sentinel1.conf

port 5000
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 6000
sentinel failover-timeout mymaster 18000
replica-announce-ip "127.0.0.1"

sentinel2.conf

port 5001
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 6000
sentinel failover-timeout mymaster 18000
replica-announce-ip "127.0.0.1"

sentinel3.conf

port 5002
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 6000
sentinel failover-timeout mymaster 18000
replica-announce-ip "127.0.0.1"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants