Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redis bgsave getting triggered during failovers #4595

Open
sreeraagbuyt opened this issue Jan 11, 2018 · 3 comments
Open

Redis bgsave getting triggered during failovers #4595

sreeraagbuyt opened this issue Jan 11, 2018 · 3 comments

Comments

@sreeraagbuyt
Copy link

sreeraagbuyt commented Jan 11, 2018

We have a 64 node (32 master and 32 slave) Redis cluster setup spawning across 4 hosts ( Each machine 16 core, 48GB), we've spawned so many nodes to better utilize CPUs.
We've totally disabled rdbsaves as we're experiencing CPU and Memory limitations on our Redis cluster.

>>redis-cli -c -p 8001 config get save
1) "save"
2) ""

However, when things start to melt down, i.e.
When this happens from multiple nodes:

12055:M 10 Jan 19:45:10.911 # Cluster state changed: fail
12055:M 10 Jan 19:45:19.527 * FAIL message received from 611a1147e459cff54f96dc625dca25eb8047b651 about 389d2923b63143dca8a907ad7b1ecc6c294399e5
....

I see that the failover happens after master and slave perform rdb save's on their end. The issue for us with this is that, the nodes got flaky because they were not getting CPU/Memory, and triggering an rdb save at that time would only worsen things further, making other nodes on the same machine fail too.
What's the way out in such cases

@WiFeng
Copy link

WiFeng commented Jan 11, 2018

On replication mode (master/slave), although you havn't set save option in the config , the master node will still execute RDB operation.

It is best to set aside at least a half of the physical memory for the RDB operation.

@chowmean
Copy link

There is no setting to stop it? @WiFeng

@WiFeng
Copy link

WiFeng commented Jun 19, 2020

As far as I know, no setting could stop it directly. But we can avoid Full-sync by setting related options, and then rdb-save operation should not happened.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants