Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redis 4.0.1, Slave blocked from reaching master node #4298

Open
ghost opened this issue Sep 10, 2017 · 0 comments
Open

Redis 4.0.1, Slave blocked from reaching master node #4298

ghost opened this issue Sep 10, 2017 · 0 comments

Comments

@ghost
Copy link

ghost commented Sep 10, 2017

Hi,
I have a problem with Redis 4.0.1 with classical cluster configuration.
First and most important thing to tell you is that I run cluster without any problem on existing configuration, when I use Redis 3.2.10.
However under Redis 4.0.1 my Redis cluster is highly unstable, and often many slave nodes permanently can't join cluster.
On one of this slaves, through log files I managed to catch bug, that I will try to post here in best possible way.
Again before further text I need to say one more time, that I am using same hardware and OS configuration without any problem with Redis 3.2.10
So here we go.
Cluster Architecture:
90 nodes on 3 machines where we have 30 master nodes and 60 slave nodes. Each master node has 2 slave nodes.
Master and 2 slave nodes are always distributed across 3 machines
Hardware configuration:
I use AWS. Here are 3 R4 instances 4xlarge. Each machine has 16 cores and 122 GB RAM, with 200GB of General Purpose SSD, and static IP addresses
OS configuration:
Amazon Linux AMI 2017.03.1 HVM,
OS parameters
sysctl -w net.core.somaxconn=65535 inside /etc/rc.local
echo 'vm.overcommit_memory = 1' >> /etc/sysctl.conf
echo never > /sys/kernel/mm/transparent_hugepage/enabled inside /etc/rc.local
fs.file-max = 100000 inside /etc/sysctl.conf
<my_user> soft nofile 4096 inside /etc/security/limits.conf
<my_user> hard nofile 10240 inside /etc/security/limits.conf
session required pam_limits.so inside /etc/pam.d/

Bug dump is attached
RedisBugDump.txt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

0 participants