Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redis slaves disconnect from master continuesly #6997

Open
hamidreza-hosseini opened this issue Mar 16, 2020 · 3 comments
Open

Redis slaves disconnect from master continuesly #6997

hamidreza-hosseini opened this issue Mar 16, 2020 · 3 comments

Comments

@hamidreza-hosseini
Copy link

hamidreza-hosseini commented Mar 16, 2020

Hi
I have 3 redis nodes (One of them is master and remain are slave and replica of master)
this is Master and slave config file: (Redis.conf)
Master:

port 6379
pidfile /var/run/redis/redis.pid
loglevel notice
logfile /var/log/redis/redis.log
save ""
appendonly no
daemonize no
supervised systemd
repl-diskless-sync yes```
#####
Slaves:
```bind 192.168.200.18
port 6379
pidfile /var/run/redis/redis.pid
loglevel notice
logfile /var/log/redis/redis.log
save ""
appendonly no
daemonize no
supervised systemd
replicaof 192.168.200.19 6379

I want my database and replica be exist just in memory (because of high I/O)
and i dont want to use AOF or another durability solutions...
But my slaves disconnect continuesly from master (even when i add repl-diskless-sync yes to master)
And this is my log files error:

Partial resynchronization not possible (no cached master)
5221:S 16 Mar 2020 21:55:08.418 * Full resync from master: d4bb766fb3c050efee7fd27b0b1a57c484d696de:784
5221:S 16 Mar 2020 21:55:13.419 # Opening the temp file needed for MASTER <-> REPLICA synchronization: Read-only file system
5221:S 16 Mar 2020 21:55:13.531 * Connecting to MASTER 192.168.200.19:6379
5221:S 16 Mar 2020 21:55:13.531 * MASTER <-> REPLICA sync started
5221:S 16 Mar 2020 21:55:13.532 * Non blocking connect for SYNC fired the event.
5221:S 16 Mar 2020 21:55:13.532 * Master replied to PING, replication can continue...
5221:S 16 Mar 2020 21:55:13.533 * Partial resynchronization not possible (no cached master)
5221:S 16 Mar 2020 21:55:19.450 * Full resync from master: d4bb766fb3c050efee7fd27b0b1a57c484d696de:798

Slaves are saying :
Partial resynchronization not possible

1.How can i arrive my purpose(having data just in ram on master and slaves side) and solving this problem?

2.And I dont want to do Full resynce every time, i want to have partial syn (it means just differences goes to slaves ) because i have 20 GB data in each server's ram...

@trevor211
Copy link
Collaborator

Can you give the redis version, master config and master log?

@hamidreza-hosseini
Copy link
Author

I change the configuration because you said in documetaion that WARNING: DISKLESS REPLICATION IS EXPERIMENTAL CURRENTLY.
I change my configuration to this:

bind 192.168.200.19
port 6379
pidfile "/var/run/redis/redis.pid"
loglevel notice
logfile "/var/log/redis/redis.log"

appendonly no
daemonize no
supervised systemd

#repl-diskless-sync yes
#repl-timeout 6
#requirepass 123456
save ""
stop-writes-on-bgsave-error no
rdbcompression no
rdbchecksum no
dbfilename "dump.rdb"
dir "/var/lib/redis"

But now when slve servers restart, master will dump all ram's data into disk even though i set save "" and that db file will transfer to slaves and slaves will load the file to ram
And Master logs says i cant do partial sync:

31180:signal-handler (1585206496) Received SIGTERM scheduling shutdown...
31180:signal-handler (1585206496) Received SIGTERM scheduling shutdown...
31180:M 26 Mar 2020 11:38:16.184 # User requested shutdown...
31180:M 26 Mar 2020 11:38:16.184 * Removing the pid file.
31180:M 26 Mar 2020 11:38:16.184 # Redis is now ready to exit, bye bye...
2581:C 26 Mar 2020 11:38:16.376 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2581:C 26 Mar 2020 11:38:16.376 # Redis version=5.0.8, bits=64, commit=00000000, modified=0, pid=2581, just started
2581:C 26 Mar 2020 11:38:16.376 # Configuration loaded
2581:C 26 Mar 2020 11:38:16.376 * supervised by systemd, will signal readiness
2581:M 26 Mar 2020 11:38:16.380 * Running mode=standalone, port=6379.
2581:M 26 Mar 2020 11:38:16.380 # Server initialized
2581:M 26 Mar 2020 11:38:18.473 * DB loaded from disk: 2.093 seconds
2581:M 26 Mar 2020 11:38:18.473 * Ready to accept connections
2581:M 26 Mar 2020 11:38:19.051 * Replica 192.168.200.18:6379 asks for synchronization
2581:M 26 Mar 2020 11:38:19.052 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'b146f34d9c277f27b832ae737bb15abf8658981d', my replication IDs are '0cc4535b2e6c4bcb56cbf3058cfbba832bb574ef' and '0000000000000000000000000000000000000000')
2581:M 26 Mar 2020 11:38:19.052 * Starting BGSAVE for SYNC with target: disk
2581:M 26 Mar 2020 11:38:19.081 * Background saving started by pid 2591
2581:M 26 Mar 2020 11:38:19.347 * Replica 192.168.200.17:6379 asks for synchronization
2581:M 26 Mar 2020 11:38:19.347 * Partial resynchronization not accepted: Replication ID mismatch (Replica asked for 'b146f34d9c277f27b832ae737bb15abf8658981d', my replication IDs are '54e863a0ecf7d1846713cea8685ca0565a4cd2cd' and '0000000000000000000000000000000000000000')
2581:M 26 Mar 2020 11:38:19.348 * Waiting for end of BGSAVE for SYNC
2591:C 26 Mar 2020 11:38:31.745 * DB saved on disk
2591:C 26 Mar 2020 11:38:31.759 * RDB: 0 MB of memory used by copy-on-write
2581:M 26 Mar 2020 11:38:31.799 * Background saving terminated with success
2581:M 26 Mar 2020 11:38:37.066 * Synchronization with replica 192.168.200.17:6379 succeeded
2581:M 26 Mar 2020 11:39:27.587 * Synchronization with replica 192.168.200.18:6379 succeeded

slave log:

23844:S 25 Mar 2020 21:32:13.746 # CONFIG REWRITE executed with success.
23844:S 25 Mar 2020 21:32:13.875 * Connecting to MASTER 192.168.200.19:6379
23844:S 25 Mar 2020 21:32:13.875 * MASTER <-> REPLICA sync started
23844:S 25 Mar 2020 21:32:13.884 * Non blocking connect for SYNC fired the event.
23844:S 25 Mar 2020 21:32:13.885 * Master replied to PING, replication can continue...
23844:S 25 Mar 2020 21:32:13.886 * Trying a partial resynchronization (request 3edf32944f0df9d7e835f94d5712e6b3c00bb5ab:1).
23844:S 25 Mar 2020 21:32:13.886 * Full resync from master: b146f34d9c277f27b832ae737bb15abf8658981d:0
23844:S 25 Mar 2020 21:32:13.886 * Discarding previously cached master state.
23844:S 25 Mar 2020 21:32:19.098 * MASTER <-> REPLICA sync: receiving 971987525 bytes from master
23844:S 25 Mar 2020 21:34:16.513 * MASTER <-> REPLICA sync: Flushing old data
23844:S 25 Mar 2020 21:34:16.533 * MASTER <-> REPLICA sync: Loading DB in memory
23844:S 25 Mar 2020 21:34:17.988 * MASTER <-> REPLICA sync: Finished with success
23844:S 26 Mar 2020 11:38:16.344 # Connection with master lost.
23844:S 26 Mar 2020 11:38:16.345 * Caching the disconnected master state.
23844:S 26 Mar 2020 11:38:17.063 * Connecting to MASTER 192.168.200.19:6379
23844:S 26 Mar 2020 11:38:17.064 * MASTER <-> REPLICA sync started
23844:S 26 Mar 2020 11:38:17.064 * Non blocking connect for SYNC fired the event.
23844:S 26 Mar 2020 11:38:17.064 # Error reply to PING from master: '-LOADING Redis is loading the dataset in memory'
23844:S 26 Mar 2020 11:38:18.067 * Connecting to MASTER 192.168.200.19:6379
23844:S 26 Mar 2020 11:38:18.068 * MASTER <-> REPLICA sync started
23844:S 26 Mar 2020 11:38:18.089 * Non blocking connect for SYNC fired the event.
23844:S 26 Mar 2020 11:38:18.093 # Error reply to PING from master: '-LOADING Redis is loading the dataset in memory'
23844:S 26 Mar 2020 11:38:19.070 * Connecting to MASTER 192.168.200.19:6379
23844:S 26 Mar 2020 11:38:19.070 * MASTER <-> REPLICA sync started
23844:S 26 Mar 2020 11:38:19.071 * Non blocking connect for SYNC fired the event.
23844:S 26 Mar 2020 11:38:19.072 * Master replied to PING, replication can continue...
23844:S 26 Mar 2020 11:38:19.072 * Trying a partial resynchronization (request b146f34d9c277f27b832ae737bb15abf8658981d:10903612).
23844:S 26 Mar 2020 11:38:19.103 * Full resync from master: 54e863a0ecf7d1846713cea8685ca0565a4cd2cd:0
23844:S 26 Mar 2020 11:38:19.103 * Discarding previously cached master state.
23844:S 26 Mar 2020 11:38:31.822 * MASTER <-> REPLICA sync: receiving 971987525 bytes from master
23844:S 26 Mar 2020 11:39:28.731 * MASTER <-> REPLICA sync: Flushing old data
23844:S 26 Mar 2020 11:39:28.769 * MASTER <-> REPLICA sync: Loading DB in memory
23844:S 26 Mar 2020 11:39:30.303 * MASTER <-> REPLICA sync: Finished with success

I want my database and replica be exist just in memory (because of high I/O)
and i dont want to use AOF or another durability solutions, and data transfer just from master's ram to slave's ram in the form of partially sync not from disk to disk.
How can i do this?
(I'm using sentinel and redis version 5.0.8)

@trevor211
Copy link
Collaborator

From the master log, we can see that replid changed and cleared.
Did the master not having any replicas for a long time?
By the way, what is the config repl-backlog-ttl of master?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants