redis crashed #855

Closed
stulentsev opened this Issue Dec 31, 2012 · 12 comments

Comments

Projects
None yet
4 participants

=== REDIS BUG REPORT START: Cut & paste starting from here ===
[10326] 30 Dec 18:53:09.781 # ------------------------------------------------
[10326] 30 Dec 18:53:09.781 # !!! Software Failure. Press left mouse button to continue
[10326] 30 Dec 18:53:09.781 # Guru Meditation: "Unknown object type" #rdb.c:990
[10326] 30 Dec 18:53:09.781 # (forcing SIGSEGV in order to print the stack trace)
[10326] 30 Dec 18:53:09.781 # ------------------------------------------------
[10326] 30 Dec 18:53:09.781 # Redis 2.6.2 crashed by signal: 11
[10326] 30 Dec 18:53:09.781 # Failed assertion: (:0)
[10326] 30 Dec 18:53:09.781 # --- STACK TRACE
/usr/sbin/redis-server(logStackTrace+0x75)[0x43aac5]
/usr/sbin/redis-server(_redisPanic+0x7f)[0x43a8cf]
/lib64/libpthread.so.0(+0xf500)[0x7f40ceb15500]
/usr/sbin/redis-server(_redisPanic+0x7f)[0x43a8cf]
/usr/sbin/redis-server(rdbLoadObject+0x7fd)[0x42847d]
/usr/sbin/redis-server(rdbLoad+0x1bb)[0x42868b]
/usr/sbin/redis-server(readSyncBulkPayload+0x1b9)[0x425509]
/usr/sbin/redis-server(aeProcessEvents+0x13c)[0x41218c]
/usr/sbin/redis-server(aeMain+0x2b)[0x41244b]
/usr/sbin/redis-server(main+0x247)[0x4186c7]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f40ce791cdd]
/usr/sbin/redis-server[0x4117b9]
[10326] 30 Dec 18:53:09.786 # --- INFO OUTPUT
[10326] 30 Dec 18:53:09.786 # # Server
redis_version:2.6.2
redis_git_sha1:00000000
redis_git_dirty:0
redis_mode:standalone
os:Linux 2.6.32-279.14.1.el6.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.6
process_id:10326
run_id:4ed4a7dea2cc78b1de056a9892a266e797e823d3
tcp_port:6379
uptime_in_seconds:811572
uptime_in_days:9
lru_clock:1471630

Clients

connected_clients:8
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

Memory

used_memory:783696
used_memory_human:765.33K
used_memory_rss:27607040
used_memory_peak:2115475416
used_memory_peak_human:1.97G
used_memory_lua:31744
mem_fragmentation_ratio:35.23
mem_allocator:jemalloc-3.0.0

Persistence

loading:1
rdb_changes_since_last_save:1910434431
rdb_bgsave_in_progress:0
rdb_last_save_time:1356082009
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
loading_start_time:1356893589
loading_total_bytes:1404835975
loading_loaded_bytes:9
loading_loaded_perc:0.00
loading_eta_seconds:-1248743080

Stats

total_connections_received:2693355
total_commands_processed:2245945160
instantaneous_ops_per_sec:24
rejected_connections:0
expired_keys:0
evicted_keys:0
keyspace_hits:3878228
keyspace_misses:7095207
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0

Replication

role:slave
master_host:redis-resque.12
master_port:6379
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:0
master_link_down_since_seconds:112
slave_priority:100
slave_read_only:1
connected_slaves:0

CPU

used_cpu_sys:29532.75
used_cpu_user:23060.87
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

Commandstats

cmdstat_get:calls=1185569,usec=7369649,usec_per_call=6.22
cmdstat_setex:calls=316566318,usec=1673175763,usec_per_call=5.29
cmdstat_del:calls=314256536,usec=993102986,usec_per_call=3.16
cmdstat_rpush:calls=160600382,usec=1081798617,usec_per_call=6.74
cmdstat_lpush:calls=16630,usec=100250,usec_per_call=6.03
cmdstat_lpop:calls=158343463,usec=845146883,usec_per_call=5.34
cmdstat_blpop:calls=3,usec=29,usec_per_call=9.67
cmdstat_llen:calls=8897151,usec=37502688,usec_per_call=4.22
cmdstat_lrange:calls=1,usec=13,usec_per_call=13.00
cmdstat_sadd:calls=157128064,usec=533112436,usec_per_call=3.39
cmdstat_srem:calls=157128079,usec=658703320,usec_per_call=4.19
cmdstat_smembers:calls=852967,usec=28450390,usec_per_call=33.35
cmdstat_zrangebyscore:calls=37499,usec=582494,usec_per_call=15.53
cmdstat_zcard:calls=246,usec=1046,usec_per_call=4.25
cmdstat_incrby:calls=323496806,usec=1145687848,usec_per_call=3.54
cmdstat_keys:calls=15968,usec=6322240,usec_per_call=395.93
cmdstat_ping:calls=80094,usec=223636,usec_per_call=2.79
cmdstat_type:calls=2,usec=13,usec_per_call=6.50
cmdstat_multi:calls=322898153,usec=874468250,usec_per_call=2.71
cmdstat_exec:calls=322898153,usec=7074102817,usec_per_call=21.91
cmdstat_info:calls=1543047,usec=406297289,usec_per_call=263.31
cmdstat_slaveof:calls=2,usec=323,usec_per_call=161.50
cmdstat_config:calls=27,usec=992,usec_per_call=36.74

Keyspace

db0:keys=21,expires=21
hash_init_value: 1356221043

[10326] 30 Dec 18:53:09.786 # --- CLIENT LIST OUTPUT
[10326] 30 Dec 18:53:09.787 # addr=172.22.12.101:45589 fd=6 age=118377 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=49 oll=0 omem=0 events=rw cmd=smembers
addr=127.0.0.1:57129 fd=5 age=811559 idle=10 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info
addr=172.22.11.101:57780 fd=10 age=82652 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=49 oll=0 omem=0 events=rw cmd=smembers
addr=172.22.13.101:40817 fd=15 age=49064 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=49 oll=0 omem=0 events=rw cmd=smembers
addr=172.22.13.101:55698 fd=11 age=23 idle=5 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
addr=172.22.12.121:45638 fd=9 age=23 idle=23 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=keys
addr=172.22.13.101:55710 fd=13 age=17 idle=7 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
addr=172.22.13.121:42244 fd=16 age=0 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=NULL

[10326] 30 Dec 18:53:09.787 # --- REGISTERS
[10326] 30 Dec 18:53:09.787 #
RAX:0000000000000000 RBX:0000000000489b8c
RCX:0000000000000000 RDX:00000000000152f0
RDI:0000000000000000 RSI:0000000000000000
RBP:000000000048a989 RSP:00007fff04505e50
R8 :00007f40ceb01e80 R9 :00007f40ceb01ed0
R10:00007f40ceb01ed0 R11:0000000000000206
R12:00000000000003de R13:0000000000000017
R14:00007f4092a4dbd0 R15:ffffffffffffffff
RIP:000000000043a8cf EFL:0000000000010202
CSGSFS:0000000000000033
[10326] 30 Dec 18:53:09.787 # (00007fff04505ec8) -> 00007f4092a4dbd0
[10326] 30 Dec 18:53:09.787 # (00007fff04505ec0) -> 0000000000000017
[10326] 30 Dec 18:53:09.787 # (00007fff04505eb8) -> 0000013bed291113
[10326] 30 Dec 18:53:09.787 # (00007fff04505eb0) -> 00007f40cdd0b300
[10326] 30 Dec 18:53:09.787 # (00007fff04505ea8) -> 00007fff04506300
[10326] 30 Dec 18:53:09.787 # (00007fff04505ea0) -> 00007fff04506300
[10326] 30 Dec 18:53:09.787 # (00007fff04505e98) -> 0000000000427b12
[10326] 30 Dec 18:53:09.788 # (00007fff04505e90) -> 00007fff04506300
[10326] 30 Dec 18:53:09.788 # (00007fff04505e88) -> 000000000000002b
[10326] 30 Dec 18:53:09.788 # (00007fff04505e80) -> 000000000000002b
[10326] 30 Dec 18:53:09.788 # (00007fff04505e78) -> 0000000000422ee8
[10326] 30 Dec 18:53:09.788 # (00007fff04505e70) -> 00007f406346fc88
[10326] 30 Dec 18:53:09.788 # (00007fff04505e68) -> 000000000042847d
[10326] 30 Dec 18:53:09.788 # (00007fff04505e60) -> 00007fff04506300
[10326] 30 Dec 18:53:09.788 # (00007fff04505e58) -> 0000000000000020
[10326] 30 Dec 18:53:09.788 # (00007fff04505e50) -> 00007fff04506300
[10326] 30 Dec 18:53:09.788 #
=== REDIS BUG REPORT END. Make sure to include from START to END. ===

Owner

antirez commented Dec 31, 2012

Strange issue... the slave, loading the DB produced by the master, crashes.

Is it possible that the data got corrupted during the movement? Could you generate an RDB on the master, transfer it to the slave, and load it manually to see if it works?

Thanks

After deleting local dump and re-slaving it worked. So, I guess, it's the corruption. What should I do now with this ticket? I can't reliably reproduce the problem.

Owner

antirez commented Jan 3, 2013

Well, in your side, check the hardware :-)
In my side, I think the problem is the following:

  • We have a CRC64 at the end of RDB files.
  • The CRC is checked only after the whole DB was loaded.

So if the problem is corrupted data that don't crash the server while loading, the CRC check is performed and the problem is detected. But... if the data corruption is of a kind that prevents the RDB to be loaded at all without a crash, there is currently no way to tell before processing it.

So probably even at the cost of a greater loading time for RDB files, we should add an option, active by default, to check the CRC before loading the RDB at all.

Maybe we could have three possibilities:

    1. Check the RDB ASAP, when loading an RDB produced by a master (when we are a slave).
    1. Perform "1", and also check the RDB even when loading an RDB from disk at startup.
    1. Don't check the CRC before loading (maximum performances, but the server may crash on corrupted files).

With the default of "1".

Well moving this issue into 2.8 milestone to start!

Owner

antirez commented Jan 3, 2013

Update, your last crash report on Twitter makes more sense:

[1066] 02 Jan 21:35:25.827 * Connecting to MASTER...
[1066] 02 Jan 21:35:25.829 # Can't create readable event for SYNC
[1066] 02 Jan 21:35:26.069 # Error allocating resoures for the client
[1066] 02 Jan 21:35:26.070 # Error allocating resoures for the client
[1066] 02 Jan 21:35:26.082 # Error allocating resoures for the client
[1066] 02 Jan 21:35:26.084 # Error allocating resoures for the client
[1066] 02 Jan 21:35:26.122 # Error allocating resoures for the client
[1066] 02 Jan 21:35:26.125 # Error allocating resoures for the client
[1066] 02 Jan 21:35:26.948 * Connecting to MASTER...
[1066] 02 Jan 21:35:26.949 # Can't create readable event for SYNC
[1066] 02 Jan 21:35:27.327 * MASTER <-> SLAVE sync: Loading DB in memory
[1066] 02 Jan 21:35:54.352 # I/O error trying to sync with MASTER: connection lost
[1066] 02 Jan 21:35:54.352 # 

=== REDIS BUG REPORT START: Cut & paste starting from here ===
[1066] 02 Jan 21:35:54.352 # === ASSERTION FAILED ===
[1066] 02 Jan 21:35:54.352 # ==> replication.c:304 'server.repl_state == REDIS_REPL_TRANSFER' is not true
[1066] 02 Jan 21:35:54.352 # (forcing SIGSEGV to print the bug report.)
[1066] 02 Jan 21:35:54.352 #     Redis 2.6.2 crashed by signal: 11
[1066] 02 Jan 21:35:54.352 #     Failed assertion: server.repl_state == REDIS_REPL_TRANSFER (replication.c:304)
[1066] 02 Jan 21:35:54.352 # --- STACK TRACE
/usr/sbin/redis-server(logStackTrace+0x75)[0x43aac5]
/usr/sbin/redis-server(_redisAssert+0x6f)[0x43a95f]
/lib64/libpthread.so.0(+0xf500)[0x7f3096d0e500]
/usr/sbin/redis-server(_redisAssert+0x6f)[0x43a95f]
/usr/sbin/redis-server(replicationAbortSyncTransfer+0x75)[0x424f25]
/usr/sbin/redis-server(readSyncBulkPayload+0xc5)[0x425415]
/usr/sbin/redis-server(aeProcessEvents+0x13c)[0x41218c]
/usr/sbin/redis-server(rdbLoad+0x23b)[0x42870b]
/usr/sbin/redis-server(readSyncBulkPayload+0x1b9)[0x425509]
/usr/sbin/redis-server(aeProcessEvents+0x13c)[0x41218c]
/usr/sbin/redis-server(aeMain+0x2b)[0x41244b]
/usr/sbin/redis-server(main+0x247)[0x4186c7]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f309698acdd]
/usr/sbin/redis-server[0x4117b9]
[1066] 02 Jan 21:35:54.359 # --- INFO OUTPUT
[1066] 02 Jan 21:35:54.359 # # Server
redis_version:2.6.2
redis_git_sha1:00000000
redis_git_dirty:0
redis_mode:standalone
os:Linux 2.6.32-279.14.1.el6.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.6
process_id:1066
run_id:699d22bdc623abe3a76c5587299065ea73642eb6
tcp_port:6379
uptime_in_seconds:654309
uptime_in_days:7
lru_clock:1498524

# Clients
connected_clients:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:927520
used_memory_human:905.78K
used_memory_rss:15810560
used_memory_peak:4176321384
used_memory_peak_human:3.89G
used_memory_lua:31744
mem_fragmentation_ratio:17.05
mem_allocator:jemalloc-3.0.0

# Persistence
loading:1
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1357162492
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:18
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
loading_start_time:1357162554
loading_total_bytes:229885520
loading_loaded_bytes:9
loading_loaded_perc:0.00
loading_eta_seconds:-689656533

# Stats
total_connections_received:2284850
total_commands_processed:359634214
instantaneous_ops_per_sec:0
rejected_connections:0
expired_keys:0
evicted_keys:0
keyspace_hits:23783126
keyspace_misses:266622
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:81562

# Replication
role:slave
master_host:redis-uniques.12
master_port:6379
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:0
master_link_down_since_seconds:7275
slave_priority:100
slave_read_only:1
connected_slaves:0

# CPU
used_cpu_sys:24492.59
used_cpu_user:43849.39
used_cpu_sys_children:6756.90
used_cpu_user_children:64623.35

# Commandstats
cmdstat_del:calls=610613,usec=2891032970,usec_per_call=4734.64
cmdstat_lpush:calls=18844,usec=225417,usec_per_call=11.96
cmdstat_lpop:calls=2,usec=41,usec_per_call=20.50
cmdstat_blpop:calls=167129,usec=1404693,usec_per_call=8.40
cmdstat_sadd:calls=333650281,usec=4434982221,usec_per_call=13.29
cmdstat_scard:calls=23890573,usec=205777068,usec_per_call=8.61
cmdstat_sunionstore:calls=150444,usec=21036062185,usec_per_call=139826.53
cmdstat_smembers:calls=74006,usec=285984862,usec_per_call=3864.35
cmdstat_keys:calls=77781,usec=42643658,usec_per_call=548.25
cmdstat_ping:calls=30118,usec=114640,usec_per_call=3.81
cmdstat_type:calls=85169,usec=1288854,usec_per_call=15.13
cmdstat_sync:calls=114,usec=5949558,usec_per_call=52189.11
cmdstat_replconf:calls=115,usec=743,usec_per_call=6.46
cmdstat_info:calls=876634,usec=342414375,usec_per_call=390.60
cmdstat_slaveof:calls=146,usec=101415,usec_per_call=694.62
cmdstat_config:calls=255,usec=6261,usec_per_call=24.55
cmdstat_object:calls=1990,usec=12868,usec_per_call=6.47

# Keyspace
hash_init_value: 1356534167

[1066] 02 Jan 21:35:54.359 # --- CLIENT LIST OUTPUT
[1066] 02 Jan 21:35:54.359 # addr=127.0.0.1:48011 fd=6 age=654292 idle=7 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=info

[1066] 02 Jan 21:35:54.359 # --- REGISTERS
[1066] 02 Jan 21:35:54.359 # 
RAX:0000000000000000 RBX:0000000000000130
RCX:0000000000000000 RDX:0000000000015290
RDI:0000000000000000 RSI:0000000000000000
RBP:0000000000489e48 RSP:00007fff7401c7d0
R8 :00007f3096cfae80 R9 :00007f3096cfaed0
R10:00007f3096cfaed0 R11:0000000000000206
R12:000000000048a218 R13:0000000000000000
R14:0000000000000001 R15:0000000000000007
RIP:000000000043a95f EFL:0000000000010206
CSGSFS:e2c1000000000033
[1066] 02 Jan 21:35:54.359 # (00007fff7401c848) -> c16bccc1749ec150
[1066] 02 Jan 21:35:54.359 # (00007fff7401c840) -> e9c15c94c1635fc1
[1066] 02 Jan 21:35:54.359 # (00007fff7401c838) -> 72eac12430c14fd7
[1066] 02 Jan 21:35:54.359 # (00007fff7401c830) -> c1477cc1543ec11b
[1066] 02 Jan 21:35:54.359 # (00007fff7401c828) -> 43c164f9c17a4fc1
[1066] 02 Jan 21:35:54.359 # (00007fff7401c820) -> 461bc17d1cc1226b
[1066] 02 Jan 21:35:54.359 # (00007fff7401c818) -> c100008258c24770
[1066] 02 Jan 21:35:54.359 # (00007fff7401c810) -> c1641cc1177fc100
[1066] 02 Jan 21:35:54.359 # (00007fff7401c808) -> 00850dc269bcc100
[1066] 02 Jan 21:35:54.359 # (00007fff7401c800) -> 008557c26bf7c166
[1066] 02 Jan 21:35:54.360 # (00007fff7401c7f8) -> 0000000000425415
[1066] 02 Jan 21:35:54.360 # (00007fff7401c7f0) -> 00007f30960c90e0
[1066] 02 Jan 21:35:54.360 # (00007fff7401c7e8) -> 0000000000424f25
[1066] 02 Jan 21:35:54.360 # (00007fff7401c7e0) -> 0000000000000000
[1066] 02 Jan 21:35:54.360 # (00007fff7401c7d8) -> 00007f30960c90e0
[1066] 02 Jan 21:35:54.360 # (00007fff7401c7d0) -> 0000000000000000
[1066] 02 Jan 21:35:54.360 # 
=== REDIS BUG REPORT END. Make sure to include from START to END. ===

Investigating.

Thanks, waiting for (good) news :)

Owner

antirez commented Jan 3, 2013

@stulentsev the problem appears to originate in the following error lines:

[1066] 02 Jan 21:35:25.829 # Can't create readable event for SYNC
[1066] 02 Jan 21:35:26.069 # Error allocating resoures for the client
[1066] 02 Jan 21:35:26.070 # Error allocating resoures for the client
[1066] 02 Jan 21:35:26.082 # Error allocating resoures for the client

Probably the Redis implementation should resist to this problems, but here the root cause is to understand why this instance can't register file events on the file descriptors.

Do you have more logs for this instance? Also I can provide a modified version of Redis that logs the problem with the event registering in a more clear way so that we can understand the cause.

@antirez Yes, we can run modified version. Also we should have that log file available, I'll look as soon as I get to the office.

Owner

antirez commented Jan 3, 2013

Cool, just committed the new stuff with better error reporting. Please just use the latest "2.6" branch commit and when the system fails adding the readable event for the file descriptor we should have more hints about what's happening.

All the rest, including the first crash, is a result of this failure apparently.

Contributor

jokea commented Jan 9, 2013

The second crash log looks very strange to me.

[1066] 02 Jan 21:35:26.948 * Connecting to MASTER...
[1066] 02 Jan 21:35:26.949 # Can't create readable event for SYNC
[1066] 02 Jan 21:35:27.327 * MASTER <-> SLAVE sync: Loading DB in memory
[1066] 02 Jan 21:35:54.352 # I/O error trying to sync with MASTER: connection lost

Since it failed to register readable event for master's fd, how did it go to loading DB in next
second? Where did the DB file came from? How did the IO error gets detected when redis
is loading RDB?

There might be some races there, like @antirez said, the full log would be helpful.
@stulentsev could you please provide the config file for this server?

@jokea, @antirez The config is quite standard, I believe. Here it is

daemonize yes
pidfile /var/run/redis/redis.pid
port 6379
timeout 1800
loglevel notice
logfile /var/log/redis/redis.log
databases 16
save 900 1
save 300 10
save 60 10000
stop-writes-on-bgsave-error yes
rdbcompression yes
rdbchecksum yes
dbfilename dump.rdb
dir /var/lib/redis/
slave-serve-stale-data yes
slave-read-only yes
slave-priority 100
appendonly no
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
lua-time-limit 5000
slowlog-log-slower-than 10000
slowlog-max-len 128
hash-max-ziplist-entries 512
hash-max-ziplist-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
zset-max-ziplist-entries 128
zset-max-ziplist-value 64
activerehashing yes
client-output-buffer-limit normal 0 0 0
client-output-buffer-limit slave 256mb 64mb 60
client-output-buffer-limit pubsub 32mb 8mb 60
Contributor

jokea commented Jan 10, 2013

@stulentsev Do you still have the full log file of the instance?

Codefor commented Mar 7, 2013

I have almostly the same problem.
The slave SYNC from the master,connection lost ,and SYNC again,connection lost again,and more and more...

Here is part of the log from SLAVE:

[29490] 07 Mar 13:34:46.648 * Connecting to MASTER...
[29490] 07 Mar 13:34:46.649 * MASTER <-> SLAVE sync started
[29490] 07 Mar 13:34:46.649 * Non blocking connect for SYNC fired the event.
[29490] 07 Mar 13:34:46.649 * Master replied to PING, replication can continue...
[29490] 07 Mar 13:36:16.834 * MASTER <-> SLAVE sync: receiving 3706317904 bytes from master
[29490] 07 Mar 13:36:43.748 # I/O error trying to sync with MASTER: connection lost
[29490] 07 Mar 13:36:45.174 * Connecting to MASTER...
[29490] 07 Mar 13:36:45.175 * MASTER <-> SLAVE sync started
[29490] 07 Mar 13:36:45.175 * Non blocking connect for SYNC fired the event.
[29490] 07 Mar 13:36:45.175 * Master replied to PING, replication can continue...

antirez closed this Jul 10, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment