redis cluster node crash

[wangchong666]

=== REDIS BUG REPORT START: Cut & paste starting from here ===
1:S 22 Feb 03:34:13.820 # Redis 4.0.11 crashed by signal: 11
1:S 22 Feb 03:34:13.821 # Crashed running the instruction at: 0x55e3d33d88c5
1:S 22 Feb 03:34:13.821 # Accessing address: (nil)
1:S 22 Feb 03:34:13.821 # Failed assertion: <no assertion failed> (<no file>:0)

------ STACK TRACE ------
EIP:
redis-server 10.2.210.178:7005 [cluster](sdsfree+0x5)[0x55e3d33d88c5]

Backtrace:
redis-server 10.2.210.178:7005 [cluster](logStackTrace+0x37)[0x55e3d3413c97]
redis-server 10.2.210.178:7005 [cluster](sigsegvHandler+0xb0)[0x55e3d34143c0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7f152696a0c0]
redis-server 10.2.210.178:7005 [cluster](sdsfree+0x5)[0x55e3d33d88c5]
redis-server 10.2.210.178:7005 [cluster](_dictClear+0x6a)[0x55e3d33d078a]
redis-server 10.2.210.178:7005 [cluster](dictRelease+0xf)[0x55e3d33d082f]
redis-server 10.2.210.178:7005 [cluster](decrRefCount+0x55)[0x55e3d33e6685]
redis-server 10.2.210.178:7005 [cluster](_dictClear+0x81)[0x55e3d33d07a1]
redis-server 10.2.210.178:7005 [cluster](dictEmpty+0x18)[0x55e3d33d1138]
redis-server 10.2.210.178:7005 [cluster](emptyDb+0xe5)[0x55e3d33eb8a5]
redis-server 10.2.210.178:7005 [cluster](readSyncBulkPayload+0x22b)[0x55e3d33ed7fb]
redis-server 10.2.210.178:7005 [cluster](aeProcessEvents+0x2a0)[0x55e3d33ce780]
redis-server 10.2.210.178:7005 [cluster](aeMain+0x2b)[0x55e3d33cea1b]
redis-server 10.2.210.178:7005 [cluster](main+0x4d3)[0x55e3d33cb673]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f15265da2e1]
redis-server 10.2.210.178:7005 [cluster](_start+0x2a)[0x55e3d33cb97a]

------ INFO OUTPUT ------
# Server
redis_version:4.0.11
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:74253224a862200c
redis_mode:cluster
os:Linux 3.10.0-693.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:6.3.0
process_id:1
run_id:d0b66f7b43c3669af0589cd8db4098312b1eec05
tcp_port:7005
uptime_in_seconds:2683494
uptime_in_days:31
hz:10
lru_clock:7302521
executable:/data/redis-server
config_file:/usr/local/etc/redis/redis.conf

# Clients
connected_clients:29
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:6495472320
used_memory_human:6.05G
used_memory_rss:21077594112
used_memory_rss_human:19.63G
used_memory_peak:20655589008
used_memory_peak_human:19.24G
used_memory_peak_perc:31.45%
used_memory_overhead:233696298
used_memory_startup:1444800
used_memory_dataset:6261776022
used_memory_dataset_perc:96.42%
total_system_memory:67253231616
total_system_memory_human:62.63G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
mem_fragmentation_ratio:3.24
mem_allocator:jemalloc-4.0.3
active_defrag_running:0
lazyfree_pending_objects:0

# Persistence
loading:0
rdb_changes_since_last_save:649458595
rdb_bgsave_in_progress:0
rdb_last_save_time:1548122952
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:229376
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

# Stats
total_connections_received:56366159
total_commands_processed:3139971137
instantaneous_ops_per_sec:0
total_net_input_bytes:375107235635
total_net_output_bytes:397869634893
instantaneous_input_kbps:187305.14
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:1
sync_partial_ok:0
sync_partial_err:1
expired_keys:4894890
expired_stale_perc:3.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:1672884722
keyspace_misses:194035384
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:250
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

# Replication
role:slave
master_host:10.2.210.176
master_port:7002
master_link_status:down
master_last_io_seconds_ago:-1
master_sync_in_progress:1
slave_repl_offset:1
master_sync_left_bytes:0
master_sync_last_io_seconds_ago:0
master_link_down_since_seconds:332
slave_priority:100
slave_read_only:1
connected_slaves:0
master_replid:1f6852d15132d47452a95662771bfb0fb8eba0c1
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:126872145629
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:126872145630
repl_backlog_histlen:0

# CPU
used_cpu_sys:39793.34
used_cpu_user:26318.38
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

# Commandstats
cmdstat_hgetall:calls=9613,usec=12241386,usec_per_call=1273.42
cmdstat_get:calls=6791205,usec=22683518,usec_per_call=3.34
cmdstat_replconf:calls=2671171,usec=3628521,usec_per_call=1.36
cmdstat_ping:calls=668226,usec=997965,usec_per_call=1.49
cmdstat_hset:calls=303022590,usec=2911309954,usec_per_call=9.61
cmdstat_expire:calls=178427323,usec=255385035,usec_per_call=1.43
cmdstat_info:calls=2,usec=144,usec_per_call=72.00
cmdstat_cluster:calls=11576,usec=1104556,usec_per_call=95.42
cmdstat_flushall:calls=1,usec=4697687,usec_per_call=4697687.00
cmdstat_exists:calls=151,usec=169,usec_per_call=1.12
cmdstat_hdel:calls=779978266,usec=2558235800,usec_per_call=3.28
cmdstat_set:calls=5487,usec=43295,usec_per_call=7.89
cmdstat_rpop:calls=647896,usec=521197,usec_per_call=0.80
cmdstat_command:calls=7552795,usec=2345798922,usec_per_call=310.59
cmdstat_psync:calls=1,usec=521,usec_per_call=521.00
cmdstat_hget:calls=1860178703,usec=34303461854,usec_per_call=18.44
cmdstat_pexpire:calls=151,usec=391,usec_per_call=2.59
cmdstat_client:calls=5980,usec=729668,usec_per_call=122.02

# Cluster
cluster_enabled:1

# Keyspace
db0:keys=1929499,expires=2246640,avg_ttl=702538631

------ CLIENT LIST OUTPUT ------
id=56365907 addr=10.2.210.126:57866 fd=267 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365908 addr=10.2.210.126:57867 fd=268 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365909 addr=10.2.210.126:57868 fd=269 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365910 addr=10.2.210.126:57874 fd=270 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365911 addr=10.2.210.126:57876 fd=271 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365912 addr=10.2.210.127:49596 fd=272 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365913 addr=10.2.210.127:49598 fd=273 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365914 addr=10.2.210.127:49600 fd=274 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365886 addr=10.2.210.128:50088 fd=246 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365887 addr=10.2.210.128:50090 fd=247 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365888 addr=10.2.210.125:39812 fd=248 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365889 addr=10.2.210.125:39810 fd=249 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365890 addr=10.2.210.125:39814 fd=250 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365891 addr=10.2.210.128:50092 fd=251 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365892 addr=10.2.210.128:50094 fd=252 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365893 addr=10.2.210.125:39816 fd=253 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365894 addr=10.2.210.125:39818 fd=254 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365895 addr=10.2.210.128:50096 fd=255 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365896 addr=10.2.210.124:39328 fd=256 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365897 addr=10.2.210.123:39802 fd=257 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365898 addr=10.2.210.123:39804 fd=258 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365899 addr=10.2.210.123:39806 fd=259 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365900 addr=10.2.210.124:39330 fd=260 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365901 addr=10.2.210.124:39331 fd=261 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365902 addr=10.2.210.124:39334 fd=262 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365903 addr=10.2.210.123:39808 fd=263 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365904 addr=10.2.210.124:39336 fd=264 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365905 addr=10.2.210.123:39810 fd=265 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command
id=56365906 addr=10.2.210.127:49594 fd=266 name= age=509 idle=509 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=command

------ REGISTERS ------
1:S 22 Feb 03:34:13.848 # 
RAX:000055e3d36fa880 RBX:00007f1525c4dcc0
RCX:0000000000000024 RDX:000055e3d33d16e0
RDI:3535312d706d6574 RSI:3535312d706d6574
RBP:00007f106bacf260 RSP:00007fffeca5f208
R8 :0000000000000097 R9 :00007f152600eea0
R10:00007f15262000c0 R11:000055e3d34ca8c0
R12:00007f106bacf270 R13:0000006264722e31
R14:0000000000000002 R15:0000000000000000
RIP:000055e3d33d88c5 EFL:0000000000010206
CSGSFS:0000000000000033
1:S 22 Feb 03:34:13.848 # (00007fffeca5f217) -> 000055e3d33ed020
1:S 22 Feb 03:34:13.848 # (00007fffeca5f216) -> 00007f1526018360
1:S 22 Feb 03:34:13.848 # (00007fffeca5f215) -> 0000000000000000
1:S 22 Feb 03:34:13.848 # (00007fffeca5f214) -> 000055e3d33d07a1
1:S 22 Feb 03:34:13.848 # (00007fffeca5f213) -> 00007f103577f858
1:S 22 Feb 03:34:13.848 # (00007fffeca5f212) -> 000055e3d33e6685
1:S 22 Feb 03:34:13.848 # (00007fffeca5f211) -> 00007f1070450ba0
1:S 22 Feb 03:34:13.848 # (00007fffeca5f210) -> 000055e3d33d082f
1:S 22 Feb 03:34:13.848 # (00007fffeca5f20f) -> 000055e3d33ed020
1:S 22 Feb 03:34:13.848 # (00007fffeca5f20e) -> 0000000000599163
1:S 22 Feb 03:34:13.848 # (00007fffeca5f20d) -> 0000000000000000
1:S 22 Feb 03:34:13.848 # (00007fffeca5f20c) -> 00007f1526018370
1:S 22 Feb 03:34:13.848 # (00007fffeca5f20b) -> 00007f1526018360
1:S 22 Feb 03:34:13.848 # (00007fffeca5f20a) -> 00007f106bacf260
1:S 22 Feb 03:34:13.848 # (00007fffeca5f209) -> 0000000000000004
1:S 22 Feb 03:34:13.848 # (00007fffeca5f208) -> 000055e3d33d078a

------ FAST MEMORY TEST ------
1:S 22 Feb 03:34:13.858 # Bio thread for job type #0 terminated
1:S 22 Feb 03:34:13.858 # Bio thread for job type #1 terminated
1:S 22 Feb 03:34:13.858 # Bio thread for job type #2 terminated
*** Preparing to test memory region 55e3d36ff000 (98304 bytes)
*** Preparing to test memory region 55e3d502d000 (135168 bytes)
*** Preparing to test memory region 7f1033a00000 (21214789632 bytes)
*** Preparing to test memory region 7f15243fe000 (8388608 bytes)
*** Preparing to test memory region 7f1524bff000 (8388608 bytes)
*** Preparing to test memory region 7f1525400000 (16777216 bytes)
*** Preparing to test memory region 7f1526955000 (16384 bytes)
*** Preparing to test memory region 7f1526b72000 (16384 bytes)
*** Preparing to test memory region 7f152729a000 (8192 bytes)
*** Preparing to test memory region 7f152729e000 (12288 bytes)
*** Preparing to test memory region 7f15272a3000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: sdsfree (base: 0x55e3d33d88c0)
Module: redis-server 10.2.210.178:7005 [cluster] (base 0x55e3d33a0000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x55e3d33d88c0 -D -b binary -m i386:x86-64 /tmp/dump.bin
------
1:S 22 Feb 03:35:25.399 # dump of function (hexdump of 133 bytes):
4885ff743b0fb647ff31d283e0073c047e0e4801d7e9062c0000660f1f440000488d1579a20d000fb6c04863148248f7da4801d7e9e72b00000f1f8000000000f3c30f1f4000662e0f1f840000000000534889fbe8171dffff0fb653ff83e20780fa04771f488d0d489f0d000fb6d2486314914801caffe20f1f840000000000488943ef5b

===

[trevor211]

Used memory was 6.05G, while used memory rss was 19.63G, which was abnormal.
Maybe there was an oom of your os.