redis 4 creash

[fuzengjie]

=== REDIS BUG REPORT START: Cut & paste starting from here ===
23961:S 24 Jun 04:30:56.348 # Redis 4.0.12 crashed by signal: 11
23961:S 24 Jun 04:30:56.348 # Crashed running the instruction at: 0x497ef8
23961:S 24 Jun 04:30:56.348 # Accessing address: 0x7f8ecaa00000
23961:S 24 Jun 04:30:56.348 # Failed assertion: (:0)

------ STACK TRACE ------
EIP:
/usr/local/redis-server4/bin/redis-server *:6630(siphash+0x58)[0x497ef8]

Backtrace:
/usr/local/redis-server4/bin/redis-server *:6630(logStackTrace+0x3c)[0x46800c]
/usr/local/redis-server4/bin/redis-server *:6630(sigsegvHandler+0xa3)[0x469243]
/lib64/libpthread.so.0[0x3c93a0f710]
/usr/local/redis-server4/bin/redis-server *:6630(siphash+0x58)[0x497ef8]
/usr/local/redis-server4/bin/redis-server *:6630(dictRehash+0x8d)[0x427cad]
/usr/local/redis-server4/bin/redis-server *:6630(dictAddRaw+0x1aa)[0x4287aa]
/usr/local/redis-server4/bin/redis-server *:6630(dictAdd+0x20)[0x428900]
/usr/local/redis-server4/bin/redis-server *:6630(dbAdd+0x6a)[0x44281a]
/usr/local/redis-server4/bin/redis-server *:6630(setKey+0x76)[0x442b26]
/usr/local/redis-server4/bin/redis-server *:6630(setGenericCommand+0xa4)[0x44cfa4]
/usr/local/redis-server4/bin/redis-server *:6630(setCommand+0xfa)[0x44d32a]
/usr/local/redis-server4/bin/redis-server *:6630(call+0x87)[0x429d57]
/usr/local/redis-server4/bin/redis-server *:6630(processCommand+0x395)[0x42e885]
/usr/local/redis-server4/bin/redis-server *:6630(processInputBuffer+0x10d)[0x43a92d]
/usr/local/redis-server4/bin/redis-server *:6630(readQueryFromClient+0x101)[0x43ab21]
/usr/local/redis-server4/bin/redis-server *:6630(aeProcessEvents+0x1d3)[0x425453]
/usr/local/redis-server4/bin/redis-server *:6630(aeMain+0x2b)[0x42577b]
/usr/local/redis-server4/bin/redis-server *:6630(main+0x500)[0x42dfc0]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3c9361ed5d]
/usr/local/redis-server4/bin/redis-server *:6630[0x422ae9]

------ INFO OUTPUT ------

Server^M

redis_version:4.0.12^M
redis_git_sha1:afc4b36c^M
redis_git_dirty:1^M
redis_build_id:7b7e61a779d11458^M
redis_mode:standalone^M
os:Linux 4.9.0 x86_64^M
arch_bits:64^M
multiplexing_api:epoll^M
atomicvar_api:sync-builtin^M
gcc_version:4.4.7^M
process_id:23961^M
run_id:625043d6946b959b01122c3a99c8f17b7587f220^M
tcp_port:6630^M
uptime_in_seconds:41051645^M
uptime_in_days:475^M
hz:90^M
lru_clock:15885942^M
executable:/usr/local/redis-server4/bin/redis-server^M
config_file:/usr/local/redis-server4/conf/6630.conf^M
^M

Clients^M

connected_clients:12^M
client_longest_output_list:0^M
client_biggest_input_buf:0^M
blocked_clients:0^M
^M

Memory^M

used_memory:19520278216^M
used_memory_human:18.18G^M
used_memory_rss:23782514688^M
used_memory_rss_human:22.15G^M
used_memory_peak:24646208544^M
used_memory_peak_human:22.95G^M
used_memory_peak_perc:79.20%^M
used_memory_overhead:7237428580^M
used_memory_startup:2536968^M
used_memory_dataset:12282849636^M
used_memory_dataset_perc:62.93%^M
total_system_memory:270444048384^M
total_system_memory_human:251.87G^M
used_cpu_user:324183.78^M
used_cpu_sys_children:3805082.75^M
used_cpu_user_children:14409088.00^M
^M

Commandstats^M

cmdstat_pexpire:calls=20557,usec=174616,usec_per_call=8.49^M
cmdstat_bgsave:calls=1,usec=447139,usec_per_call=447139.00^M
cmdstat_ping:calls=553472327,usec=686823957,usec_per_call=1.24^M
cmdstat_setex:calls=5639667,usec=204356896,usec_per_call=36.24^M
cmdstat_command:calls=3,usec=3439,usec_per_call=1146.33^M
cmdstat_hset:calls=136829,usec=4152824,usec_per_call=30.35^M
cmdstat_expireat:calls=346652324,usec=2280844160,usec_per_call=6.58^M
cmdstat_replconf:calls=27508697,usec=46412714,usec_per_call=1.69^M
cmdstat_publish:calls=108519810,usec=1439888026,usec_per_call=13.27^M
cmdstat_multi:calls=2,usec=2,usec_per_call=1.00^M
cmdstat_expire:calls=136829,usec=1087227,usec_per_call=7.95^M
cmdstat_set:calls=422330833,usec=11719933753,usec_per_call=27.75^M
cmdstat_client:calls=301880,usec=1308107,usec_per_call=4.33^M
cmdstat_exec:calls=8,usec=59626,usec_per_call=7453.25^M
cmdstat_hgetall:calls=1,usec=4,usec_per_call=4.00^M
cmdstat_exists:calls=61664527,usec=524134175,usec_per_call=8.50^M
cmdstat_del:calls=3575309,usec=57068735,usec_per_call=15.96^M
cmdstat_get:calls=7895171283,usec=57602426975,usec_per_call=7.30^M
cmdstat_select:calls=182,usec=276,usec_per_call=1.52^M
cmdstat_incr:calls=2,usec=38,usec_per_call=19.00^M
cmdstat_unlink:calls=90663485,usec=5339905816,usec_per_call=58.90^M
cmdstat_slowlog:calls=443886,usec=49377006,usec_per_call=111.24^M
cmdstat_psync:calls=3,usec=599985,usec_per_call=199995.00^M
cmdstat_ttl:calls=5,usec=103,usec_per_call=20.60^M
cmdstat_zremrangebyscore:calls=214283538,usec=2625948730,usec_per_call=12.25^M
cmdstat_mset:calls=20557,usec=633536,usec_per_call=30.82^M
cmdstat_info:calls=54392696,usec=6169897161,usec_per_call=113.43^M
cmdstat_zadd:calls=228914122,usec=5055772865,usec_per_call=22.09^M
cmdstat_config:calls=31963334,usec=1007950668,usec_per_call=31.53^M
cmdstat_subscribe:calls=176288,usec=1596745,usec_per_call=9.06^M
^M

Cluster^M

cluster_enabled:0^M
^M

Keyspace^M

db0:keys=81026377,expires=76484657,avg_ttl=7354308897^M

------ CLIENT LIST OUTPUT ------
id=9091179 addr=10.27.17.46:57624 fd=177 name= age=3123 idle=0 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=9091202 addr=10.27.4.181:6630 fd=12 name= age=2626 idle=0 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=set
id=9091180 addr=10.27.16.46:63792 fd=178 name= age=3123 idle=0 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=9091181 addr=10.27.16.45:65086 fd=179 name= age=3123 idle=0 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=9091184 addr=10.27.9.231:42281 fd=182 name= age=3123 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=config
id=9091002 addr=10.27.16.45:13062 fd=8 name=sentinel-716b7d31-cmd age=3393 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=publish
id=9091003 addr=10.27.17.46:5374 fd=9 name=sentinel-3f3ead71-cmd age=3393 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9091006 addr=10.27.17.44:54030 fd=17 name=sentinel-6c575b9d-cmd age=3393 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9091173 addr=10.27.17.45:59781 fd=171 name= age=3123 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9091174 addr=10.27.16.46:63036 fd=172 name= age=3123 idle=1 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=ping
id=9091177 addr=10.27.17.44:18053 fd=175 name= age=3123 idle=0 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe
id=9091178 addr=10.27.17.45:60419 fd=176 name= age=3123 idle=0 flags=N db=0 sub=1 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=subscribe

------ CURRENT CLIENT INFO ------
id=9091202 addr=10.27.4.181:6630 fd=12 name= age=2626 idle=0 flags=M db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=set
argv[0]: 'SET'
argv[1]: 'ops_sla_monitor_G1Mb6ifW'
argv[2]: 'ops_sla_monitor'
23961:S 24 Jun 04:30:56.350 # key 'ops_sla_monitor_G1Mb6ifW' found in DB containing the following object:
23961:S 24 Jun 04:30:56.350 # Object type: 0
23961:S 24 Jun 04:30:56.350 # Object encoding: 8
23961:S 24 Jun 04:30:56.350 # Object refcount: 1
23961:S 24 Jun 04:30:56.350 # Object raw string len: 15
23961:S 24 Jun 04:30:56.350 # Object raw string content: "ops_sla_monitor"

------ REGISTERS ------
23961:S 24 Jun 04:30:56.350 #
RAX:1c5094fbe11a0eb5 RBX:00007f8ecd170400
RCX:0000000000000000 RDX:c913a70678eabc75
RDI:00007f8eca9ffff9 RSI:fffffffff2d15688
RBP:00007f8a832b3880 RSP:00007ffc2d0ae748
R14:0000000000000000 R15:0000000000000000
RIP:0000000000497ef8 EFL:0000000000010293
CSGSFS:002b000000000033
23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae757) -> 00007f8959a63f48
23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae756) -> 00007f8d413b2281
23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae755) -> 00000000aad3ba51
23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae754) -> 00007f8ecd170400
23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae753) -> 00007f895ba00000
23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae752) -> 00007f894663e340
23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae751) -> e00434f666fd1e8f
23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae750) -> 0000000037e8f478
23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae74f) -> 00007f8959a63f48
23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae74e) -> 00000000004287aa
23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae74d) -> 0000000000000000
23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae74c) -> 0000000000000000
23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae74b) -> 00007f8d413b2281
23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae74a) -> 00007f8ecd170400
23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae749) -> 00007f8ecd170400
23961:S 24 Jun 04:30:56.350 # (00007ffc2d0ae748) -> 0000000000427cad

------ FAST MEMORY TEST ------
23961:S 24 Jun 04:30:56.350 # Bio thread for job type #0 terminated
23961:S 24 Jun 04:30:56.350 # Bio thread for job type #1 terminated
23961:S 24 Jun 04:30:56.350 # Bio thread for job type #2 terminated
*** Preparing to test memory region 74e000 (98304 bytes)
*** Preparing to test memory region 2197000 (135168 bytes)
*** Preparing to test memory region 3c93021000 (4096 bytes)
*** Preparing to test memory region 3c9398f000 (20480 bytes)
*** Preparing to test memory region 3c93c19000 (16384 bytes)
*** Preparing to test memory region 7f88a4000000 (26417823744 bytes)
*** Preparing to test memory region 7f8ecabfe000 (8388608 bytes)
*** Preparing to test memory region 7f8ecb3ff000 (8388608 bytes)
*** Preparing to test memory region 7f8ecbc00000 (10485760 bytes)
*** Preparing to test memory region 7f8ecc800000 (8388608 bytes)
*** Preparing to test memory region 7f8ecd000000 (2097152 bytes)
*** Preparing to test memory region 7f8ed3200000 (2097152 bytes)
*** Preparing to test memory region 7f8ed3561000 (16384 bytes)
*** Preparing to test memory region 7f8ed3579000 (4096 bytes)
*** Preparing to test memory region 7f8ed357a000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: siphash (base: 0x497ea0)
Module: /usr/local/redis-server4/bin/redis-server *:6630 (base 0x400000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x497ea0 -D -b binary -m i386:x86-64 /tmp/dump.bin

23961:S 24 Jun 04:32:23.733 # dump of function (hexdump of 216 bytes):
4c8b42084989f2488b0a4983e2f848ba736574796264657448b86d6f646e61726f644e8d14174989f349b96172656e6567796c4c31c24c31c049b875657370656d6f7349c1e3384931c94931c84c39d7744b660f1f440000488b0f4e8d040048c1c8334883c7084c31c049c1c8204831ca4e8d0c0a48c1ca304c31ca4901c148c1c82f4e8d040248c1ca2b4c31c849c1c9204c31c24931c84939fa75bb83e607ff24f5787a5100660f1f840000000000410fb64a0648c1e1304909cb410fb64a0548c1e1284909cb410fb64a0448c1e1204909cb410fb64a

[madolson]

Hey @fuzengjie, can you provide any more context on what your application was doing at the time or any ability to reproduce it? we've seen a couple of previous issues with similar crashes which we think our memory corruption related. (e.g. #5835 / #7358)

Do you know if the memory you are running with has ECC?