Redis 5.0.5 crashed by signal: 11

[shanthivardhan]

=== REDIS BUG REPORT START: Cut & paste starting from here ===
30910:M 12 Aug 2019 00:55:10.575 # Redis 5.0.5 crashed by signal: 11
30910:M 12 Aug 2019 00:55:10.575 # Crashed running the instruction at: 0x4e1525
30910:M 12 Aug 2019 00:55:10.575 # Accessing address: 0x11dd0
30910:M 12 Aug 2019 00:55:10.575 # Failed assertion: (:0)

------ STACK TRACE ------
EIP:
/usr/bin/redis-server *:6379(je_malloc_usable_size+0x85)[0x4e1525]

Backtrace:
/usr/bin/redis-server *:6379(logStackTrace+0x45)[0x473e75]
/usr/bin/redis-server *:6379(sigsegvHandler+0xb9)[0x474639]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7ffa201fa390]
/usr/bin/redis-server *:6379(je_malloc_usable_size+0x85)[0x4e1525]
/usr/bin/redis-server *:6379(zfree+0xe)[0x43750e]
/usr/bin/redis-server *:6379(rdbSave+0x26e)[0x4515de]
/usr/bin/redis-server *:6379(saveCommand+0x5f)[0x45416f]
/usr/bin/redis-server *:6379(call+0xaf)[0x43003f]
/usr/bin/redis-server *:6379(processCommand+0x377)[0x430707]
/usr/bin/redis-server *:6379(processInputBuffer+0x175)[0x440b45]
/usr/bin/redis-server *:6379(aeProcessEvents+0x36d)[0x429ccd]
/usr/bin/redis-server *:6379(aeMain+0x2b)[0x429efb]
/usr/bin/redis-server *:6379(main+0x4ae)[0x426aae]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7ffa1fe3f830]
/usr/bin/redis-server *:6379(_start+0x29)[0x426d19]

------ INFO OUTPUT ------

Server

redis_version:5.0.5
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:96192a109b5fb62e
redis_mode:standalone
os:Linux 4.4.0-1088-aws x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:5.4.0
process_id:30910
run_id:29e96ab638b845e66b891da0daebf300efdd5745
tcp_port:6379
uptime_in_seconds:37887
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:5290222
executable:/usr/bin/redis-server
config_file:/etc/redis/redis.conf

Clients

connected_clients:24
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:0

Memory

used_memory:1462264
used_memory_human:1.39M
used_memory_rss:4812800
used_memory_rss_human:4.59M
used_memory_peak:1562904
used_memory_peak_human:1.49M
used_memory_peak_perc:93.56%
used_memory_overhead:1329704
used_memory_startup:791368
used_memory_dataset:132560
used_memory_dataset_perc:19.76%
allocator_allocated:1454416
allocator_active:1822720
allocator_resident:4751360
total_system_memory:16222552064
total_system_memory_human:15.11G
used_memory_lua:32768
used_memory_lua_human:32.00K
used_memory_scripts:496
used_memory_scripts_human:496B
number_of_cached_scripts:2
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.25
allocator_frag_bytes:368304
allocator_rss_ratio:2.61
allocator_rss_bytes:2928640
rss_overhead_ratio:1.01
rss_overhead_bytes:61440
mem_fragmentation_ratio:3.39
mem_fragmentation_bytes:3392512
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:537216
mem_aof_buffer:0
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0

Persistence

loading:0
rdb_changes_since_last_save:122
rdb_bgsave_in_progress:0
rdb_last_save_time:1565571048
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:368640
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

Stats

total_connections_received:29
total_commands_processed:245040
instantaneous_ops_per_sec:13
total_net_input_bytes:13067558
total_net_output_bytes:1307439
instantaneous_input_kbps:0.76
instantaneous_output_kbps:0.05
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:1
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:15156
keyspace_misses:15160
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:227
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

Replication

role:master
connected_slaves:0
master_replid:969dbee204ba92baf8bc8a65b0bd1287455aafdc
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

CPU

used_cpu_sys:8.012000
used_cpu_user:8.640000
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000

Commandstats

cmdstat_info:calls=8,usec=373,usec_per_call=46.62
cmdstat_ping:calls=2,usec=0,usec_per_call=0.00
cmdstat_config:calls=4,usec=15,usec_per_call=3.75
cmdstat_smembers:calls=1,usec=1,usec_per_call=1.00
cmdstat_lpop:calls=166669,usec=138867,usec_per_call=0.83
cmdstat_hgetall:calls=7,usec=4,usec_per_call=0.57
cmdstat_expire:calls=7577,usec=11036,usec_per_call=1.46
cmdstat_get:calls=15154,usec=10471,usec_per_call=0.69
cmdstat_spop:calls=4,usec=7,usec_per_call=1.75
cmdstat_zrangebyscore:calls=7577,usec=22197,usec_per_call=2.93
cmdstat_time:calls=8845,usec=12057,usec_per_call=1.36
cmdstat_set:calls=13,usec=26,usec_per_call=2.00
cmdstat_scard:calls=7577,usec=3585,usec_per_call=0.47
cmdstat_hset:calls=8845,usec=24348,usec_per_call=2.75
cmdstat_sadd:calls=10,usec=27,usec_per_call=2.70
cmdstat_evalsha:calls=15156,usec=221512,usec_per_call=14.62
cmdstat_setnx:calls=7577,usec=14854,usec_per_call=1.96
cmdstat_auth:calls=7,usec=8,usec_per_call=1.14
cmdstat_hdel:calls=5,usec=12,usec_per_call=2.40
cmdstat_script:calls=2,usec=76,usec_per_call=38.00

Cluster

cluster_enabled:0

Keyspace

db0:keys=11,expires=1,avg_ttl=175783

[itamarhaber]

Hello @shanthivardhan

Thanks for reporting this crash.

a) It looks like the crash log is missing its final part - was it not present in the log?
b) Is this crash reproducible?

[xeron]

Not sure if related but 5.0.6 crashes with Segmentation fault: 11 message on macOS Catalina if built with MALLOC=jemalloc. Removing MALLOC=jemalloc fixes the issue.

[mitolH]

@itamarhaber HELP! the same crash info

=== REDIS BUG REPORT START: Cut & paste starting from here ===
74557:C 14 Jan 2020 07:51:40.046 # Redis 5.0.4 crashed by signal: 11
74557:C 14 Jan 2020 07:51:40.047 # Crashed running the instruction at: 0x44a54b
74557:C 14 Jan 2020 07:51:40.047 # Accessing address: 0xffffffffffffffff
74557:C 14 Jan 2020 07:51:40.047 # Failed assertion: (:0)

------ STACK TRACE ------
EIP:
redis-rdb-bgsave 127.0.0.1:6379(rdbSaveObject+0x1fb)[0x44a54b]

Backtrace:
redis-rdb-bgsave 127.0.0.1:6379(logStackTrace+0x41)[0x46f931]
redis-rdb-bgsave 127.0.0.1:6379(sigsegvHandler+0xc9)[0x4700b9]
/lib64/libpthread.so.0(+0xf5f0)[0x7fa0ebe185f0]
redis-rdb-bgsave 127.0.0.1:6379(rdbSaveObject+0x1fb)[0x44a54b]
redis-rdb-bgsave 127.0.0.1:6379(rdbSaveKeyValuePair+0x111)[0x44acf1]
redis-rdb-bgsave 127.0.0.1:6379(rdbSaveRio+0x232)[0x44b212]
redis-rdb-bgsave 127.0.0.1:6379(rdbSave+0xb4)[0x44b714]
redis-rdb-bgsave 127.0.0.1:6379(rdbSaveBackground+0x15e)[0x44ba4e]
redis-rdb-bgsave 127.0.0.1:6379(serverCron+0x80c)[0x42ae7c]
redis-rdb-bgsave 127.0.0.1:6379(aeProcessEvents+0x360)[0x423d00]
redis-rdb-bgsave 127.0.0.1:6379(aeMain+0x2b)[0x423f5b]
redis-rdb-bgsave 127.0.0.1:6379(main+0x4e0)[0x420bd0]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7fa0eba5d505]
redis-rdb-bgsave 127.0.0.1:6379[0x420e23]

------ INFO OUTPUT ------

Server

redis_version:5.0.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:2eba388922a6d94c
redis_mode:standalone
os:Linux 3.10.0-514.el7.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:atomic-builtin
gcc_version:4.8.5
process_id:74557
run_id:316ebb623b82a7a69f8a4489b93075ae104536cb
tcp_port:6379
uptime_in_seconds:783029
uptime_in_days:9
hz:10
configured_hz:10
lru_clock:1929996
executable:/usr/bin/redis-server
config_file:/etc/redis.conf

Clients

connected_clients:0
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:0

Memory

used_memory:142424968
used_memory_human:135.83M
used_memory_rss:149819392
used_memory_rss_human:142.88M
used_memory_peak:144066640
used_memory_peak_human:137.39M
used_memory_peak_perc:98.86%
used_memory_overhead:790064
used_memory_startup:787728
used_memory_dataset:141634904
used_memory_dataset_perc:100.00%
allocator_allocated:142527208
allocator_active:145948672
allocator_resident:149781504
total_system_memory:67387404288
total_system_memory_human:62.76G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.02
allocator_frag_bytes:3421464
allocator_rss_ratio:1.03
allocator_rss_bytes:3832832
rss_overhead_ratio:1.00
rss_overhead_bytes:37888
mem_fragmentation_ratio:1.05
mem_fragmentation_bytes:7395544
mem_not_counted_for_evict:0
mem_replication_backlog:0
mem_clients_slaves:0
mem_clients_normal:0
mem_aof_buffer:0
mem_allocator:jemalloc-3.6.0
active_defrag_running:0
lazyfree_pending_objects:0

Persistence

loading:0
rdb_changes_since_last_save:5091
rdb_bgsave_in_progress:0
rdb_last_save_time:1578770170
rdb_last_bgsave_status:err
rdb_last_bgsave_time_sec:0
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:3334144
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:0

Stats

total_connections_received:289
total_commands_processed:119348554
instantaneous_ops_per_sec:0
total_net_input_bytes:14215984722
total_net_output_bytes:40321500052
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:43734
expired_stale_perc:0.00
expired_time_cap_reached_count:0
evicted_keys:0
keyspace_hits:107772349
keyspace_misses:4847992
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:4558
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0

Replication

role:master
connected_slaves:0
master_replid:a85ce0437bcd3124177405237b1236e02108549a
master_replid2:0000000000000000000000000000000000000000
master_repl_offset:0
second_repl_offset:-1
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

CPU

used_cpu_sys:0.000000
used_cpu_user:0.005697
used_cpu_sys_children:0.000000
used_cpu_user_children:0.000000

Commandstats

cmdstat_zrevrangebyscore:calls=1934,usec=22417,usec_per_call=11.59
cmdstat_zremrangebyscore:calls=12857,usec=56830,usec_per_call=4.42
cmdstat_hgetall:calls=3222197,usec=31101396,usec_per_call=9.65
cmdstat_zcount:calls=38764,usec=225483,usec_per_call=5.82
cmdstat_del:calls=55210,usec=217406,usec_per_call=3.94
cmdstat_sismember:calls=6,usec=28,usec_per_call=4.67
cmdstat_zscan:calls=145,usec=1903,usec_per_call=13.12
cmdstat_zrank:calls=1926,usec=11179,usec_per_call=5.80
cmdstat_zadd:calls=17872,usec=184309,usec_per_call=10.31
cmdstat_pexpireat:calls=44538,usec=103805,usec_per_call=2.33
cmdstat_pexpire:calls=6,usec=22,usec_per_call=3.67
cmdstat_mget:calls=1902,usec=9289,usec_per_call=4.88
cmdstat_smembers:calls=3909330,usec=16213668,usec_per_call=4.15
cmdstat_expire:calls=69147,usec=375048,usec_per_call=5.42
cmdstat_zrevrange:calls=6,usec=36,usec_per_call=6.00
cmdstat_hmget:calls=23978398,usec=92445320,usec_per_call=3.86
cmdstat_set:calls=88687,usec=721826,usec_per_call=8.14
cmdstat_rpush:calls=427,usec=36770,usec_per_call=86.11
cmdstat_exists:calls=4476,usec=9696,usec_per_call=2.17
cmdstat_zrangebyscore:calls=42990,usec=342713,usec_per_call=7.97
cmdstat_get:calls=13649,usec=29525,usec_per_call=2.16
cmdstat_zrem:calls=11615,usec=99401,usec_per_call=8.56
cmdstat_sadd:calls=1924,usec=10544,usec_per_call=5.48
cmdstat_hmset:calls=1205089,usec=7489794,usec_per_call=6.22
cmdstat_zscore:calls=8248,usec=77990,usec_per_call=9.46
cmdstat_psetex:calls=386,usec=2125,usec_per_call=5.51
cmdstat_hset:calls=5191153,usec=36940034,usec_per_call=7.12
cmdstat_zrange:calls=1971,usec=5216,usec_per_call=2.65
cmdstat_lrange:calls=11748,usec=60446,usec_per_call=5.15
cmdstat_auth:calls=244,usec=525,usec_per_call=2.15
cmdstat_zincrby:calls=28595,usec=356329,usec_per_call=12.46
cmdstat_hget:calls=81380749,usec=324720775,usec_per_call=3.99
cmdstat_srem:calls=2365,usec=13627,usec_per_call=5.76

Cluster

cluster_enabled:0

Keyspace

db0:keys=14,expires=10,avg_ttl=1379401000

------ CLIENT LIST OUTPUT ------

------ REGISTERS ------
74557:C 14 Jan 2020 07:51:40.049 #
RAX:00007fa0dac72380 RBX:00000000004db2f4
RCX:0000000000000000 RDX:00007fa0dac72380
RDI:00007fa0eb0d3680 RSI:0000000000000000
RBP:00007ffd9d33dcb0 RSP:00007ffd9d33d940
R8 :00007fa0eb043ce0 R9 :0000000000000001
R10:00007fa0e0969c15 R11:0000000000000031
R12:0000000000011b9d R13:00007fa0eb1bf070
R14:00000000004db308 R15:0000000000000000
RIP:000000000044a54b EFL:0000000000010202
CSGSFS:0000000000000033
74557:C 14 Jan 2020 07:51:40.049 # (00007ffd9d33d94f) -> 00007ffd9d33dc40
74557:C 14 Jan 2020 07:51:40.049 # (00007ffd9d33d94e) -> 00007fa0eba8827d
74557:C 14 Jan 2020 07:51:40.049 # (00007ffd9d33d94d) -> 0000000000000000
74557:C 14 Jan 2020 07:51:40.049 # (00007ffd9d33d94c) -> 00007ffd9d33dc40
74557:C 14 Jan 2020 07:51:40.049 # (00007ffd9d33d94b) -> 0000000000000009
74557:C 14 Jan 2020 07:51:40.049 # (00007ffd9d33d94a) -> 0000000000000001
74557:C 14 Jan 2020 07:51:40.049 # (00007ffd9d33d949) -> 00007fa0eb043c00
74557:C 14 Jan 2020 07:51:40.049 # (00007ffd9d33d948) -> ff39f2ae007f80b7
74557:C 14 Jan 2020 07:51:40.049 # (00007ffd9d33d947) -> 00007fa0ec97d8d3
74557:C 14 Jan 2020 07:51:40.049 # (00007ffd9d33d946) -> 00007ffd9d33dbc0
74557:C 14 Jan 2020 07:51:40.049 # (00007ffd9d33d945) -> 00007fa0eba481a8
74557:C 14 Jan 2020 07:51:40.049 # (00007ffd9d33d944) -> 3900000500000000
74557:C 14 Jan 2020 07:51:40.049 # (00007ffd9d33d943) -> 0000000000000000
74557:C 14 Jan 2020 07:51:40.049 # (00007ffd9d33d942) -> 00007fa000000005
74557:C 14 Jan 2020 07:51:40.049 # (00007ffd9d33d941) -> 00007fa0eb0f7190
74557:C 14 Jan 2020 07:51:40.049 # (00007ffd9d33d940) -> 0000000000000009

------ FAST MEMORY TEST ------
*** Preparing to test memory region 71c000 (90112 bytes)
*** Preparing to test memory region 7fa0da400000 (146800640 bytes)
*** Preparing to test memory region 7fa0e32d4000 (8388608 bytes)
*** Preparing to test memory region 7fa0e3ad5000 (8388608 bytes)
*** Preparing to test memory region 7fa0e42d6000 (8388608 bytes)
*** Preparing to test memory region 7fa0eb000000 (8388608 bytes)
*** Preparing to test memory region 7fa0ebe04000 (20480 bytes)
*** Preparing to test memory region 7fa0ec021000 (16384 bytes)
*** Preparing to test memory region 7fa0ec966000 (4096 bytes)
*** Preparing to test memory region 7fa0ecb59000 (20480 bytes)
*** Preparing to test memory region 7fa0ecb85000 (8192 bytes)
*** Preparing to test memory region 7fa0ecb87000 (4096 bytes)
*** Preparing to test memory region 7fa0ecb8a000 (4096 bytes)
.O.O.O.O.O.O.O.O.O.O.O.O.O
Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.

------ DUMPING CODE AROUND EIP ------
Symbol: rdbSaveObject (base: 0x44a350)
Module: redis-rdb-bgsave 127.0.0.1:6379 (base 0x400000)
$ xxd -r -p /tmp/dump.hex /tmp/dump.bin
$ objdump --adjust-vma=0x44a350 -D -b binary -m i386:x86-64 /tmp/dump.bin

74557:C 14 Jan 2020 07:51:40.723 # dump of function (hexdump of 635 bytes):
4157415641554154554889fd534881ec3802000064488b042528000000488984242802000031c00fb60689c283e20f0f84db00000080fa010f84d202000080fa020f84f900000080fa030f846005000080fa040f842701000080fa060f844003000080fa050f85ae0700004c8b6608498b1c2448c74424100000000048897c2418c744242800000000c744242c0000000048c744243000000000488b3348895c2420e849e2ffff83f8ff0f845002000048984801442410498b742408488d7c2410ff531831f64889efe822e2ffff83f8ff0f8429020000488b7c2430489848014424104885ff740fe8132a0500488b7c2430e84973feff8b44242885c00f85fd010000488b442410eb0b660f1f440000e82bf2ffff488b8c24280200006448330c25280000000f85a10600004881c4380200005b5d415c415d415e415fc3669083e0f03c200f849f0500003c600f857f060000488b7e084889742408e8ff7b0200488b7424084889c24889ef488b7608e86beeffffeb9e660f1f84000000000083e0f03c500f843c0500003c200f1f000f8575060000488b7e084889742408e87cbdfdff488b7424084989c54889ef488b4608488b704848037028e830e1ffff4c63e04983fcff0f840d050000488d1dd00d09004c8d35dd0d09000f1f4400004c89efe898bdfdff4885c04889c20f84e6040000488b324c8b7a080fb64eff89ca83e20780fa040f87f70500000fb6d2486314934801daffe20f1f80000000008b56f74889efe8b5edffff4883f8ff0f847b040000410fb657ff4901c489d083e0073c040f87b30500000fb6c0496304864c01f0ffe0662e0f1f8400000000000fb756fbebbd662e0f1f8400000000000fb656fdebad662e0f1f84

=== REDIS BUG REPORT END. Make sure to include from START to END. ===

[itamarhaber]

@mitolH this doesn't look like the same crash - please submit a new issue.

[mitolH]

tks for your replay #6781 @itamarhaber the new issue.

[antirez]

Please can you attach your Redis executable to the issue?

…

On Thu, Jan 16, 2020, 07:58 Mitol ***@***.***> wrote: tks for your replay #6781 <#6781> @itamarhaber <https://github.com/itamarhaber> the new issue. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6314>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAQAYFSPC34OAIMQZCNTGLQ6AARBANCNFSM4ILABGOA> .