Redis 2.8.4 Crashes (Bug report included)

[praveenbalaji-blippar]

=== REDIS BUG REPORT START: Cut & paste starting from here ===
[23934] 27 Oct 19:29:31.807 # Redis 2.8.4 crashed by signal: 11
[23934] 27 Oct 19:29:31.807 # Failed assertion: (:0)
[23934] 27 Oct 19:29:31.807 # --- STACK TRACE
redis-server *:6394(logStackTrace+0x33)[0x442ec3]
/usr/lib/x86_64-linux-gnu/libjemalloc.so.1(+0xb565)[0x7fbc90e58565]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7fbc90c3f340]
/usr/lib/x86_64-linux-gnu/libjemalloc.so.1(+0xb565)[0x7fbc90e58565]
/usr/lib/x86_64-linux-gnu/libjemalloc.so.1(+0xb6cc)[0x7fbc90e586cc]
/usr/lib/x86_64-linux-gnu/libjemalloc.so.1(+0xcedf)[0x7fbc90e59edf]
/usr/lib/x86_64-linux-gnu/libjemalloc.so.1(+0xd076)[0x7fbc90e5a076]
/usr/lib/x86_64-linux-gnu/libjemalloc.so.1(+0x260ef)[0x7fbc90e730ef]
/usr/lib/x86_64-linux-gnu/libjemalloc.so.1(+0xe70e)[0x7fbc90e5b70e]
/usr/lib/x86_64-linux-gnu/libjemalloc.so.1(realloc+0x2cd)[0x7fbc90e5238d]
redis-server *:6394(zrealloc+0x26)[0x41c866]
redis-server *:6394[0x41ff77]
redis-server *:6394(listTypePush+0xac)[0x42f11c]
redis-server *:6394(pushGenericCommand+0x90)[0x4302d0]
redis-server *:6394(call+0x86)[0x418686]
redis-server *:6394(processCommand+0x36d)[0x41a78d]
redis-server *:6394(processInputBuffer+0x4f)[0x422fdf]
redis-server *:6394(readQueryFromClient+0xad)[0x4230fd]
redis-server *:6394(aeProcessEvents+0x250)[0x413e30]
redis-server *:6394(aeMain+0x2b)[0x41406b]
redis-server *:6394(main+0x31a)[0x412e3a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fbc9088bec5]
redis-server *:6394[0x412f5e]
[23934] 27 Oct 19:29:31.819 # --- INFO OUTPUT
[23934] 27 Oct 19:29:31.819 # # Server
redis_version:2.8.4
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:a44a05d76f06a5d9
redis_mode:standalone
os:Linux 3.13.0-48-generic x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.8.2
process_id:23934
run_id:96e64d890328d75f859cba383ca35f455dc1f32c
tcp_port:6394
uptime_in_seconds:1919
uptime_in_days:0
hz:10
lru_clock:1991081
config_file:

Clients

connected_clients:1
client_longest_output_list:0
client_biggest_input_buf:4863
blocked_clients:0

Memory

used_memory:13900739024
used_memory_human:12.95G
used_memory_rss:14194286592
used_memory_peak:13900007280
used_memory_peak_human:12.95G
used_memory_lua:33792
mem_fragmentation_ratio:1.02
mem_allocator:jemalloc-3.4.1

Persistence

loading:0
rdb_changes_since_last_save:258765061
rdb_bgsave_in_progress:0
rdb_last_save_time:1445972252
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok

Stats

total_connections_received:5
total_commands_processed:257516301
instantaneous_ops_per_sec:477731
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:1
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0

Replication

role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

CPU

used_cpu_sys:88.16
used_cpu_user:434.63
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

Commandstats

cmdstat_lpush:calls=257516295,usec=262762405,usec_per_call=1.02
cmdstat_lrange:calls=1,usec=74,usec_per_call=74.00
cmdstat_keys:calls=3,usec=1393265,usec_per_call=464421.66
cmdstat_flushall:calls=2,usec=2036848,usec_per_call=1018424.00

Keyspace

db0:keys=19264057,expires=0,avg_ttl=0
hash_init_value: 1445272964

[23934] 27 Oct 19:29:31.819 # --- CLIENT LIST OUTPUT
[23934] 27 Oct 19:29:31.821 # addr=127.0.0.1:58010 fd=6 name= age=528 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=4863 qbuf-free=27931 obl=788 oll=0 omem=0 events=rw cmd=lpush

[23934] 27 Oct 19:29:31.821 # --- CURRENT CLIENT INFO
[23934] 27 Oct 19:29:31.821 # client: addr=127.0.0.1:58010 fd=6 name= age=528 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=4863 qbuf-free=27931 obl=788 oll=0 omem=0 events=rw cmd=lpush
[23934] 27 Oct 19:29:31.821 # argv[0]: 'LPUSH'
[23934] 27 Oct 19:29:31.821 # argv[1]: 'Call_of_Duty:_Modern_Warfare_3.olnk'
[23934] 27 Oct 19:29:31.821 # argv[2]: 'National_Gendarmerie_Intervention_Group'
[23934] 27 Oct 19:29:31.821 # key 'Call_of_Duty:_Modern_Warfare_3.olnk' found in DB containing the following object:
[23934] 27 Oct 19:29:31.821 # Object type: 1
[23934] 27 Oct 19:29:31.821 # Object encoding: 5
[23934] 27 Oct 19:29:31.821 # Object refcount: 1
[23934] 27 Oct 19:29:31.821 # List length: 63
[23934] 27 Oct 19:29:31.821 # --- REGISTERS
[23934] 27 Oct 19:29:31.821 #
RAX:0000000000000000 RBX:0000000000000000
RCX:00000000000003fa RDX:0000000000000006
RDI:00007fbc904000c0 RSI:0000000000000000
RBP:00007fbc904000c0 RSP:00007fff98d68810
R8 :0000000000000000 R9 :0000000000000000
R10:0000000000000022 R11:0000000000000246
R12:0000000000000006 R13:00007fbc910803a0
R14:00007fbc904000c8 R15:0000000000000016
RIP:00007fbc90e58565 EFL:0000000000010202
CSGSFS:0000000000000033
[23934] 27 Oct 19:29:31.821 # (00007fff98d6881f) -> 00007fbc90e59edf
[23934] 27 Oct 19:29:31.821 # (00007fff98d6881e) -> 00007fbc904000c0
[23934] 27 Oct 19:29:31.821 # (00007fff98d6881d) -> 0000000000000000
[23934] 27 Oct 19:29:31.821 # (00007fff98d6881c) -> 00007fbc90400ef0
[23934] 27 Oct 19:29:31.821 # (00007fff98d6881b) -> 00007fbc91080110
[23934] 27 Oct 19:29:31.821 # (00007fff98d6881a) -> 0000000000000001
[23934] 27 Oct 19:29:31.821 # (00007fff98d68819) -> 00007fbc90e58c26
[23934] 27 Oct 19:29:31.821 # (00007fff98d68818) -> 0000000000010001
[23934] 27 Oct 19:29:31.821 # (00007fff98d68817) -> 00007fbc90e586cc
[23934] 27 Oct 19:29:31.821 # (00007fff98d68816) -> 00007fbc910803a0
[23934] 27 Oct 19:29:31.821 # (00007fff98d68815) -> 00007fbc904000c8
[23934] 27 Oct 19:29:31.821 # (00007fff98d68814) -> 00007fbc904000c0
[23934] 27 Oct 19:29:31.821 # (00007fff98d68813) -> 0000000000000000
[23934] 27 Oct 19:29:31.821 # (00007fff98d68812) -> 00007fbc904000c0
[23934] 27 Oct 19:29:31.821 # (00007fff98d68811) -> 0000000000000000
[23934] 27 Oct 19:29:31.821 # (00007fff98d68810) -> 0000000000009000
[23934] 27 Oct 19:29:31.821 # --- FAST MEMORY TEST
[23934] 27 Oct 19:29:31.821 # Bio thread for job type #0 terminated
[23934] 27 Oct 19:29:31.821 # Bio thread for job type #1 terminated
Testing 69b000 86016
Testing 7fb941400000 14185136128
Testing 7fbc8efff000 8388608
Testing 7fbc8f800000 16777216
Testing 7fbc90c2a000 20480
Testing 7fbc90e49000 16384
Testing 7fbc91080000 4096
Testing 7fbc91598000 16384
Testing 7fbc915a5000 16384
Testing 7fbc915ab000 4096

[antirez]

Hello, this looks like a possible jemalloc corruption, it is more likely to be so if the system was near out of memory. Another issue may be bad memory chips. A Redis bug is not impossible but very unlikely, there are no known issues that would cause this, but nothing is impossible. My suggestion is to upgrade to latest Redis version available, even if the jemalloc bug that fixes this (very rare) issue is yet not shipped with Redis 3.0.x, but will be with Redis 3.2.x and possibly with next patch levels of 3.0 as well. Closing since this is mostly a non actionable issue. Thanks.