Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

redis 3.2.1 BGREWRITEAOF fail #5295

Open
austingod opened this issue Aug 29, 2018 · 2 comments
Open

redis 3.2.1 BGREWRITEAOF fail #5295

austingod opened this issue Aug 29, 2018 · 2 comments

Comments

@austingod
Copy link

=== REDIS BUG REPORT START: Cut & paste starting from here ===
9938:C 29 Aug 14:33:05.312 # === ASSERTION FAILED ===
9938:C 29 Aug 14:33:05.312 # ==> ziplist.c:411 'NULL' is not true
9938:C 29 Aug 14:33:05.312 # (forcing SIGSEGV to print the bug report.)
9938:C 29 Aug 14:33:05.312 # Redis 3.2.1 crashed by signal: 11
9938:C 29 Aug 14:33:05.312 # Crashed running the instuction at: 0x45b04f
9938:C 29 Aug 14:33:05.312 # Accessing address: 0xffffffffffffffff
9938:C 29 Aug 14:33:05.312 # Accessing address: 0xffffffffffffffff
9938:C 29 Aug 14:33:05.312 # Failed assertion: NULL (ziplist.c:411)
LL (ziplist.c:411)

------ STACK TRACE ------
EIP:
redis-aof-rewrite *:6379(_serverAssert+0x6f)[0x45b04f]

Backtrace:
redis-aof-rewrite *:6379(logStackTrace+0x3c)[0x45b45c]
redis-aof-rewrite *:6379(sigsegvHandler+0xa1)[0x45c341]
/lib64/libpthread.so.0(+0xf710)[0x7f9340d73710]
/lib64/libpthread.so.0(+0xf710)[0x7f9340d73710]
redis-aof-rewrite *:6379(_serverAssert+0x6f)[0x45b04f]
redis-aof-rewrite *:6379[0x4303ef]
redis-aof-rewrite *:6379(ziplistGet+0x72)[0x430602]
redis-aof-rewrite *:6379(quicklistNext+0x1ec)[0x41e34c]
redis-aof-rewrite *:6379(rewriteListObject+0x8a)[0x457eba]
redis-aof-rewrite *:6379(rewriteAppendOnlyFile+0x4ab)[0x458f1b]
redis-aof-rewrite *:6379(rewriteAppendOnlyFileBackground+0x87)[0x459397]
redis-aof-rewrite *:6379(bgrewriteaofCommand+0x3a)[0x45952a]
redis-aof-rewrite *:6379(call+0x72)[0x424042]
redis-aof-rewrite *:6379(processCommand+0x365)[0x428bc5]
redis-aof-rewrite *:6379(processInputBuffer+0x109)[0x434ec9]
redis-aof-rewrite *:6379(aeProcessEvents+0x13d)[0x41f71d]
redis-aof-rewrite *:6379(aeMain+0x2b)[0x41fa1b]
redis-aof-rewrite *:6379(main+0x371)[0x4270c1]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f93409eed5d]
redis-aof-rewrite *:6379[0x41cf09]

------ INFO OUTPUT ------

Server

redis_version:3.2.1
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:528409f24ee40d15
redis_mode:standalone
os:Linux 3.10.106-1-tlinux2-0044 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.6
process_id:9938
run_id:a5a6891bc84dd060dfc3410b6d97f06dcebaa146
tcp_port:6379
uptime_in_seconds:14040
uptime_in_days:0
hz:10
lru_clock:8797719
executable:/data/codecc/redis-3.2.1/src/./redis-server
config_file:/data/codecc/redis-3.2.1/redis.conf

Clients

connected_clients:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

Memory

used_memory:19409982896
used_memory_human:18.08G
used_memory_rss:19902554112
used_memory_rss_human:18.54G
used_memory_peak:19409982896
used_memory_peak_human:18.08G
total_system_memory:134801420288
total_system_memory_human:125.54G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
mem_fragmentation_ratio:1.03
mem_allocator:jemalloc-4.0.3

Persistence

loading:0
rdb_changes_since_last_save:0
rdb_bgsave_in_progress:0
rdb_last_save_time:1535510335
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:34
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:err
aof_last_write_status:ok

Stats

total_connections_received:3
total_commands_processed:9
instantaneous_ops_per_sec:0
total_net_input_bytes:348
total_net_output_bytes:5963702
instantaneous_input_kbps:0.00
instantaneous_output_kbps:0.00
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
evicted_keys:0
keyspace_hits:1
keyspace_misses:2
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:272540
migrate_cached_sockets:0

Replication

role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

CPU

used_cpu_sys:4.48
used_cpu_user:0.06
used_cpu_sys_children:0.00
used_cpu_user_children:0.00

Commandstats

cmdstat_get:calls=2,usec=8,usec_per_call=4.00
cmdstat_lrange:calls=1,usec=14,usec_per_call=14.00
cmdstat_auth:calls=3,usec=6,usec_per_call=2.00
cmdstat_bgrewriteaof:calls=2,usec=540559,usec_per_call=270279.50
cmdstat_command:calls=1,usec=414,usec_per_call=414.00

Cluster

cluster_enabled:0

Keyspace

db0:keys=7218573,expires=0,avg_ttl=0
db1:keys=1,expires=0,avg_ttl=0
hash_init_value: 1535278348

------ CLIENT LIST OUTPUT ------
id=4 addr=127.0.0.1:58732 fd=6 name= age=820 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=bgrewriteaof

------ CURRENT CLIENT INFO ------

------ CLIENT LIST OUTPUT ------
id=4 addr=127.0.0.1:58732 fd=6 name= age=820 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=bgrewriteaof

------ CURRENT CLIENT INFO ------
id=4 addr=127.0.0.1:58732 fd=6 name= age=820 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=bgrewriteaof
argv[0]: 'BGREWRITEAOF'

------ REGISTERS ------
9938:C 29 Aug 14:33:05.313 #
RAX:0000000000000000 RBX:000000000000019b
RCX:0000000000000000 RDX:00007f9340d5fed8
RDI:0000000000000000 RSI:0000000000000000
RBP:00000000004e0b20 RSP:00007fff3c83e240
R8 :00007f934150e740 R9 :0000000000000000
R10:7562206568742074 R11:0000000000000206
R12:00000000004e10b7 R13:00007fff3c83e338
R14:00000000004308f0 R15:0000000000000000
RIP:000000000045b04f EFL:0000000000010206
CSGSFS:0000000000000033
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e24f) -> 0000000000000001
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e24f) -> 0000000000000001
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e24e) -> 00007fff3c83e310
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e24d) -> 00007f8e98461d10
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e24d) -> 00007f8e98461d10
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e24c) -> 00007fff3c83e4c0
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e24b) -> 00007f91a19c1c7f
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e24a) -> 000000c800000002
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e249) -> 0000000000000001
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e248) -> 0000000000000001
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e248) -> 0000000000000001
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e247) -> 0000000000430602
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e246) -> 0000000000000002
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e245) -> 00007f9340a37a4d
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e244) -> 0000000000000004
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e244) -> 0000000000000004
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e243) -> 00000000004303ef
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e242) -> 00007fff3c83e330
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e241) -> 00007fff3c83e328
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e241) -> 00007fff3c83e328
9938:C 29 Aug 14:33:05.313 # (00007fff3c83e240) -> 00007f91a19c1c7f

------ FAST MEMORY TEST ------
*** Preparing to test memory region 721000 (114688 bytes)
*** Preparing to test memory region 976000 (135168 bytes)
*** Preparing to test memory region 7f8e98400000 (19979567104 bytes)
*** Preparing to test memory region 7f933f3ff000 (8388608 bytes)
*** Preparing to test memory region 7f933fc00000 (8388608 bytes)
*** Preparing to test memory region 7f9340400000 (4194304 bytes)
*** Preparing to test memory region 7f9340d5f000 (20480 bytes)
*** Preparing to test memory region 7f9340f7d000 (16384 bytes)
*** Preparing to test memory region 7f934150e000 (12288 bytes)
*** Preparing to test memory region 7f934151e000 (8192 bytes)
*** Preparing to test memory region 7f9341624000 (16384 bytes)
*** Preparing to test memory region 7f934162a000 (4096 bytes)
..O.O.O.O.O.O.O.O.O.O.O.O

Fast memory test PASSED, however your memory can still be broken. Please run a memory test for several hours if possible.
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
@itamarhaber
Copy link
Member

Hello @youngsunyy

Thanks for reporting this crash - the error seems quite serious (ziplist heisenbug?).

I recommend that you upgrade to a newer Redis version as v3.2.1 is quite ripe.

@austingod
Copy link
Author

Hello @youngsunyy

Thanks for reporting this crash - the error seems quite serious (ziplist heisenbug?).

I recommend that you upgrade to a newer Redis version as v3.2.1 is quite ripe.

Hello @itamarhaber
Thank you for your answer, my Redis version is v3.2.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@itamarhaber @austingod