Skip to content

Loading…

Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis. #990

Closed
webdev opened this Issue · 2 comments

2 participants

@webdev

I know this has been covered before, but I still don't feel like I saw the solution.

We are on EC2 with ephemeral disk.

We used appendfsync everysec directive, it ran just fine for 2 weeks, and all of a sudden it started slowing down, and was pretty much brought to its knees, so we had to set appendfsync to, which helped, but I understand that it's the riskiest option and would love to see if there is anything else that could be tweaked to make everysec option work.

Our AOF file is far from huge, it's only ~95Mb.

Below is our info

redis 127.0.0.1:6379> info
# Server
redis_version:2.6.10
redis_git_sha1:00000000
redis_git_dirty:0
redis_mode:standalone
os:Linux 2.6.32-31-server x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.3
process_id:18576
run_id:62d8f682d82a55167f02784a17c9343a5bd7876a
tcp_port:6379
uptime_in_seconds:61125
uptime_in_days:0
lru_clock:2061576

# Clients
connected_clients:39
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:145629000
used_memory_human:138.88M
used_memory_rss:160317440
used_memory_peak:157994816
used_memory_peak_human:150.68M
used_memory_lua:31744
mem_fragmentation_ratio:1.10
mem_allocator:jemalloc-3.2.0

# Persistence
loading:0
rdb_changes_since_last_save:68512
rdb_bgsave_in_progress:0
rdb_last_save_time:1362783573
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:12
rdb_current_bgsave_time_sec:-1
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:13
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_current_size:102364560
aof_base_size:79834947
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:325

# Stats
total_connections_received:41420
total_commands_processed:13596179
instantaneous_ops_per_sec:2
rejected_connections:0
expired_keys:63000
evicted_keys:0
keyspace_hits:81805433
keyspace_misses:9700290
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:215364

# Replication
role:master
connected_slaves:1
slave0:127.0.0.1,6379,online

# CPU
used_cpu_sys:928.57
used_cpu_user:933.89
used_cpu_sys_children:2.00
used_cpu_user_children:7.82

# Keyspace
db0:keys=295111,expires=19866
@ajsharp

ping @antirez @pietern any ideas on this? Thanks.

@webdev

I figured out the issue. My server is in the cloud and we had noisy neighbor that ate up all the IO. So the disk was waiting on AOF syncing.

@webdev webdev closed this
@JackieXie168 JackieXie168 pushed a commit that referenced this issue
@AtnNn AtnNn Fetch protobuf if check fails
Review 894 by @tryneus
Closes #990
3c11cd5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.