Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Multi Part AOF mechanism to avoid AOFRW overheads. #9788

Merged
merged 65 commits into from
Jan 3, 2022
Merged
Show file tree
Hide file tree
Changes from 56 commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
4be5c24
Support Multi Part Aof
chenyang8094 Nov 10, 2021
a55874a
Fix typo
chenyang8094 Nov 25, 2021
bb5e815
Make openNewIncrAofForAppend atomic
chenyang8094 Nov 25, 2021
51eb9d6
Add comments
chenyang8094 Nov 25, 2021
3652b2f
Add comments
chenyang8094 Nov 25, 2021
8b24ff1
Update comments
chenyang8094 Nov 25, 2021
31d6a3e
Remove aof-child-rewrite-delay config
chenyang8094 Nov 25, 2021
3989593
Change file suffix
chenyang8094 Nov 25, 2021
43b6d20
Update test
chenyang8094 Nov 25, 2021
7be0c38
Update test
chenyang8094 Nov 26, 2021
41084af
Update aof_last_incr_size when server start
chenyang8094 Nov 26, 2021
b63dd4f
Trigger ci
chenyang8094 Nov 26, 2021
71eb11f
Unlink base file when rename failed
chenyang8094 Nov 28, 2021
ef150ae
Add format suffix to file name
chenyang8094 Nov 28, 2021
47e8d22
Merge branch 'unstable' into feature-multi-aof-use-meta
chenyang8094 Nov 28, 2021
3d079a3
Change TCL test name
chenyang8094 Nov 29, 2021
243ebbb
Add comments
chenyang8094 Nov 29, 2021
e9febd0
Fix typo
chenyang8094 Nov 30, 2021
e77ea5e
Add error msg when load err
chenyang8094 Nov 30, 2021
1f8d724
Use strchr to judge line
chenyang8094 Nov 30, 2021
69e86ff
Refact aofLoadManifestFromDisk
chenyang8094 Dec 1, 2021
341484f
Enhance test
chenyang8094 Dec 1, 2021
d410061
Add comments
chenyang8094 Dec 1, 2021
a1e0cc2
Merge branch 'unstable' into feature-multi-aof-use-meta
chenyang8094 Dec 1, 2021
74394dc
Scheduled an AOFRW when upgrading from an old version redis
chenyang8094 Dec 1, 2021
b510cdf
Merge branch 'unstable' into feature-multi-aof-use-meta
chenyang8094 Dec 3, 2021
8d59d96
Use dir to track AOFs
chenyang8094 Dec 16, 2021
ee6c516
Use aofname as temp dir prefix
chenyang8094 Dec 17, 2021
2e800e5
Use redis_stat to avoid CodeQL
chenyang8094 Dec 17, 2021
f3d4a26
Remove dir check
chenyang8094 Dec 20, 2021
2eaebec
Merge branch 'unstable' into feature-multi-aof-use-meta
chenyang8094 Dec 20, 2021
b71354a
Fix codeql
chenyang8094 Dec 20, 2021
f506c9b
Fix crash in sentinel mode
chenyang8094 Dec 20, 2021
7ac8973
Change ret type of dirCreateIfMissing
chenyang8094 Dec 21, 2021
089e7b3
Merge branch 'unstable' into feature-multi-aof-use-meta
chenyang8094 Dec 21, 2021
eda8aff
Add appenddirname configration
chenyang8094 Dec 22, 2021
94f8f03
Fix memleak
chenyang8094 Dec 22, 2021
3f38035
Code format
chenyang8094 Dec 22, 2021
55cf2a3
Update config comments
chenyang8094 Dec 24, 2021
9b77d3a
Chnage name rule
chenyang8094 Dec 24, 2021
85b066f
Update comments
chenyang8094 Dec 24, 2021
1abe145
Update comments
chenyang8094 Dec 25, 2021
b8bf24a
Update comments
chenyang8094 Dec 27, 2021
aed4631
Merge branch 'unstable' into feature-multi-aof-use-meta
chenyang8094 Dec 27, 2021
7f7dda8
Only the last aof can be truncated
chenyang8094 Dec 27, 2021
83d3b96
Update test
chenyang8094 Dec 28, 2021
0fbaba6
Enhance test
chenyang8094 Dec 28, 2021
4d4e2be
Fix test
chenyang8094 Dec 28, 2021
29d7d74
Fix test
chenyang8094 Dec 28, 2021
04cf84e
Remove aof_rewrite_buffer_length field
chenyang8094 Dec 28, 2021
5a17380
Update conf comments
chenyang8094 Dec 28, 2021
cd367d4
Update conf comments
chenyang8094 Dec 28, 2021
be84849
Update conf comments
chenyang8094 Dec 28, 2021
59b0f15
Update test
chenyang8094 Dec 28, 2021
ed6d1e3
Update rdb-preamble.aof
chenyang8094 Dec 28, 2021
1ce724d
Add load check
chenyang8094 Dec 28, 2021
b26f50c
Valgrind suppression
chenyang8094 Dec 29, 2021
3467f51
Update .gitignore
chenyang8094 Dec 29, 2021
6a62c8d
Make sure aofManifestFree safe when prepareForShutdown
chenyang8094 Dec 30, 2021
3619111
Update log messages
chenyang8094 Dec 31, 2021
ed656d3
Update comments
chenyang8094 Dec 31, 2021
c2c108b
Add ENOENT check
chenyang8094 Dec 31, 2021
f0df365
Merge remote-tracking branch 'origin/unstable' into feature-multi-aof…
oranagra Jan 2, 2022
8ab1c3b
Code review cleanups.
yossigo Jan 3, 2022
f4bae58
Merge branch 'unstable' into feature-multi-aof-use-meta
chenyang8094 Jan 3, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 30 additions & 10 deletions redis.conf
Original file line number Diff line number Diff line change
Expand Up @@ -1330,10 +1330,34 @@ disable-thp yes

appendonly no

# The name of the append only file (default: "appendonly.aof")
# The base name of the append only file.
#
# Redis can persist the data to a set of append-only files, they are divided
# into two types, one is the the base type, it represents an initial (RDB
# or AOF format) snapshot of the data present when the AOF is rewritten. The
chenyang8094 marked this conversation as resolved.
Show resolved Hide resolved
# others are incremental type, they contain the incremental commands since the last
# base AOF file was created.
#
# In order to manage these AOF files, Redis uses a manifest file to track them.
#
# These files mentioned above have a certain naming rule. The appendfilename
# config will be used as the base part of the file name, which will be added
# suffixes to denote role and sequence number.
#
# For example:
#
# The base could be: appendonly.aof.1.base.rdb
# The incremental ones: appendonly.aof.1.incr.aof, appendonly.aof.2.incr.aof
# The manifest file: appendonly.aof.manifest

appendfilename "appendonly.aof"
oranagra marked this conversation as resolved.
Show resolved Hide resolved

# The directory name of the append only files.
#
# All AOF files and manifest file will be placed in this directory.
oranagra marked this conversation as resolved.
Show resolved Hide resolved

appenddirname "appendonlydir"

# The fsync() call tells the Operating System to actually write data on disk
# instead of waiting for more data in the output buffer. Some OS will really flush
# data on disk, some other OS will just try to do it ASAP.
Expand Down Expand Up @@ -1426,15 +1450,11 @@ auto-aof-rewrite-min-size 64mb
# will be found.
aof-load-truncated yes

# When rewriting the AOF file, Redis is able to use an RDB preamble in the
# AOF file for faster rewrites and recoveries. When this option is turned
# on the rewritten AOF file is composed of two different stanzas:
#
# [RDB file][AOF tail]
#
# When loading, Redis recognizes that the AOF file starts with the "REDIS"
# string and loads the prefixed RDB file, then continues loading the AOF
# tail.
# The base AOF encodeing type.
#
# The base file can be in AOF (commands) format or RDB (serialized) format.
# When this option is turned on, the base file uses RDB format, otherwise
# it uses AOF format.
aof-use-rdb-preamble yes

# Redis supports recording timestamp annotations in the AOF to support restoring
Expand Down
Loading