New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

config: enlarge max-manifest-file-size to 128MB #4140

Merged
merged 1 commit into from Jan 29, 2019

Conversation

Projects
None yet
3 participants
@huachaohuang
Copy link
Member

huachaohuang commented Jan 29, 2019

What have you changed? (mandatory)

There two benefits:

  1. Save more versions for debugging when necessary.
  2. Avoid rewritting the whole manifest file frequently.

However, a larger manifest file needs more time to recover.
According to test, a 128MB manifest file takes about 4s to recover.

More information about the manifest file: https://github.com/facebook/rocksdb/wiki/MANIFEST

What are the type of the changes? (mandatory)

  • Improvement (non-breaking change which is an improvement to an existing feature)

How has this PR been tested? (mandatory)

Unit test

Does this PR affect documentation (docs) update? (mandatory)

No

Does this PR affect tidb-ansible update? (mandatory)

No

config: enlarge max-manifest-file-size to 128MB
There two benefits:
1. Save more versions for debugging when necessary.
2. Avoid rewritting the whole manifest file frequently.

However, a larger manifest file needs more time to recover.
According to test, a 128MB manifest file takes about 4s to recover.

Signed-off-by: Huachao Huang <huachao.huang@gmail.com>

@huachaohuang huachaohuang force-pushed the huachaohuang:manifest branch from 4c4c965 to 4261983 Jan 29, 2019

@siddontang
Copy link
Contributor

siddontang left a comment

LGTM

@@ -515,7 +515,7 @@ impl Default for DbConfig {
wal_size_limit: ReadableSize::kb(0),
max_total_wal_size: ReadableSize::gb(4),
max_background_jobs: 6,
max_manifest_file_size: ReadableSize::mb(20),
max_manifest_file_size: ReadableSize::mb(128),

This comment has been minimized.

@siddontang

siddontang Jan 29, 2019

Contributor

do we need to adjust Raft DB?

This comment has been minimized.

@huachaohuang

huachaohuang Jan 29, 2019

Author Member

It doesn't matter, RaftDB has only a few files.

@huachaohuang

This comment has been minimized.

Copy link
Member Author

huachaohuang commented Jan 29, 2019

/rebuild

@huachaohuang huachaohuang merged commit be84bd5 into tikv:master Jan 29, 2019

2 checks passed

DCO All commits are signed off!
Details
idc-jenkins-ci/test Jenkins job succeeded.
Details

@huachaohuang huachaohuang deleted the huachaohuang:manifest branch Jan 29, 2019

huachaohuang added a commit to huachaohuang/tikv that referenced this pull request Feb 20, 2019

config: enlarge max-manifest-file-size to 128MB (tikv#4140)
There two benefits:
1. Save more versions for debugging when necessary.
2. Avoid rewritting the whole manifest file frequently.

However, a larger manifest file needs more time to recover.
According to test, a 128MB manifest file takes about 4s to recover.

Signed-off-by: Huachao Huang <huachao.huang@gmail.com>

huachaohuang added a commit to huachaohuang/tikv that referenced this pull request Feb 20, 2019

config: enlarge max-manifest-file-size to 128MB (tikv#4140)
There two benefits:
1. Save more versions for debugging when necessary.
2. Avoid rewritting the whole manifest file frequently.

However, a larger manifest file needs more time to recover.
According to test, a 128MB manifest file takes about 4s to recover.

Signed-off-by: Huachao Huang <huachao.huang@gmail.com>

huachaohuang added a commit that referenced this pull request Feb 22, 2019

config: enlarge max-manifest-file-size to 128MB (#4140) (#4237)
There two benefits:
1. Save more versions for debugging when necessary.
2. Avoid rewritting the whole manifest file frequently.

However, a larger manifest file needs more time to recover.
According to test, a 128MB manifest file takes about 4s to recover.

Signed-off-by: Huachao Huang <huachao.huang@gmail.com>

huachaohuang added a commit that referenced this pull request Feb 22, 2019

config: enlarge max-manifest-file-size to 128MB (#4140) (#4248)
There two benefits:
1. Save more versions for debugging when necessary.
2. Avoid rewritting the whole manifest file frequently.

However, a larger manifest file needs more time to recover.
According to test, a 128MB manifest file takes about 4s to recover.

Signed-off-by: Huachao Huang <huachao.huang@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment