Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable rocksdb force_consistency_checks for all CFs #5491

Merged
merged 7 commits into from Sep 26, 2019

Conversation

@yiwu-arbug
Copy link
Contributor

commented Sep 20, 2019

Signed-off-by: Yi Wu yiwu@pingcap.com

What have you changed?

Enable force_consistency_checks for all CFs. If rocksdb detects LSM corruption, it will either abort (before facebook/rocksdb@a281822) or return background error in subsequent writes (after that patch). This is to prevent LSM corruption to propagate further and corrupt data in tikv.

What is the type of the changes?

  • Engineering (engineering change which doesn't change any feature or fix any issue)

How is the PR tested?

Unit test and bench_bot.

Does this PR affect documentation (docs) or should it be mentioned in the release notes?

Updated CHANGELOG.md.

Does this PR affect tidb-ansible?

No.

Refer to a related PR or issue link (optional)

The related rocksdb PR is to cherry-pick into corresponding tikv-3.0 and tikv-2.1:
pingcap/rocksdb#120
pingcap/rocksdb#121

Benchmark result if necessary (optional)

Any examples? (optional)

Signed-off-by: Yi Wu <yiwu@pingcap.com>
@yiwu-arbug yiwu-arbug changed the title Enable rocksdb force_consistency_checks for all CFs [DNM] Enable rocksdb force_consistency_checks for all CFs Sep 20, 2019
@yiwu-arbug

This comment has been minimized.

Copy link
Contributor Author

commented Sep 20, 2019

/bench

@yiwu-arbug

This comment has been minimized.

Copy link
Contributor Author

commented Sep 20, 2019

/run-all-tests

@sre-bot

This comment has been minimized.

Copy link
Collaborator

commented Sep 21, 2019

@@                               Benchmark Diff                               @@
================================================================================
tidb: f792e8d7879c13c5ee250c6242269527303369a2
--- tikv: 15082b6cb7d1be3414badc0b74a21b67a505a29e
+++ tikv: 4bea74ce4ed6e2e6a011ef29f335ca291ae84a5a
pd: 4ca2265d180531191dcfc531e468ab8380737a33
================================================================================
test-1: < oltp_read_write >
    * QPS : 37162.04 ± 0.8569% (std=215.72) delta: -0.40%
    * AvgMs : 138.32 ± 0.8618% (std=0.81) delta: 0.42%
    * PercentileMs99 : 264.55 ± 1.0818% (std=2.34) delta: 0.73%
            
test-2: < oltp_point_select >
    * QPS : 82676.10 ± 1.4607% (std=840.66) delta: 1.72%
    * AvgMs : 3.10 ± 1.4540% (std=0.03) delta: -1.68%
    * PercentileMs99 : 6.57 ± 3.2857% (std=0.14) delta: -1.44%
            
test-3: < oltp_insert >
    * QPS : 21779.17 ± 0.8789% (std=133.79) delta: -0.07%
    * AvgMs : 11.75 ± 0.8682% (std=0.07) delta: 0.09%
    * PercentileMs99 : 24.21 ± 1.0658% (std=0.21) delta: 1.08%
            
test-4: < oltp_update_index >
    * QPS : 17164.09 ± 0.9031% (std=101.79) delta: -0.05%
    * AvgMs : 14.81 ± 0.1687% (std=0.02) delta: -0.34%
    * PercentileMs99 : 30.48 ± 1.0827% (std=0.27) delta: -0.01%
            
test-5: < oltp_update_non_index >
    * QPS : 29014.91 ± 0.3287% (std=67.51) delta: 0.31%
    * AvgMs : 8.82 ± 0.3629% (std=0.02) delta: -0.34%
    * PercentileMs99 : 18.48 ± 1.0715% (std=0.16) delta: -0.71%
            

https://perf.pingcap.com

Signed-off-by: Yi Wu <yiwu@pingcap.com>
@yiwu-arbug yiwu-arbug changed the title [DNM] Enable rocksdb force_consistency_checks for all CFs Enable rocksdb force_consistency_checks for all CFs Sep 24, 2019
@yiwu-arbug yiwu-arbug removed the S: DNM label Sep 24, 2019
@yiwu-arbug

This comment has been minimized.

Copy link
Contributor Author

commented Sep 24, 2019

/run-all-tests

yiwu-arbug added 3 commits Sep 24, 2019
Signed-off-by: Yi Wu <yiwu@pingcap.com>
@yiwu-arbug

This comment has been minimized.

Copy link
Contributor Author

commented Sep 24, 2019

/run-all-tests

@yiwu-arbug

This comment has been minimized.

Copy link
Contributor Author

commented Sep 24, 2019

Ready for review. Thanks.

Copy link
Member

left a comment

LGTM

Copy link
Contributor

left a comment

LGTM

@yiwu-arbug

This comment has been minimized.

Copy link
Contributor Author

commented Sep 25, 2019

/run-all-tests

@yiwu-arbug yiwu-arbug merged commit 0ec7cc0 into tikv:master Sep 26, 2019
5 of 6 checks passed
5 of 6 checks passed
idc-jenkins-ci-tikv/integration-cop-push-down-test Jenkins job failed
Details
DCO All commits are signed off!
Details
idc-jenkins-ci-tikv/integration-common-test Jenkins job succeeded.
Details
idc-jenkins-ci-tikv/integration-compatibility-test Jenkins job succeeded.
Details
idc-jenkins-ci-tikv/integration-ddl-test Jenkins job succeeded.
Details
idc-jenkins-ci/test Jenkins job succeeded.
Details
@yiwu-arbug yiwu-arbug deleted the yiwu-arbug:consistency branch Sep 26, 2019
@sre-bot

This comment has been minimized.

Copy link
Collaborator

commented Sep 27, 2019

cherry pick to release-3.0 failed

@sre-bot

This comment has been minimized.

Copy link
Collaborator

commented Sep 27, 2019

cherry pick to release-2.1 failed

yiwu-arbug added a commit to yiwu-arbug/tikv that referenced this pull request Oct 16, 2019
Signed-off-by: Yi Wu <yiwu@pingcap.com>

<!--
Thank you for contributing to TiKV!

If you haven't already, please read TiKV's [CONTRIBUTING](https://github.com/tikv/tikv/blob/master/CONTRIBUTING.md) document.

If you're unsure about anything, just ask; somebody should be along to answer within a day or two.
-->

Enable `force_consistency_checks` for all CFs. If rocksdb detects LSM corruption, it will either abort (before facebook/rocksdb@a281822) or return background error in subsequent writes (after that patch). This is to prevent LSM corruption to propagate further and corrupt data in tikv.

- Engineering (engineering change which doesn't change any feature or fix any issue)

Unit test and bench_bot.

Updated CHANGELOG.md.

No.

The related rocksdb PR is to cherry-pick into corresponding tikv-3.0 and tikv-2.1:
pingcap/rocksdb#120
pingcap/rocksdb#121
yiwu-arbug added a commit to yiwu-arbug/tikv that referenced this pull request Oct 16, 2019
Signed-off-by: Yi Wu <yiwu@pingcap.com>

<!--
Thank you for contributing to TiKV!

If you haven't already, please read TiKV's [CONTRIBUTING](https://github.com/tikv/tikv/blob/master/CONTRIBUTING.md) document.

If you're unsure about anything, just ask; somebody should be along to answer within a day or two.
-->

Enable `force_consistency_checks` for all CFs. If rocksdb detects LSM corruption, it will either abort (before facebook/rocksdb@a281822) or return background error in subsequent writes (after that patch). This is to prevent LSM corruption to propagate further and corrupt data in tikv.

- Engineering (engineering change which doesn't change any feature or fix any issue)

Unit test and bench_bot.

Updated CHANGELOG.md.

No.

The related rocksdb PR is to cherry-pick into corresponding tikv-3.0 and tikv-2.1:
pingcap/rocksdb#120
pingcap/rocksdb#121
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.