Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add blob-run-mode for titan #4865

Merged
merged 8 commits into from Jun 12, 2019

Conversation

@tabokie
Copy link
Member

tabokie commented Jun 10, 2019

What have you changed? (mandatory)

blob-run-mode is used for smoothly downgrading Titan to native RocksDB.
There are three modes for blob-run-mode option:

  • "normal": Titan process read/write as normal.
  • "read-only": Titan stop writing value into blob log during flush and compaction. Existing values in blob log is still readable.
  • "fallback": On flush and compaction, Titan will convert blob index into real value, by reading from blob log, and store the value in SST file.

This PR updates TitanDBOptions and BlobRunMode w.r.t. upstream change.

What are the type of the changes? (mandatory)

  • Improvement (change which is an improvement to an existing feature)

How has this PR been tested? (mandatory)

Unit test.

Does this PR affect documentation (docs) or release note? (mandatory)

No.

Does this PR affect tidb-ansible update? (mandatory)

No.

Signed-off-by: tabokie <xy.tao@outlook.com>
@tabokie tabokie added the C: Titan label Jun 10, 2019
@tabokie tabokie requested review from yiwu-arbug and Connor1996 Jun 10, 2019
Copy link
Member

Connor1996 left a comment

LGTM

@@ -116,6 +116,24 @@ pub mod compression_type_level_serde {
}
}

#[derive(Copy, Clone, Debug, PartialEq, Serialize, Deserialize)]
#[serde(rename_all = "kebab-case")]
pub enum BlobRunMode {

This comment has been minimized.

Copy link
@yiwu-arbug

yiwu-arbug Jun 10, 2019

Contributor

Do we need to implement serializer and deserializer for BlobRunMode, like DBCompressionType above?

This comment has been minimized.

Copy link
@tabokie

tabokie Jun 11, 2019

Author Member

No, what's above is for serializing DBCompressionType array.

This comment has been minimized.

Copy link
@yiwu-arbug

yiwu-arbug Jun 11, 2019

Contributor

How does, e.g. blob-run-mode = "normal" deserialize into numeric value? Since DBTitanDBBlobRunMode is a numeric enum.

This comment has been minimized.

Copy link
@tabokie

tabokie Jun 11, 2019

Author Member

All done by macro from serde I think.

This comment has been minimized.

Copy link
@yiwu-arbug

yiwu-arbug Jun 11, 2019

Contributor

wow. looks like magic to me.

Signed-off-by: tabokie <xy.tao@outlook.com>
@tabokie

This comment has been minimized.

Copy link
Member Author

tabokie commented Jun 11, 2019

/run-all-tests

@tabokie

This comment has been minimized.

Copy link
Member Author

tabokie commented Jun 11, 2019

/run-all-tests

@tabokie tabokie force-pushed the tabokie:add-option branch from ecb7b12 to 32f8905 Jun 11, 2019
Signed-off-by: tabokie <xy.tao@outlook.com>
@tabokie tabokie force-pushed the tabokie:add-option branch from 32f8905 to 36b7449 Jun 11, 2019
@tabokie

This comment has been minimized.

Copy link
Member Author

tabokie commented Jun 11, 2019

/run-all-tests

Copy link
Contributor

yiwu-arbug left a comment

LGTM.

Copy link
Member

Connor1996 left a comment

LGTM

@Connor1996 Connor1996 added the S: LGT2 label Jun 12, 2019
tabokie added 3 commits Jun 12, 2019
@tabokie tabokie merged commit 0af7a0a into tikv:master Jun 12, 2019
2 checks passed
2 checks passed
DCO All commits are signed off!
Details
idc-jenkins-ci/test Jenkins job succeeded.
Details
Connor1996 added a commit to Connor1996/tikv that referenced this pull request Jul 15, 2019
* add blob-run-mode for titan

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comment

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix config test

Signed-off-by: tabokie <xy.tao@outlook.com>
Connor1996 added a commit to Connor1996/tikv that referenced this pull request Jul 15, 2019
* add blob-run-mode for titan

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comment

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix config test

Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Connor1996 added a commit to Connor1996/tikv that referenced this pull request Jul 15, 2019
* add blob-run-mode for titan

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comment

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix config test

Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
youjiali1995 added a commit that referenced this pull request Jul 15, 2019
* count size of blob files in used size (#5060)

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* Check max required open files for titan (#5026)

* Check max required open files for titan

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>

* Add comment to new change

Signed-off-by: Xiaoguang Sun <sunxiaoguang@zhihu.com>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* Add blob-run-mode for titan (#4865)

* add blob-run-mode for titan

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comment

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix config test

Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* Set blob_run_mode=ReadOnly for non-default CFs (#4991)

* Set blob_run_mode=read_only for non-default CFs

Signed-off-by: Yi Wu <yiwu@pingcap.com>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* engine: add titan metrics (#4772)

* add titan metrics

Signed-off-by: Connor1996 <zbk602423539@gmail.com>

* metrics: add titan metrics (#4836)

* Add titan blob-related metrics

Signed-off-by: tabokie <xy.tao@outlook.com>

* fmt

Signed-off-by: tabokie <xy.tao@outlook.com>

* reset cargo

Signed-off-by: tabokie <xy.tao@outlook.com>
Signed-off-by: Connor1996 <zbk602423539@gmail.com>
sticnarf pushed a commit to sticnarf/tikv that referenced this pull request Oct 27, 2019
* add blob-run-mode for titan

Signed-off-by: tabokie <xy.tao@outlook.com>

* address comment

Signed-off-by: tabokie <xy.tao@outlook.com>

* fix config test

Signed-off-by: tabokie <xy.tao@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.