New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

os/bluestore: kvdb histogram #12620

Merged
merged 6 commits into from Jan 25, 2017

Conversation

Projects
None yet
5 participants
@varadakari
Contributor

varadakari commented Dec 22, 2016

Generates a histogram of key value distribution for kvdb(rocksdb for now)

@varadakari

This comment has been minimized.

Show comment
Hide comment
@varadakari

varadakari Dec 23, 2016

Contributor

@ifed01 fixed the review comments, please review again.

Contributor

varadakari commented Dec 23, 2016

@ifed01 fixed the review comments, please review again.

@varadakari

This comment has been minimized.

Show comment
Hide comment
@varadakari

varadakari Dec 23, 2016

Contributor

sample output of the histogram is below:

{
"prefix": "B",
"key-hist": {
"[0,32)": 208,
"max_len": 17,
"value-hist": {
"[0,64)": 208,
"max_len": 16
}
},
"prefix": "C",
"key-hist": {
"[0,32)": 9,
"max_len": 10,
"value-hist": {
"[0,64)": 9,
"max_len": 10
}
},
"prefix": "M",
"key-hist": {
"[0,32)": 47,
"max_len": 26,
"value-hist": {
"[0,64)": 31,
"max_len": 25,
"[128,192)": 8,
"max_len": 186,
"[832,896)": 8,
"max_len": 847
},
"[32,64)": 12807,
"max_len": 42,
"value-hist": {
"[128,192)": 12807,
"max_len": 182
}
},

Contributor

varadakari commented Dec 23, 2016

sample output of the histogram is below:

{
"prefix": "B",
"key-hist": {
"[0,32)": 208,
"max_len": 17,
"value-hist": {
"[0,64)": 208,
"max_len": 16
}
},
"prefix": "C",
"key-hist": {
"[0,32)": 9,
"max_len": 10,
"value-hist": {
"[0,64)": 9,
"max_len": 10
}
},
"prefix": "M",
"key-hist": {
"[0,32)": 47,
"max_len": 26,
"value-hist": {
"[0,64)": 31,
"max_len": 25,
"[128,192)": 8,
"max_len": 186,
"[832,896)": 8,
"max_len": 847
},
"[32,64)": 12807,
"max_len": 42,
"value-hist": {
"[128,192)": 12807,
"max_len": 182
}
},

@ifed01

@varadakari sorry for splitting my review into 2 parts - didn't have enough time for complete review yesterday.

Show outdated Hide outdated src/os/bluestore/BlueStore.cc
Show outdated Hide outdated src/os/bluestore/BlueStore.cc
Show outdated Hide outdated src/os/bluestore/BlueStore.cc
Show outdated Hide outdated src/os/bluestore/BlueStore.cc
Show outdated Hide outdated src/os/bluestore/BlueStore.cc
@varadakari

This comment has been minimized.

Show comment
Hide comment
@varadakari

varadakari Dec 26, 2016

Contributor

@ifed01 addressed the comments, please review again. Have to move the local variables(maps for the histogram) to make the updates as a function. Otherwise rest of the logic is same.

Contributor

varadakari commented Dec 26, 2016

@ifed01 addressed the comments, please review again. Have to move the local variables(maps for the histogram) to make the updates as a function. Otherwise rest of the logic is same.

@ifed01

ifed01 approved these changes Dec 26, 2016

LGTM except code style violation for 'else' statements in void BlueStore::get_db_histogram

@varadakari

This comment has been minimized.

Show comment
Hide comment
@varadakari

varadakari Dec 26, 2016

Contributor

@ifed01 Thanks for review, fixed the styling of if-else.

Contributor

varadakari commented Dec 26, 2016

@ifed01 Thanks for review, fixed the styling of if-else.

@varadakari

This comment has been minimized.

Show comment
Hide comment
@varadakari

varadakari Dec 27, 2016

Contributor

i am not hitting this compilation error in my local copy. will rebase and try it again.

Contributor

varadakari commented Dec 27, 2016

i am not hitting this compilation error in my local copy. will rebase and try it again.

@varadakari

This comment has been minimized.

Show comment
Hide comment
@varadakari

varadakari Dec 28, 2016

Contributor

Fixed all the issues and all tests are passing now.

Contributor

varadakari commented Dec 28, 2016

Fixed all the issues and all tests are passing now.

@liewegas liewegas changed the title from BlueStore kvdb histogram to os/bluestore: kvdb histogram Dec 29, 2016

@markhpc markhpc added the performance label Jan 4, 2017

@varadakari

This comment has been minimized.

Show comment
Hide comment
@varadakari

varadakari Jan 9, 2017

Contributor

jenkins retest this please

Contributor

varadakari commented Jan 9, 2017

jenkins retest this please

@varadakari

This comment has been minimized.

Show comment
Hide comment
@varadakari

varadakari Jan 11, 2017

Contributor

@liewegas could you please take look at this?

Contributor

varadakari commented Jan 11, 2017

@liewegas could you please take look at this?

Show outdated Hide outdated src/os/bluestore/BlueStore.cc
Show outdated Hide outdated src/os/bluestore/BlueStore.cc
Show outdated Hide outdated src/osd/OSD.cc

Varada Kari added some commits Dec 22, 2016

Varada Kari
os/bluefs: add interface to dump perf counters to formatter
Signed-off-by: Varada Kari <varada.kari@sandisk.com>
Varada Kari
os/kv: beautify kvdb statistics
Improves the presentation of rocksdb dump statistics

Signed-off-by: Varada Kari <varada.kari@sandisk.com>
Varada Kari
os/kv: add interface to size of key and value from wholespaceiter
Avoids converting the key and value to string or bufferlist

Signed-off-by: Varada Kari <varada.kari@sandisk.com>
@varadakari

This comment has been minimized.

Show comment
Hide comment
@varadakari

varadakari Jan 12, 2017

Contributor

@liewegas Made the changes and repushed after rebasing, could you please review again?

Contributor

varadakari commented Jan 12, 2017

@liewegas Made the changes and repushed after rebasing, could you please review again?

@varadakari

This comment has been minimized.

Show comment
Hide comment
@varadakari

varadakari Jan 12, 2017

Contributor

jenkins retest this please

Contributor

varadakari commented Jan 12, 2017

jenkins retest this please

@liewegas

Few more nits, but otherwise looks good!

Show outdated Hide outdated src/os/bluestore/BlueStore.cc
Show outdated Hide outdated src/os/bluestore/BlueStore.cc
Show outdated Hide outdated src/os/bluestore/BlueStore.cc
Show outdated Hide outdated src/os/bluestore/BlueStore.h
Show outdated Hide outdated src/os/bluestore/BlueStore.h
Varada Kari
os/bluestore: interface to dump key value distribution in kvdb
Adding a asok to command to dump the key value distribution in a
histogram fashion.

Signed-off-by: Varada Kari <varada.kari@sandisk.com>
@varadakari

This comment has been minimized.

Show comment
Hide comment
@varadakari

varadakari Jan 13, 2017

Contributor

@liewegas Addressed the comments. Please have relook at the changes.

Contributor

varadakari commented Jan 13, 2017

@liewegas Addressed the comments. Please have relook at the changes.

@liewegas liewegas added the needs-qa label Jan 13, 2017

@liewegas

This comment has been minimized.

Show comment
Hide comment
@liewegas

liewegas Jan 13, 2017

Member

Can you add a test to store_test.cc that triggers this so that the code path is tested?

Member

liewegas commented Jan 13, 2017

Can you add a test to store_test.cc that triggers this so that the code path is tested?

@liewegas liewegas removed the needs-qa label Jan 13, 2017

@liewegas

This comment has been minimized.

Show comment
Hide comment
@liewegas
Member

liewegas commented Jan 17, 2017

Varada Kari added some commits Jan 18, 2017

Varada Kari
os/bluestore: Add unittest to test kvdb histogram
Signed-off-by: Varada Kari <varada.kari@sandisk.com>
Varada Kari
os/bluestore: Add unittest to test kvdb dump stats
Signed-off-by: Varada Kari <varada.kari@sandisk.com>
@varadakari

This comment has been minimized.

Show comment
Hide comment
@varadakari

varadakari Jan 18, 2017

Contributor

@liewegas Added the unit test cases for stats and histogram. Please have a look.

Contributor

varadakari commented Jan 18, 2017

@liewegas Added the unit test cases for stats and histogram. Please have a look.

@ceph-jenkins

This comment has been minimized.

Show comment
Hide comment
@ceph-jenkins

ceph-jenkins Jan 18, 2017

Collaborator

submodules for project are unmodified

Collaborator

ceph-jenkins commented Jan 18, 2017

submodules for project are unmodified

@liewegas liewegas added the needs-qa label Jan 18, 2017

@varadakari

This comment has been minimized.

Show comment
Hide comment
@varadakari

varadakari Jan 24, 2017

Contributor

@liewegas if the tests are complete, could you merge this please?

Contributor

varadakari commented Jan 24, 2017

@liewegas if the tests are complete, could you merge this please?

@liewegas liewegas merged commit 801fd3c into ceph:master Jan 25, 2017

2 of 3 checks passed

Unmodifed Submodules checking if PR has modified submodules
Details
Signed-off-by all commits in this PR are signed
Details
default Build finished.
Details

@varadakari varadakari deleted the varadakari:wip-rocksdb-histogram branch Jan 26, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment