Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

I/O statistic for key-value databases #294

Merged
merged 17 commits into from
Jan 2, 2020
Merged

I/O statistic for key-value databases #294

merged 17 commits into from
Jan 2, 2020

Conversation

NikVolf
Copy link
Contributor

@NikVolf NikVolf commented Dec 26, 2019 •
edited

This provides basic i/o stats api for key-value databases.

User can:

  1. Query overall stats. They persist during all database instance lifetime.
  2. Query intermediate stats since the last querying.

@arkpar
Copy link
Member

arkpar commented Dec 26, 2019

Would be great to have byte stats as well. Number of bytes written to the database before any amplification. I.e. combined key and value size.

@arkpar
Copy link
Member

arkpar commented Dec 26, 2019

Is there a way to get stats without clearing them? I'd expect stats to be accumulating over all of the run time.

@NikVolf
Copy link
Contributor Author

NikVolf commented Dec 26, 2019

Is there a way to get stats without clearing them? I'd expect stats to be accumulating over all of the run time.

I'll add another function for it

@NikVolf
Copy link
Contributor Author

NikVolf commented Dec 26, 2019

@arkpar addressed!

arkpar
arkpar reviewed Dec 30, 2019
View reviewed changes
kvdb-rocksdb/src/lib.rs Outdated Show resolved Hide resolved
arkpar
arkpar reviewed Dec 30, 2019
View reviewed changes
kvdb-rocksdb/src/lib.rs Outdated Show resolved Hide resolved
arkpar
arkpar reviewed Dec 30, 2019
View reviewed changes
kvdb/src/io_stats.rs Outdated Show resolved Hide resolved
dvdplm
dvdplm reviewed Dec 30, 2019
View reviewed changes
kvdb-rocksdb/src/lib.rs Outdated Show resolved Hide resolved
kvdb-rocksdb/src/lib.rs Outdated Show resolved Hide resolved
kvdb-rocksdb/src/lib.rs Outdated Show resolved Hide resolved
kvdb-rocksdb/src/lib.rs Outdated Show resolved Hide resolved
kvdb-rocksdb/src/lib.rs Outdated Show resolved Hide resolved
@NikVolf
Copy link
Contributor Author

NikVolf commented Dec 30, 2019

@dvdplm @arkpar everything addressed!

ordian
ordian reviewed Dec 31, 2019
View reviewed changes
kvdb-rocksdb/src/lib.rs Show resolved Hide resolved
kvdb-rocksdb/src/lib.rs Outdated Show resolved Hide resolved
kvdb/src/io_stats.rs Show resolved Hide resolved
@NikVolf NikVolf changed the title I/O statistic for key-value databases [wip] I/O statistic for key-value databases Jan 1, 2020
@NikVolf NikVolf changed the title [wip] I/O statistic for key-value databases I/O statistic for key-value databases Jan 1, 2020
@NikVolf
Copy link
Contributor Author

NikVolf commented Jan 1, 2020

Hey guys, I decided to refactor a bit.

The main change is now overall statistics is always kept, so if user cares only about overall stats, he will always have it, no matter if any intermediate queries were performed.

Pls re-review @arkpar @dvdplm @ordian

ordian
ordian approved these changes Jan 1, 2020
View reviewed changes
Copy link
Member

@ordian ordian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

ordian
ordian reviewed Jan 1, 2020
View reviewed changes
kvdb/src/lib.rs Outdated Show resolved Hide resolved
kvdb-rocksdb/src/stats.rs Outdated Show resolved Hide resolved
kvdb-rocksdb/src/stats.rs Outdated Show resolved Hide resolved
@arkpar arkpar merged commit fe98963 into master Jan 2, 2020
@arkpar arkpar deleted the nv-io-stats branch January 2, 2020 11:10
dvdplm
dvdplm reviewed Jan 2, 2020
View reviewed changes
kvdb-rocksdb/src/lib.rs
.map(|r| r.map(|v| v.to_vec()))
.map_err(other_io_err),
None => {
let aquired_val = cfs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo here: "acquire"

dvdplm added a commit that referenced this pull request Jan 14, 2020
@dvdplm
Merge branch 'master' into dp/chore/secp256k1-from-upstream
48a2ef6 
* master: (27 commits)
  update parity-util-mem (#309)
  Update features and feature dependencies (#307)
  Use proper memory queries to rocksdb (#308)
  Draft version updates and changelog (#299)
  Use custom error type for `from_hex` (#305)
  Fix typo. (#303)
  kvdb: remove KeyValueDBHandler (#304)
  extract common kvdb tests into a crate (#301)
  Expose to_hex and from_hex from impl-serde (#302)
  Add a top level function to parity-util-mem (#298)
  I/O statistic for key-value databases (#294)
  use for_tuples (#300)
  Add memory extensions for LRUCache, hashbrown (#293)
  Add memory stats for kvdb-s (#292)
  Provide #[derive(MallocSizeOf)] that is actually working (#291)
  Move and extend impls for locking primitives (#290)
  Update uint README (#288)
  Update README.md (#287)
  [ci]: remove feature flags in virtual workspace (#289)
  Prepare releases (#286)
  ...
@ordian ordian mentioned this pull request Feb 3, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dvdplm dvdplm dvdplm left review comments

@ordian ordian ordian approved these changes

@arkpar arkpar arkpar approved these changes

Assignees
No one assigned
Labels
A8-looksgood
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@NikVolf @arkpar @dvdplm @ordian