Collect compaction stats by priority and dump to info LOG #5050
Conversation
miasantreble
force-pushed
the
compaction-stats-by-pri
branch
2 times, most recently
from
0f52391
to
38b1c64
Compare
miasantreble
force-pushed
the
compaction-stats-by-pri
branch
from
38b1c64
to
55d9a93
Compare
There was a problem hiding this comment.
@miasantreble has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
There was a problem hiding this comment.
Thanks @miasantreble for adding this feature. I have taken a pass and left a few comments.
db/db_impl_compaction_flush.cc
Outdated
| @@ -1943,7 +1947,7 @@ void DBImpl::SchedulePendingPurge(std::string fname, std::string dir_to_sync, | |||
| void DBImpl::BGWorkFlush(void* db) { | |||
| IOSTATS_SET_THREAD_POOL_ID(Env::Priority::HIGH); | |||
| TEST_SYNC_POINT("DBImpl::BGWorkFlush"); | |||
| reinterpret_cast<DBImpl*>(db)->BackgroundCallFlush(); | |||
| reinterpret_cast<DBImpl*>(db)->BackgroundCallFlush(Env::Priority::HIGH); | |||
There was a problem hiding this comment.
Flush can also be scheduled onto a thread in the low-priority pool. See https://github.com/facebook/rocksdb/blob/master/db/db_impl_compaction_flush.cc#L1791
There was a problem hiding this comment.
You may need to add a new member variable to BGFlushArg and set it in MaybeScheduleFlushOrCompaction. Then BackgroundCallFlush can read it while popping elements from the queue.
db/db_impl_compaction_flush.cc
Outdated
| @@ -2146,7 +2151,7 @@ void DBImpl::BackgroundCallFlush() { | |||
| } | |||
|
|
|||
| void DBImpl::BackgroundCallCompaction(PrepickedCompaction* prepicked_compaction, | |||
| Env::Priority bg_thread_pri) { | |||
| Env::Priority thread_pri) { | |||
There was a problem hiding this comment.
Any reason for the renaming from bg_thread_pri to thread_pri?
|
@miasantreble has updated the pull request. Re-import the pull request |
There was a problem hiding this comment.
@miasantreble has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@miasantreble has updated the pull request. Re-import the pull request |
There was a problem hiding this comment.
@miasantreble has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
@miasantreble merged this pull request in a291f3a. |
) Summary: In order to better understand compaction done by different priority thread pool, we now collect compaction stats by priority and also print them to info LOG through stats dump. ``` ** Compaction Stats [default] ** Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Low 0/0 0.00 KB 0.0 16.8 11.3 5.5 5.6 0.1 0.0 0.0 406.4 136.1 42.24 34.96 45 0.939 13M 8865K High 0/0 0.00 KB 0.0 0.0 0.0 0.0 11.4 11.4 0.0 0.0 0.0 76.2 153.00 35.74 12185 0.013 0 0 ``` Pull Request resolved: facebook#5050 Differential Revision: D14408583 Pulled By: miasantreble fbshipit-source-id: e53746586ea27cb8abc9fec35805bd80ed30f608
In order to better understand compaction done by different priority thread pool, we now collect compaction stats by priority and also print them to info LOG through stats dump.