Add micro-benchmark support #8493
Conversation
jay-zhuang
force-pushed
the
microbench_test2
branch
2 times, most recently
from
0f63e26
to
6351857
Compare
|
@jay-zhuang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
| @@ -370,6 +378,16 @@ jobs: | |||
| - run: CC=gcc-10 CXX=g++-10 V=1 SKIP_LINK=1 ROCKSDB_CXX_STANDARD=c++20 make -j16 all | .circleci/cat_ignore_eagain # Linking broken because libgflags compiled with newer ABI | |||
| - post-steps | |||
|
|
|||
| build-linux-microbench: | |||
There was a problem hiding this comment.
I assume you're intending CircleCI to function here as a smoke test for building & running the benchmarks, not for gathering high quality data (because running on a VM)?
There was a problem hiding this comment.
Yes, just to make sure the tests pass. added comment.
| // (found in the LICENSE.Apache file in the root directory). | ||
|
|
||
| #include <benchmark/benchmark.h> | ||
|
|
There was a problem hiding this comment.
This seems like a reasonable starting point, but filter_bench is more representative of how filters are used in RocksDB, e.g. by queries hitting different filters occupying a large memory space. We should express here that the goal is to eventually get those kinds of results into the reusable benchmarking framework. One could get the impression that this file is exclusively focused on "more micro" benchmarking of filter behavior than filter_bench does, but I do not believe that is the goal. Others could assume this file is "authoritative" and no other benchmarking for filters exists.
In broader terms, I fear that even using the term "microbenchmark" might distract from the goal (as I see it) of taking meaningful measurements of subsystem performance to catch regressions (and track improvements) by focusing too much on individual components (e.g. querying a filter) rather than treating them as subsystems (e.g. querying one of many filters).
There was a problem hiding this comment.
I agree both.
- this file is just the first benchmark that more for demonstrating the integration. Compare to
filter_bench, it's missing lots tests, but I guess if we were having this microbenchmark framework before, we might just use it instead of creating stand-alonefilter_bench. I'll add comment to refer tofilter_benchfor better filter benchmark. - For
microbenchmarkfocusing too much on small component, yes, I think it should not replace db_bench, it just gives extra, targeted information. it's also up to the test designer, so when writing future benchmark tests, we should have a more meaningful benchmark and metrics. Some metrics might be useful during development (like this one) but maybe not useful (or likely change) after feature is done, which we could exclude in the run or just ignore the result.
Summary: Add google benchmark for microbench. Add ribbon_bench for benchmark ribbon filter vs. other filters. Test Plan: added test to CI
jay-zhuang
force-pushed
the
microbench_test2
branch
from
6351857
to
50a931a
Compare
|
@jay-zhuang has updated the pull request. You must reimport the pull request before landing. |
|
@jay-zhuang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
|
@jay-zhuang merged this pull request in 5dd18a8. |
|
This change will now generate build errors/warnings on any platform/host that does not have the benchmark installed: make dbg I believe this is part of the dependency generation step. |
Summary: Add google benchmark for microbench.
Add ribbon_bench for benchmark ribbon filter vs. other filters.
Test Plan: added test to CI