change rocksdb init: use OptionsUtil #3056

StevenLuMT · 2022-02-16T13:32:33Z

Descriptions of the changes in this PR:

Motivation

some old parameters in rocksDB is not configurable
for all the tuning of rocksdb in the future, there is no need to update the code or introduce configuration to bookie

Changes

rocks all old parameter change to be configurable
use OptionsUtil to init all params for rocksdb

the old pr #3006 has some rebase error,open a new pr

StevenLuMT · 2022-03-05T13:21:09Z

@dlg99 @eolivelli @pkumar-singh @zymap @hangc0276 @lordcheng10
If you have time, please help me review it, thank you.

lordcheng10

LGTM

dlg99

LGTM

hangc0276

Good job.

…ns (#3860) ### Motivation After PR #3056 , Bookkeeper set `level_compaction_dynamic_level_bytes=true` as `TableOptions` in `entry_location_rocksdb.conf.default` , which will cause `level_compaction_dynamic_level_bytes` lose efficacy and will cause rocksDB .sst file compact sort chaos when update bookie release. As RocksDB conf, `level_compaction_dynamic_level_bytes` need set as `CFOptions` https://github.com/facebook/rocksdb/blob/master/examples/rocksdb_option_file_example.ini <img width="703" alt="image" src="https://user-images.githubusercontent.com/84127069/224640399-d5481fe5-7b75-4229-ac06-3d280aa9ae6d.png"> <img width="240" alt="image" src="https://user-images.githubusercontent.com/84127069/224640621-737d0a42-4e01-4f38-bd5a-862a93bc4b32.png"> ### Changes 1. Change `level_compaction_dynamic_level_bytes=true` from `TableOptions` to `CFOptions` in `entry_location_rocksdb.conf.default` ;

* Fix memory leak issue of reading small entries (apache#3844) * Make read entry request recyclable (apache#3842) * Make read entry request recyclable * Move recycle to finally block * Fix test and comments * Fix test * Avoid unnecessary force write. (apache#3847) * Avoid unnecessary force write. * code clean. * fix style * Correct the running job name for the test group (apache#3851) --- ### Motivation The running tests job name doesn't match the tests. Correct the job name. * add timeout for two flaky timeout tests (apache#3855) * add V2 protocal and warmupMessages support for benchMark (apache#3856) * disable trimStackTrack for code-coverage profile (apache#3854) * Fix bkperf log directory not found (apache#3858) ### Motivation When using the bkperf command `bin/bkperf journal append -j data -n 100000000 --sync true` to test the BookKeeper journal performance, it failed with the following exception ``` [0.002s][error][logging] Error opening log file '/Users/hangc/Downloads/tmp/tc/batch/ta/bookkeeper-all-4.16.0-SNAPSHOT/logs/bkperf-gc.log': No such file or directory [0.002s][error][logging] Initialization of output 'file=/Users/hangc/Downloads/tmp/tc/batch/ta/bookkeeper-all-4.16.0-SNAPSHOT/logs/bkperf-gc.log' using options 'filecount=5,filesize=64m' failed. Invalid -Xlog option '-Xlog:gc=info:file=/Users/hangc/Downloads/tmp/tc/batch/ta/bookkeeper-all-4.16.0-SNAPSHOT/logs/bkperf-gc.log::filecount=5,filesize=64m', see error log for details. Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. ``` The root cause is that the `logs` directory was not created. ### Modifications Create the `logs` directory before bkperf started. * [improve] Fix indexDirs upgrade failed (apache#3762) * fix indexDirs upgrade failed * Bump checkstyle-plugin from 3.1.2 to 3.2.1 (apache#3850) * [Flaky] Fix flaky test in testRaceGuavaEvictAndReleaseBeforeRetain (apache#3857) * Fix flaky test in testRaceGuavaEvictAndReleaseBeforeRetain * format code * Fix NPE in BenchThroughputLatency (apache#3859) * Update website to 4.15.4 (apache#3862) --- ### Motivation Update website to 4.15.4 * change rocksDB config level_compaction_dynamic_level_bytes to CFOptions (apache#3860) ### Motivation After PR apache#3056 , Bookkeeper set `level_compaction_dynamic_level_bytes=true` as `TableOptions` in `entry_location_rocksdb.conf.default` , which will cause `level_compaction_dynamic_level_bytes` lose efficacy and will cause rocksDB .sst file compact sort chaos when update bookie release. As RocksDB conf, `level_compaction_dynamic_level_bytes` need set as `CFOptions` https://github.com/facebook/rocksdb/blob/master/examples/rocksdb_option_file_example.ini <img width="703" alt="image" src="https://user-images.githubusercontent.com/84127069/224640399-d5481fe5-7b75-4229-ac06-3d280aa9ae6d.png"> <img width="240" alt="image" src="https://user-images.githubusercontent.com/84127069/224640621-737d0a42-4e01-4f38-bd5a-862a93bc4b32.png"> ### Changes 1. Change `level_compaction_dynamic_level_bytes=true` from `TableOptions` to `CFOptions` in `entry_location_rocksdb.conf.default` ; * Correct the running job flag for the test group. (apache#3865) * Release note for 4.15.4 (apache#3831) --- ### Motivation Release note for 4.15.4 * Add trigger entry location index rocksDB compact interface. (apache#3802) ### Motivation After the bookie instance running long time, the bookie entry location index rocksDB `.sst` file size maybe expand to 20-30GB as one ledger data dir's location index in some case, which will cause the rocksDB scan operator cost more time and cause the bookie client request timeout. Add trigger entry location index rocksDB compact REST API which can trigger entry location rocksDB compaction and get the compaction status. The full range entry location index rocksDB compact will cause the entry location index dir express higher IOUtils. So we'd better trigger the entry location rocksDB compact by the api in low data flow period. **Some case before rocksDB compact:** <img width="232" alt="image" src="https://user-images.githubusercontent.com/84127069/220893469-e6fbc1a3-c767-4ffe-8ae9-f05ad1833c50.png"> <img width="288" alt="image" src="https://user-images.githubusercontent.com/84127069/220891359-dc37e139-37b0-461b-8001-dcc48517366c.png"> **After rocksDB compact:** <img width="255" alt="image" src="https://user-images.githubusercontent.com/84127069/220891419-24267fa7-348c-4fbd-8b3e-70a99840bce5.png"> ### Changes 1. Add REST API to trigger entry location index rocksDB compact. * Pick the higher leak detection level between netty and bookkeeper. (apache#3794) ### Motivation 1. Pick the higher leak detection level between netty and bookkeeper. 2. Enhance the bookkeeper leak detection value match rule, now it's case insensitive. There are detailed information about it: https://lists.apache.org/thread/d3zw8bxhlg0wxfhocyjglq0nbxrww3sg * Disable code coverage and codecov report (apache#3863) ### Motivation There're two reasons that we want to disable the code coverage. 1. The current report result is not accurate. 2. We can't get the PR's unit test's code coverage because of the apache Codecov permission. * Add small files check in garbage collection (apache#3631) ### Motivation When we use `TransactionalEntryLogCompactor` to compact the entry log files, it will generate a lot of small entry log files, and for those files, the file usage is usually greater than 90%, which can not be compacted unless the file usage decreased. ![image](https://user-images.githubusercontent.com/5436568/201135615-4d6072f5-e353-483d-9afb-48fad8134044.png) ### Changes We introduce the entry log file size check during compaction, and the checker is controlled by `gcEntryLogSizeRatio`. If the total entry log file size is less than `gcEntryLogSizeRatio * logSizeLimit`, the entry log file will be compacted even though the file usage is greater than 90%. This feature is disabled by default and the `gcEntryLogSizeRatio` default value is `0.0` * [improvement] Delay all audit task when have a already delayed bookie check task (apache#3818) ### Motivation Fixes apache#3817 For details, see: apache#3817 ### Changes When there is an `auditTask` during the `lostBookieRecoveryDelay` delay, other detection tasks should be skipped. * Change order of doGcLedgers and extractMetaFromEntryLogs (apache#3869) * [Bugfix] make metadataDriver initialization more robust (apache#3873) Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com> * Enable CI for the streamstorage python client (apache#3875) * Fix compaction threshold default value precision problem. (apache#3871) * Fix compaction threshold precision problem. * Fix compaction threshold precision problem. * Single buffer for small add requests (apache#3783) * Single buffer for small add requests * Fixed checkstyle * Fixed treating of ComposityByteBuf * Fixed merge issues * Fixed merge issues * WIP * Fixed test and removed dead code * Removed unused import * Fixed BookieJournalTest * removed unused import * fix the checkstyle * fix failed test * fix failed test --------- Co-authored-by: chenhang <chenhang@apache.org> * Add log for entry log file delete. (apache#3872) * Add log for entry log file delete. * add log info. * Address the comment. * Address the comment. * revert the code. * Improve group and flush add-responses after journal sync (apache#3848) Descriptions of the changes in this PR: This is an improvement for apache#3837 ### Motivation 1. Now if the maxPendingResponsesSize is expanded large, it will not decrease. => We should make it flexible. 2. Now after prepareSendResponseV2 to the channel, then we trigger all channels to flush pendingSendResponses, maybe there is only a few channels that need to flush, but if we trigger all channels, it's a waste. => We only flush the channel which prepareSendResponseV2. --------- Co-authored-by: Penghui Li <penghui@apache.org> Co-authored-by: Yong Zhang <zhangyong1025.zy@gmail.com> Co-authored-by: Hang Chen <chenhang@apache.org> Co-authored-by: wenbingshen <oliver.shen999@gmail.com> Co-authored-by: ZhangJian He <shoothzj@gmail.com> Co-authored-by: lixinyang <84127069+Nicklee007@users.noreply.github.com> Co-authored-by: YANGLiiN <ielin@qq.com> Co-authored-by: Lishen Yao <yaalsn@gmail.com> Co-authored-by: Andrey Yegorov <8622884+dlg99@users.noreply.github.com> Co-authored-by: ZanderXu <zanderxu@apache.org> Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com> Co-authored-by: Matteo Merli <mmerli@apache.org>

…ns (#3860) ### Motivation After PR #3056 , Bookkeeper set `level_compaction_dynamic_level_bytes=true` as `TableOptions` in `entry_location_rocksdb.conf.default` , which will cause `level_compaction_dynamic_level_bytes` lose efficacy and will cause rocksDB .sst file compact sort chaos when update bookie release. As RocksDB conf, `level_compaction_dynamic_level_bytes` need set as `CFOptions` https://github.com/facebook/rocksdb/blob/master/examples/rocksdb_option_file_example.ini <img width="703" alt="image" src="https://user-images.githubusercontent.com/84127069/224640399-d5481fe5-7b75-4229-ac06-3d280aa9ae6d.png"> <img width="240" alt="image" src="https://user-images.githubusercontent.com/84127069/224640621-737d0a42-4e01-4f38-bd5a-862a93bc4b32.png"> ### Changes 1. Change `level_compaction_dynamic_level_bytes=true` from `TableOptions` to `CFOptions` in `entry_location_rocksdb.conf.default` ; (cherry picked from commit 5542afe)

…ns (apache#3860) After PR apache#3056 , Bookkeeper set `level_compaction_dynamic_level_bytes=true` as `TableOptions` in `entry_location_rocksdb.conf.default` , which will cause `level_compaction_dynamic_level_bytes` lose efficacy and will cause rocksDB .sst file compact sort chaos when update bookie release. As RocksDB conf, `level_compaction_dynamic_level_bytes` need set as `CFOptions` https://github.com/facebook/rocksdb/blob/master/examples/rocksdb_option_file_example.ini <img width="703" alt="image" src="https://user-images.githubusercontent.com/84127069/224640399-d5481fe5-7b75-4229-ac06-3d280aa9ae6d.png"> <img width="240" alt="image" src="https://user-images.githubusercontent.com/84127069/224640621-737d0a42-4e01-4f38-bd5a-862a93bc4b32.png"> 1. Change `level_compaction_dynamic_level_bytes=true` from `TableOptions` to `CFOptions` in `entry_location_rocksdb.conf.default` ; (cherry picked from commit 5542afe)

Descriptions of the changes in this PR: 1. some old parameters in rocksDB is not configurable 2. for all the tuning of rocksdb in the future, there is no need to update the code or introduce configuration to bookie 1) rocks all old parameter change to be configurable 2) use OptionsUtil to init all params for rocksdb the old pr apache#3006 has some rebase error,open a new pr Reviewers: Andrey Yegorov <None>, LinChen <None> This closes apache#3056 from StevenLuMT/master_improveRocksDB (cherry picked from commit 3edbc98)

Descriptions of the changes in this PR: ### Motivation 1. some old parameters in rocksDB is not configurable 2. for all the tuning of rocksdb in the future, there is no need to update the code or introduce configuration to bookie ### Changes 1) rocks all old parameter change to be configurable 2) use OptionsUtil to init all params for rocksdb the old pr apache#3006 has some rebase error,open a new pr Reviewers: Andrey Yegorov <None>, LinChen <None> This closes apache#3056 from StevenLuMT/master_improveRocksDB

…ns (apache#3860) ### Motivation After PR apache#3056 , Bookkeeper set `level_compaction_dynamic_level_bytes=true` as `TableOptions` in `entry_location_rocksdb.conf.default` , which will cause `level_compaction_dynamic_level_bytes` lose efficacy and will cause rocksDB .sst file compact sort chaos when update bookie release. As RocksDB conf, `level_compaction_dynamic_level_bytes` need set as `CFOptions` https://github.com/facebook/rocksdb/blob/master/examples/rocksdb_option_file_example.ini <img width="703" alt="image" src="https://user-images.githubusercontent.com/84127069/224640399-d5481fe5-7b75-4229-ac06-3d280aa9ae6d.png"> <img width="240" alt="image" src="https://user-images.githubusercontent.com/84127069/224640621-737d0a42-4e01-4f38-bd5a-862a93bc4b32.png"> ### Changes 1. Change `level_compaction_dynamic_level_bytes=true` from `TableOptions` to `CFOptions` in `entry_location_rocksdb.conf.default` ;

lhotari · 2024-08-15T05:51:09Z

bookkeeper-server/src/main/java/org/apache/bookkeeper/conf/ServerConfiguration.java

+        URL defURL = getClass().getClassLoader().getResource(defaultPath);
+        if (defURL != null) {
+            defaultPath = defURL.getPath();
+        }


This is very surprising behavior that suddenly the classloader is used to locate the resources.
At least there should be a check that the URL is pointing to a file which isn't embedded in another Jar file.

lushiji added 6 commits February 16, 2022 21:24

change rocksdb init: use OptionsUtil

ce9e253

update rocksdb config dir

aa13209

change rocksdb config load

2a20432

add test config for test case

4dc7c06

add License for rocksdb config

7cad5aa

update ledgerMetadataRocksdbConf default value

22ddab4

zymap requested review from Vanlightly, eolivelli and merlimat February 22, 2022 03:37

zymap assigned StevenLuMT Feb 22, 2022

lordcheng10 approved these changes Mar 9, 2022

View reviewed changes

dlg99 approved these changes Mar 9, 2022

View reviewed changes

dlg99 added this to the 4.15.0 milestone Mar 10, 2022

dlg99 added area/bookie release/4.15.0 type/improvement labels Mar 10, 2022

dlg99 merged commit 3edbc98 into apache:master Mar 10, 2022

hangc0276 reviewed Mar 13, 2022

View reviewed changes

zymap mentioned this pull request Oct 12, 2022

Make the rocksDB configuration compatible with previous versions #3523

Merged

Nicklee007 mentioned this pull request Mar 13, 2023

Modify incorrect rocksDB config level_compaction_dynamic_level_bytes to CFOptions #3860

Merged

lhotari mentioned this pull request Aug 15, 2024

Upgrade RocksDB to version 9.2.1 #4422

Open

lhotari mentioned this pull request Aug 15, 2024

[improve][misc] Set format_version=5, checksum=kxxHash in Bookkeeper RocksDB configs apache/pulsar#23175

Merged

4 tasks

lhotari reviewed Aug 15, 2024

View reviewed changes

lhotari mentioned this pull request Aug 15, 2024

Set default format_version to 5 for RocksDB databases #4480

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

change rocksdb init: use OptionsUtil #3056

change rocksdb init: use OptionsUtil #3056

StevenLuMT commented Feb 16, 2022 •

edited

Loading

StevenLuMT commented Mar 5, 2022

lordcheng10 left a comment

dlg99 left a comment

hangc0276 left a comment

lhotari Aug 15, 2024 •

edited

Loading

change rocksdb init: use OptionsUtil #3056

change rocksdb init: use OptionsUtil #3056

Conversation

StevenLuMT commented Feb 16, 2022 • edited Loading

Motivation

Changes

StevenLuMT commented Mar 5, 2022

lordcheng10 left a comment

Choose a reason for hiding this comment

dlg99 left a comment

Choose a reason for hiding this comment

hangc0276 left a comment

Choose a reason for hiding this comment

lhotari Aug 15, 2024 • edited Loading

Choose a reason for hiding this comment

StevenLuMT commented Feb 16, 2022 •

edited

Loading

lhotari Aug 15, 2024 •

edited

Loading