Issue 2289: Tune RocksDB settings

[RaulGracia]

Change log description
This PR exposes 3 parameters of RocksDB as "turning knobs" in the Pravega configuration file (writeBufferSizeMB, readCacheSizeMB, and cacheBlockSizeKB) that will take effect when instantiating RocksDB cache instances (RocksDBCache.java). We expect these parameters to be enough for a user to set an adequate RocksDB configuration in terms of memory usage/performance for a specific deployment.

Purpose of the change
Fixes #2289.

What the code does
This PR exposes in the Pravega configuration file parameters to tune RocksDB. To keep configuration complexity low, we decided to expose three main "turning knobs":

• writeBufferSizeMB: RocksDB allows to buffer writes in-memory (memtables) to improve write performance, thus executing an async flush process of writes to disk. This parameter bounds the maximum amount of memory devoted to absorb writes.
• readCacheSizeMB: RocksDB caches (uncompressed) data blocks in memory to serve read requests with high performance in case of a cache hit. This parameter bounds the maximum amount of memory devoted to cache uncompressed data blocks.
• cacheBlockSizeKB: RocksDB stores data in memory related to internal indexes (e.g., it may range between 5% to 30% of the total memory consumption depending on the configuration and data at hand). The size of the internal indexes in RocksDB mainly depend on the size of cached data blocks. If you increase this parameter, the number of blocks will decrease, so the index size will also reduce linearly (but increasing read amplification).

Note that this PR only provides the means for configuring RocksDB. While some local tests have been done, there is still a need for a thorough analysis to understand the impact of RocksDB configuration on the performance of a Pravega deployment.

How to verify it
All system test should pass as before. With the default settings, longevity runs should exhibit a similar behavior as the one described in #2760.

[RaulGracia]

Tune RocksDB configuration

[codecov]

Codecov Report

Merging #2763 into master will increase coverage by 0.08%.
The diff coverage is 100%.

@@             Coverage Diff              @@
##             master    #2763      +/-   ##
============================================
+ Coverage     78.18%   78.27%   +0.08%     
- Complexity     7112     7156      +44     
============================================
  Files           547      548       +1     
  Lines         27924    28102     +178     
  Branches       2619     2634      +15     
============================================
+ Hits          21833    21997     +164     
+ Misses         4554     4549       -5     
- Partials       1537     1556      +19

Impacted Files	Coverage Δ	Complexity Δ
...egmentstore/storage/impl/rocksdb/RocksDBCache.java	68.8% <100%> (+4.22%)	14 <1> (ø)	⬇️
...gmentstore/storage/impl/rocksdb/RocksDBConfig.java	100% <100%> (ø)	7 <3> (+3)	⬆️
...o/pravega/client/stream/impl/SegmentWithRange.java	84.61% <0%> (-7.06%)	8% <0%> (ø)
...oller/server/rpc/auth/StrongPasswordProcessor.java	87.87% <0%> (-6.07%)	9% <0%> (-1%)
...a/io/pravega/controller/store/stream/ZKStream.java	96.12% <0%> (-2.89%)	138% <0%> (+6%)
...tore/server/reading/RedirectedReadResultEntry.java	86.88% <0%> (-1.64%)	25% <0%> (-1%)
...ga/controller/task/Stream/StreamMetadataTasks.java	83.44% <0%> (-1.02%)	133% <0%> (-2%)
.../server/logs/SegmentMetadataUpdateTransaction.java	88.39% <0%> (-0.9%)	82% <0%> (-1%)
.../io/pravega/client/stream/impl/ControllerImpl.java	82.71% <0%> (-0.28%)	129% <0%> (ø)
...a/segmentstore/server/logs/OperationProcessor.java	84.92% <0%> (ø)	40% <0%> (+1%)	⬆️
... and 14 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5360823...3fabd1a. Read the comment docs.

[andreipaduroiu]

"int", not "Integer".

fix below too

[RaulGracia]

Changed in RocksDBCache.java and RocksDBConfig.java.

[andreipaduroiu]

It would be really helpful to a sys admin if each of these config settings has its own individual description. What I've done throughout this file was to simply copy the Javadoc from each of these here, and add a Recommended value to each.

This is in addition to the blob of text you added just above this.

[RaulGracia]

Thanks, I added the description of each parameter with the comment available in RocksDBConfig.java. Moreover, I think that a "recommended value" for these parameters is somewhat complex in this context as it may depend on various aspects (e.g., resources, requirements). For this reason, I will create a Wiki page in the Pravega repository (https://github.com/pravega/pravega/wiki/RocksDB-Configuration) including experiments and guidelines for users to configure RocksDB depending on their needs. Does this sound reasonable?