RATIS-1901. Update Counter example to benchmark performance.#953
RATIS-1901. Update Counter example to benchmark performance.#953szetszwo merged 1 commit intoapache:masterfrom
Conversation
SzyWilliam
left a comment
SzyWilliam
left a comment
There was a problem hiding this comment.
+1 the changes look good to me. Thanks @szetszwo for the benchmark!
I think we need to notice users the potential problems if we adopt a larger segment size.
|
|
||
| String SEGMENT_SIZE_MAX_KEY = PREFIX + ".segment.size.max"; | ||
| SizeInBytes SEGMENT_SIZE_MAX_DEFAULT = SizeInBytes.valueOf("8MB"); | ||
| SizeInBytes SEGMENT_SIZE_MAX_DEFAULT = SizeInBytes.valueOf("128MB"); |
There was a problem hiding this comment.
+1 for a larger segment size. However, a larger segment size may cause heavier memory pressure, since we need to retain the open segment in memory. Let me update the configuration docs to include this notice after the patch being merged.
There was a problem hiding this comment.
How about changing it to a small size, say 32MB? Currently, 8MB is very small. The servers will roll the log frequently and create a lot of small files. Let me revert the change here and do it separately.
There was a problem hiding this comment.
+1 8MB is small. Actually I think 128MB is good. Kafka even uses 1GB as the segment size. It's only that I did experiments earlier and found 32MB is the optimal for IOTDB default workload. The experiments show:
- (end to end throughput) improves substantially as segmented size grow from 8MB to 32MB, slowly from 32MB to 128MB.
- The memory usage in JVM young increases as the segmented size grow, even if we restricted the max-cached-size and max-cached-segments. It turned out to be the open segment.
There was a problem hiding this comment.
It is good to know. Let's change to 32MB then. Filed RATIS-1924.
|
Hit the NPE fixed by RATIS-1920 in my test. Will sync with the master branch. |
|
@SzyWilliam , thanks a lot for reviewing this! |
2 participants
See RATIS-1901