Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

performance: tuning rocksdb bloom filter #4191

Merged
merged 2 commits into from
Oct 30, 2023

Conversation

quake
Copy link
Member

@quake quake commented Oct 13, 2023

What problem does this PR solve?

We previously used a global lru cache as rocksdb cache (default 128M) and setting the cache_index_and_filter_blocks parameter to true, the rocksdb's IndexBlock is placed in the cache also (note, we did not use FilterBlock in current code). With the increasing of stored data, DataBlock and IndexBlock will compete for cache expiration, there will be more cache misses, resulting in read performance degradation.

rocksdb statistics log

Block cache LRUCache@0x7f41c443d3d0#3747917 capacity: 128.00 MB seed: 886014176 usage: 117.24 MB table_size: 32768 occupancy: 5795 collections: 155 last_copies: 19 last_secs: 0.001642 secs_since: 0
Block cache entry stats(count,size,portion): DataBlock(5543,23.49 MB,18.3507%) IndexBlock(251,93.22 MB,72.8312%) Misc(1,0.00 KB,0%)

What is changed and how it works?

This PR tries to adjust the cache configuration, and also uses a bloom filter to improve the performance of prefix seek.

through the mainnet synchronization test, the overall performance has been improved by about 6%.

QQ20231013-110427@2x

and memory usage has also decreased
QQ20231013-110228@2x

Check List

Tests

  • Unit test
  • Integration test

Release note

Title Only: Include only the PR title in the release note.

@zhangsoledad
Copy link
Member

@quake quake marked this pull request as ready for review October 26, 2023 23:59
@quake quake requested a review from a team as a code owner October 26, 2023 23:59
@quake quake requested review from doitian and zhangsoledad and removed request for a team October 26, 2023 23:59
@eval-exec eval-exec added the t:performance Type: Performance tuning label Oct 27, 2023
@quake quake added this pull request to the merge queue Oct 30, 2023
@quake quake removed this pull request from the merge queue due to a manual request Oct 30, 2023
@zhangsoledad zhangsoledad added this pull request to the merge queue Oct 30, 2023
Merged via the queue into nervosnetwork:develop with commit 8215a2f Oct 30, 2023
35 checks passed
@doitian doitian mentioned this pull request Dec 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
t:performance Type: Performance tuning
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants