Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perf/sorted batch #6142

Merged
merged 5 commits into from
Sep 29, 2023
Merged

Perf/sorted batch #6142

merged 5 commits into from
Sep 29, 2023

Conversation

asdacap
Copy link
Contributor

@asdacap asdacap commented Sep 28, 2023

  • The write batch keys are sorted before putting into rocksdb. This improve throughput as rocksdb's skiplist has a fast path when keys are sorted in order.
  • However, with path based layout, the size of the key doubled and the number of key doubled. This causes the overhead of sorting to be high enough that disabling it improve throughput.
  • Additionally, with path based layout the keys are already partially ordered when inserted, making this sort not very useful.
  • Disabling this optimization improve sync throughput for path based layout by about 20% to 25%.
  • Perform exactly the same on hash based layout.

Changes

  • Extract the sort logic to its own class.
  • Enable it for TrieStore and receipt tx index.

Types of changes

What types of changes does your code introduce?

  • Optimization

Testing

Requires testing

  • Yes
  • No

If yes, did you write tests?

  • Yes
  • No

Copy link
Member

@LukaszRozmej LukaszRozmej left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asdacap
Copy link
Contributor Author

asdacap commented Sep 29, 2023

Well, it does add the same functionality, but we're just interested in the sorting which increase throughput from about 0.9 to 1.25 mil keys per second. Already check if using the write batch with index automatically sort it, nope does not work.

@asdacap asdacap merged commit b3bc144 into master Sep 29, 2023
61 checks passed
@asdacap asdacap deleted the perf/sorted-batch branch September 29, 2023 10:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants