-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pilorama: merge in-queue batches #2231
Conversation
Codecov Report
@@ Coverage Diff @@
## support/v0.35 #2231 +/- ##
==============================================
Coverage 30.87% 30.88%
==============================================
Files 383 383
Lines 28389 28393 +4
==============================================
+ Hits 8766 8770 +4
Misses 18876 18876
Partials 747 747
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Commit msg not formated; deserves a CHANGELOG record.
To achieve high performance we must choose proper values for both batch size and delay. For user operations we want to set low delay. However it would prevent tree synchronization operations to form big enough batches. For these operations, batching gives the most benefit not only in terms of on-CPU execution cost, but also by speeding up transaction persist (`fsync`). In this commit we try merging batches that are already _triggered_, but not yet _started to execute_. This way we can still query batches for execution after the provided delay while also allowing multiple formed batches to execute faster. Signed-off-by: Evgenii Stratonikov <e.stratonikov@yadro.com>
What are the numbers in ops/s? |
I don't have exact numbers saved (but @anatoly-bogatyrev can help here). |
Wrong PR number in commit msg 😬. |
To achieve high performance we must choose proper values for both batch size and delay. For user operations we want to set low delay. However it would prevent tree synchronization operations to form big enough batches. For these operations, batching gives the most benefit not only in terms of on-CPU execution cost, but also by speeding up transaction persist (
fsync
).In this commit we try merging batches that are already triggered, but not yet started to execute. This way we can still query batches for execution after the provided delay while also allowing multiple formed batches to execute faster.
Hardware performance tests show ~10x improvement for small objects put in one bucket.
Signed-off-by: Evgenii Stratonikov e.stratonikov@yadro.com