New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support backpressure for sharding #7100
Conversation
35a603b
to
ed1a2a3
Compare
2706 tests run: 2574 passed, 0 failed, 132 skipped (full report)Code coverage* (full report)
* collected from Rust tests only The comment gets automatically updated with the latest test results
8031bd0 at 2024-03-18T21:14:21.993Z :recycle: |
14f21fd
to
35ec187
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice change -- some comments inline. Let's figure out whether we're going to want ShardCount in PageserverFeedback for later safekeeper logic: if so, it probably makes sense to add it now while we're changing the format.
(I leave it to the postgres experts to review the C changes)
b8e68e0
to
1450c93
Compare
1450c93
to
22f0e32
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approved, but let's fix graceful shutdown before merge.
Add shard_number to PageserverFeedback and parse it on the compute side. When compute receives a new ps_feedback, it calculates min LSNs among feedbacks from all shards, and uses those LSNs for backpressure.
Add
test_sharding_backpressure
to verify that backpressure slows down compute to wait for the slowest shard.