Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support backpressure for sharding #7100

Merged
merged 9 commits into from Mar 18, 2024
Merged

Support backpressure for sharding #7100

merged 9 commits into from Mar 18, 2024

Conversation

petuhovskiy
Copy link
Member

@petuhovskiy petuhovskiy commented Mar 12, 2024

Add shard_number to PageserverFeedback and parse it on the compute side. When compute receives a new ps_feedback, it calculates min LSNs among feedbacks from all shards, and uses those LSNs for backpressure.

Add test_sharding_backpressure to verify that backpressure slows down compute to wait for the slowest shard.

Copy link

github-actions bot commented Mar 12, 2024

2706 tests run: 2574 passed, 0 failed, 132 skipped (full report)


Code coverage* (full report)

  • functions: 28.3% (7127 of 25167 functions)
  • lines: 46.8% (43716 of 93313 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
8031bd0 at 2024-03-18T21:14:21.993Z :recycle:

@petuhovskiy petuhovskiy force-pushed the sharding-backpressure branch 2 times, most recently from 14f21fd to 35ec187 Compare March 12, 2024 23:43
@petuhovskiy petuhovskiy marked this pull request as ready for review March 13, 2024 21:28
@petuhovskiy petuhovskiy requested review from a team as code owners March 13, 2024 21:28
Copy link
Contributor

@jcsp jcsp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice change -- some comments inline. Let's figure out whether we're going to want ShardCount in PageserverFeedback for later safekeeper logic: if so, it probably makes sense to add it now while we're changing the format.

(I leave it to the postgres experts to review the C changes)

libs/utils/src/pageserver_feedback.rs Show resolved Hide resolved
pgxn/neon/walproposer.h Show resolved Hide resolved
libs/utils/src/pageserver_feedback.rs Show resolved Hide resolved
pgxn/neon/walproposer_pg.c Show resolved Hide resolved
Copy link
Contributor

@arssher arssher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, but let's fix graceful shutdown before merge.

pgxn/neon/walproposer.c Outdated Show resolved Hide resolved
pgxn/neon/walproposer.c Outdated Show resolved Hide resolved
pgxn/neon/walproposer.c Outdated Show resolved Hide resolved
pgxn/neon/walproposer.c Outdated Show resolved Hide resolved
pgxn/neon/walproposer.h Show resolved Hide resolved
pgxn/neon/walproposer_pg.c Show resolved Hide resolved
test_runner/regress/test_sharding.py Show resolved Hide resolved
@petuhovskiy petuhovskiy enabled auto-merge (squash) March 18, 2024 20:31
@petuhovskiy petuhovskiy merged commit ad5efb4 into main Mar 18, 2024
53 checks passed
@petuhovskiy petuhovskiy deleted the sharding-backpressure branch March 18, 2024 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants