Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Epic: compute support for sharding phase 1 #5508

Closed
3 tasks
jcsp opened this issue Oct 9, 2023 · 3 comments
Closed
3 tasks

Epic: compute support for sharding phase 1 #5508

jcsp opened this issue Oct 9, 2023 · 3 comments
Assignees
@knizhnik @save-buffer
Labels
c/compute Component: compute, excluding postgres itself t/Epic Issue type: Epic t/feature Issue type: feature, for new features or requests

Comments

@jcsp
Copy link
Contributor

jcsp commented Oct 9, 2023 •
edited
Loading

See #5507 for the related pageserver work.

RFC in progress here: #5432

Tasks

  • Update configuration to accept a list of pageserver connection strings, and metadata on stripe size & shard count, i.e. the ShardMap described in the RFC.
  • For getpage requests, calculate the key and calculate the corresponding shard, then dispatch the RPC to the pageserver holding that shard, including error handling (e.g. if the pageserver denies ownership of the shard, backoff-retry in expectation of the control plane supplying updated configuration)
  • If necessary, adjust prefetch code to clamp prefetched ranges to the boundaries of a stripe, to avoid issuing requests that cover pages belonging to multiple pageservers.
@jcsp jcsp added t/feature Issue type: feature, for new features or requests c/cloud/compute t/Epic Issue type: Epic labels Oct 9, 2023
@knizhnik knizhnik mentioned this issue Nov 9, 2023
Closed
5 tasks
@vadim2404 vadim2404 added c/compute Component: compute, excluding postgres itself and removed c/cloud/compute labels Dec 12, 2023
@vadim2404
Copy link
Contributor

vadim2404 commented Dec 19, 2023 •
edited
Loading

PR: #5837
PR: #6205

knizhnik pushed a commit that referenced this issue Dec 20, 2023
Support sharding at compute side
f43f152 
refer #5508
@knizhnik knizhnik mentioned this issue Dec 20, 2023
Merged
5 tasks
jcsp pushed a commit that referenced this issue Dec 22, 2023
@jcsp
Support sharding at compute side
218a8a7 
refer #5508
knizhnik pushed a commit that referenced this issue Dec 27, 2023
Support sharding at compute side
c64ab9d 
refer #5508
jcsp pushed a commit that referenced this issue Jan 2, 2024
@jcsp
Support sharding at compute side
860b5c9 
refer #5508
jcsp pushed a commit that referenced this issue Jan 2, 2024
@jcsp
Support sharding at compute side
b06b042 
refer #5508
@vadim2404
Copy link
Contributor

@hlinnaka , waiting for your final review

jcsp pushed a commit that referenced this issue Jan 3, 2024
@jcsp
Support sharding at compute side
48e373c 
refer #5508
jcsp pushed a commit that referenced this issue Jan 7, 2024
@jcsp
Support sharding at compute side
8bd0fc4 
refer #5508
knizhnik pushed a commit that referenced this issue Jan 10, 2024
Support sharding at compute side
185f0e8 
refer #5508
jcsp pushed a commit that referenced this issue Jan 11, 2024
@jcsp
Support sharding at compute side
71170d8 
refer #5508
jcsp pushed a commit that referenced this issue Jan 16, 2024
@jcsp
Support sharding at compute side
3cf84ee 
refer #5508
jcsp pushed a commit that referenced this issue Jan 17, 2024
@jcsp
Support sharding at compute side
d476373 
refer #5508
jcsp pushed a commit that referenced this issue Jan 17, 2024
@jcsp
Support sharding at compute side
a2a1bb0 
refer #5508
jcsp pushed a commit that referenced this issue Jan 17, 2024
@jcsp
Support sharding at compute side
7449d97 
refer #5508
jcsp pushed a commit that referenced this issue Jan 18, 2024
@jcsp
Support sharding at compute side
0390978 
refer #5508
@vadim2404
Copy link
Contributor

to merge it this week

knizhnik added a commit that referenced this issue Jan 25, 2024
@knizhnik @jcsp @hlinnaka
Add support for PS sharding in compute (#6205)
19ed230 
refer #5508

replaces #5837

## Problem

This PR implements sharding support at compute side. Relations are
splinted in stripes and `get_page` requests are redirected to the
particular shard where stripe is located. All other requests (i.e. get
relation or database size) are always send to shard 0.

## Summary of changes

Support of sharding at compute side include three things:
1. Make it possible to specify and change in runtime connection to more
retain one page server
2. Send `get_page` request to the particular shard (determined by hash
of page key)
3. Support multiple servers in prefetch ring requests

## Checklist before requesting a review

- [ ] I have performed a self-review of my code.
- [ ] If it is a core feature, I have added thorough tests.
- [ ] Do we need to implement analytics? if so did you add the relevant
metrics to the dashboard?
- [ ] If this PR requires public announcement, mark it with
/release-notes label and add several sentences in this section.

## Checklist before merging

- [ ] Do not forget to reformat commit message to not include the above
checklist

---------

Co-authored-by: Konstantin Knizhnik <knizhnik@neon.tech>
Co-authored-by: John Spray <john@neon.tech>
Co-authored-by: Heikki Linnakangas <heikki@neon.tech>
@jcsp jcsp closed this as completed Jun 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@knizhnik knizhnik

@save-buffer save-buffer

Labels
c/compute Component: compute, excluding postgres itself t/Epic Issue type: Epic t/feature Issue type: feature, for new features or requests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@knizhnik @jcsp @vadim2404 @save-buffer