Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

roachpb: add batching to the GetRangeDescriptors API #92785

Open
arulajmani opened this issue Dec 1, 2022 · 0 comments
Open

roachpb: add batching to the GetRangeDescriptors API #92785

arulajmani opened this issue Dec 1, 2022 · 0 comments
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)

Comments

@arulajmani
Copy link
Collaborator

arulajmani commented Dec 1, 2022

Is your feature request related to a problem? Please describe.

We introduced the GetRangeDescriptors API in #92603 as an alternative to scanning meta ranges to fetch RangeDescriptors. The intention here was to allow secondary tenants to fetch range descriptors without directly interacting with the meta ranges.

We set up this API to be able to stream range descriptors back to the client. However, we don't meaningfully make use of this just yet. Instead, we bulk load the requested range descriptors in memory and send them as a single response. This is likely okay for the initial implementation, given we don't expect RangeDescriptors to be particularly large (assuming reasonable keys, ofcourse).

We could explore adding some sort of batching here in the future. Concretely, we'd need to ship a timestamp from the client and use it as the timestamp to perform a paginated scan of meta2 at. We could then stream results back to the client as we paginate through the descriptors. One thing to note is the scan happens in a retryable, and as such, can be restarted multiple times -- we'd need to communicate this back to the client so that it can throw away the accumulated range descriptors so far.

cc @ajwerner following our conversation about this earlier today. @ecwall you might find this interesting as well.

Jira issue: CRDB-21962
Epic: CRDB-26091

@arulajmani arulajmani added the C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) label Dec 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
Projects
None yet
Development

No branches or pull requests

1 participant