Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: implement StateRead::nonverifiable_range #2885

Merged
merged 15 commits into from
Aug 21, 2023
Merged

Conversation

erwanor
Copy link
Member

@erwanor erwanor commented Aug 1, 2023

This PR adds support for range queries over the nonverifiable storage. It does so by:

  • extending the StateRead trait with a new method:
    /// Retrieve all values for keys in a range from the non-verifiable key-value store, as raw bytes.
    /// This method does not support inclusive ranges, and will return an error if passed one.
    ///
    /// Users should generally prefer to use wrapper methods in an extension trait.
    fn nonverifiable_range_raw(
        &self,
        prefix: Option<&[u8]>,
        range: impl RangeBounds<Vec<u8>>,
    ) -> Result<Self::NonconsensusRangeRawStream>;
  • use a rocksdb range iterator to create an underlying stream of keys and values
  • implements a StateDelta-level stream that interleaves caching layers and underlying storage.

Example API:

The first intended use of range queries is to fetch a range of compact_blocks from an arbitrary height:

let start_height = compact_block::height(20); // "000000000000000020"
// Return a stream of compact blocks starting at height 20 to the latest one.
let mut compact_block_stream = state.nonverifiable_range_query(Some(compact_block::prefix()), start_height..);

Alternative APIs considered:

        fn nonverifiable_range_raw<T, U>(
        &self,
        prefix: Option<&[u8]>,
        range: T,
    ) -> anyhow::Result<Self::NonconsensusRangeRawStream>
    where
        T: std::ops::RangeBounds<U>,
        U: Into<Vec<u8>>;

and

    fn nonverifiable_range_raw<T, U>(
        &self,
        prefix: Option<&[u8]>,
        range: T,
    ) -> anyhow::Result<Self::NonconsensusRangeRawStream>
    where
        T: std::ops::RangeBounds<U>,
        U: AsRef<[u8]>;

either are fine choices, but my preference goes to the simpler signature.

@erwanor erwanor temporarily deployed to smoke-test August 1, 2023 01:33 — with GitHub Actions Inactive
@erwanor erwanor self-assigned this Aug 1, 2023
@erwanor erwanor force-pushed the range_queries_stateread branch from 39036c1 to b67b934 Compare August 4, 2023 01:04
@erwanor erwanor temporarily deployed to smoke-test August 4, 2023 01:04 — with GitHub Actions Inactive
@erwanor erwanor force-pushed the range_queries_stateread branch from b67b934 to 70c3e2c Compare August 7, 2023 13:32
@erwanor erwanor temporarily deployed to smoke-test August 7, 2023 13:32 — with GitHub Actions Inactive
@erwanor erwanor force-pushed the range_queries_stateread branch from 226a866 to 638c5f5 Compare August 11, 2023 12:31
@erwanor erwanor temporarily deployed to smoke-test August 11, 2023 12:31 — with GitHub Actions Inactive
@erwanor erwanor temporarily deployed to smoke-test August 11, 2023 15:59 — with GitHub Actions Inactive
@erwanor erwanor changed the title storage: implement StateRead::nonverfiable_range (wip) storage: implement StateRead::nonverifiable_range (wip) Aug 11, 2023
@erwanor erwanor changed the title storage: implement StateRead::nonverifiable_range (wip) storage: implement StateRead::nonverifiable_range Aug 11, 2023
@erwanor erwanor temporarily deployed to smoke-test August 11, 2023 20:00 — with GitHub Actions Inactive
@erwanor erwanor temporarily deployed to smoke-test August 11, 2023 20:37 — with GitHub Actions Inactive
@erwanor erwanor marked this pull request as ready for review August 11, 2023 20:43
@erwanor erwanor requested a review from hdevalence August 11, 2023 20:47
@erwanor erwanor temporarily deployed to smoke-test August 11, 2023 22:09 — with GitHub Actions Inactive
@erwanor erwanor temporarily deployed to smoke-test August 14, 2023 15:07 — with GitHub Actions Inactive
@hdevalence
Copy link
Member

Is there any reason not to merge this for the current testnet?

@erwanor
Copy link
Member Author

erwanor commented Aug 21, 2023

Not really, I left it open to invite reviews but merging now

@erwanor erwanor merged commit f3e27d2 into main Aug 21, 2023
@erwanor erwanor deleted the range_queries_stateread branch August 21, 2023 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Status: Testnet 59: Enceladus
Development

Successfully merging this pull request may close these issues.

2 participants