feat: addition of more range types #82
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fominok
approved these changes
Mar 11, 2022
QuantumExplorer
requested changes
Mar 14, 2022
fominok
approved these changes
Mar 14, 2022
QuantumExplorer
approved these changes
Mar 16, 2022
QuantumExplorer
added a commit
that referenced
this pull request
May 24, 2022
* test range from proof * add unbounded right * test for range to proof * add left unbounded * add absence tuple checks to test * test range to inclusive * test for range after proof * test for range after proof * test for range after to inclusive * test for range full proof * update verify proof func to account for unbounded ranges * add verification check for right non inclusive ranges * refactor bounds to return options * update verify function to work with range_after_* queries * fix range_after_proof test * fix range_after_to_proof test * fix range_after_to_proof_inclusive test * fix range_full_proof * modify create_full_proof function to take optional limit and offset values * add failing test to limit result set to 1 item * removed unnecessary comments * implement limit functionality in merk * add limit test for range from proof * feat: addition of more range types (#82) * test range from proof * add unbounded right * test for range to proof * add left unbounded * add absence tuple checks to test * test range to inclusive * test for range after proof * test for range after proof * test for range after to inclusive * test for range full proof * update verify proof func to account for unbounded ranges * add verification check for right non inclusive ranges * refactor bounds to return options * update verify function to work with range_after_* queries * fix range_after_proof test * fix range_after_to_proof test * fix range_after_to_proof_inclusive test * fix range_full_proof * define KVDigest node variant * implement encode for kvdigest * implement decode for kvdigest * add ability to convert node to kvdigest * add test for node variant conversions * add node hash computation for kvdigest node variant * feat: new node type Node::KVDigest (#89) * define KVDigest node variant * implement encode for kvdigest * implement decode for kvdigest * add ability to convert node to kvdigest * add test for node variant conversions * add node hash computation for kvdigest node variant * extract main verify proof body into closure * update verify_query to work with kvdigest node * fixed absence proof tests * remove marker comments * push KVDigest for non inclusive bound keys * add limit argument to verify_query function * add failing test for proof with limit verification * implement proof with limit verification * fix range_from proof for limits * add limit test for range_to_proof * add limit test for range_to_proof_inclusive * fix bug: don't reserve limit slot for non inclusive nodes * add limit test for range_after_proof * add limit test for range_after_to_proof * add limit test for range_full_proof * add poc failing test for offset * ignore limit as long as offset exists and is not zero * updated create_proof to make use of offset * pass offset as input to verify_query function * modify verify_query function to work with offset * fix typos * make proof_with_offset test verbose * add offset test for range_from_proof * add offset test for range_to_proof * add offset test for range_to_proof_inclusive * add offset test for range_after and range_after_to * add offset test for range_after_to_inclusive and range_full * add failing offset test for range proof * fix bug: prevent reservation of offset slot for non query keys * add offset test for range_proof_inclusive * add encoding for child_inverted and parent_inverted op codes * add encoding for push inverted op code * modify execute function to take into account the new op codes * add right_to_left logic to proof creation * modify create_full_proof api to take left_to_right bool * add poc verification that right_to_left generates correct elements * add failing test for right_to_left proof verification * modify verify_query function to take the left_to_right bool as input * modify verify_query to work with right_to_left * cleanup * add right_to_left test for range_proof, range_proof_inclusive and range_from_proof * add right_to_left proof test for remaning range types * add absence tests * minor cleanup * modify verify_query function to return limit and offset * refactor test to use new verification interface * add test to check validity of limit and offset returned after verification * define interface for proof construction and execution * wip * generate proof up to root - 1 * generating proof up to root * combine individual proofs into single structure * reading proof data from proof buffer during execution * extract proof verification algo into execute_proof function * fmt * make proof verification field public * verify proof up to root - 1 * fix bug where value hash gets cleared on decode * v0.5 implementation of path query proofs without subqueries * implement read_proof function * refactor execute proof to use proof reader * cleanup * make use of left_to_right from query * make use of limit and offset values provided by query * add test for path_query_proofs_without_subquery * get rid of non result errors * journey begins * refactor prove function to return limit and offset values * fix test to use new proof result interface * add basic test to verify the result of proof construction limit and offset propagation * cleanup * get all elements in the subtree * define interface for recursion + algo * implement prove_subqueries to construct proof without subqueries * add temporary default subquery paths function * recursively prove tree elements based on subqueries * add basic visibility test for subquery proofs * propagate limit and offset values in prove_subqueries function * generating sized merk proofs for leaf nodes of path queries * fix path_query_proof_without_subquery test * refactor proof types as enums * implement convertion from proof_type enum to bytes and vice versa * add read_proof function to proof_reader that makes no assumption of proof type * fix test_path_query_proof_without_subquery verification test * it works!! * send proof of all children for non leaf subquery nodes * update test for default subquery to get result set across subtrees * add range type subquery test * define new schema for more convuluted tree * add variable leaf size length * add test for with 3 subqueries * extract deep tree builder into a separate function * add test for subquery key * wip * add support for subquery_key without subquery * generate proof for subquery key existence * verification of intermediate subquery key proof * optionally run subquery key logic * switch get subquery to use conditional subquery getter * prevent subquery proof logic from running if no subquery * add test for conditional subqueries * remove old proof test * add test for conditional subquery + default subquery * add failing test for sized query * sized path query works * add failing test for path query proof with direction * fix directional path query test * add interesting test with mixed query directions * remove unnecessary comments * extract subquery subroutines into separate functions * basic clean up * more cleanup * don't generate or verify proof for empty path * add more comments to execute proof routine * extract merk proof to bytes generation into separate routine * extract merk proof execution into seperate routine * remove duplication from proof reader * clean up + more todos * encode and decode value_hash for kv node * fix test_follow_reference create reference element before creating reference * on reference insert, get the referenced element and pass the value hash * refactor reference insertion to use value hash of referenced value * references compute to the same hash as their referenced element * add test for path query with reference elements * allow merk to call grovedb methods using traits * Revert "allow merk to call grovedb methods using traits" This reverts commit 74731b4. * add new prove function that doesn't encode values * created intermidiary zone for future proof value substitution * proof properly handles referencesgit add . * fixed reference test * return error from generate_and_store_merk_proof * handle errors in proof reader * add test to show references must point to a valid item before insertion * fix test_too_many_indirections * clean insert function * add documenatation + cleanup * fmt * more clean up * add fix comment * cleanup * remove macro use in proof (never generating a proof on transactional data) * remove unwrap from get kv pairs function * allow large proof size with larger length encoding * reduce vector allocation + make debug asserts explicit * read directly into known size slice * cleanup * refactor: proof logic (#107) * range proof test * fix get kv pairs * add directional iter for queries * remove all_key queries * stateless get_kv wip * add failing test for unique subquery * add subquery key logic back into proof generation function * subquery logic for verification wip * refactor: extract limit and offset to struct * refactor: extracted result set to struct * update result set for subquery key logic * fix failing subquery key test * add more passing tests * all query tests work with proof * refactor proof query test to compare all result set elements * implement kv pairs iterator (prevents storing all kv pairs in memory before consumption) * cleanup * add documenation for kviterator * remove inefficient use of query position * resolve conflicts Co-authored-by: Quantum Explorer <quantum@dash.org>
QuantumExplorer
added a commit
that referenced
this pull request
Jun 27, 2022
* feat: var int serialization (#94) * feat: var int serialization * fix: removed unused imports * feat: added in serialized byte size of an element * feat: added node_byte_size * feat(storage): multi-context batches (#96) add batches to do deferred operations on multiple paths * feat: GroveDB batch operations (#106) * wip * wip * abstract transaction in batch application * remove unnecessary 'ctx lifetime * wip * add validation and tests * wip * wip, add pretty debugs * wip passing tests * fix validation * qfix conflicting operations * fix Merk cleanup * fix unnecessary temp_tree insertion * keep insertions over deletions in a batch * feat: proofs v2 (#103) * test range from proof * add unbounded right * test for range to proof * add left unbounded * add absence tuple checks to test * test range to inclusive * test for range after proof * test for range after proof * test for range after to inclusive * test for range full proof * update verify proof func to account for unbounded ranges * add verification check for right non inclusive ranges * refactor bounds to return options * update verify function to work with range_after_* queries * fix range_after_proof test * fix range_after_to_proof test * fix range_after_to_proof_inclusive test * fix range_full_proof * modify create_full_proof function to take optional limit and offset values * add failing test to limit result set to 1 item * removed unnecessary comments * implement limit functionality in merk * add limit test for range from proof * feat: addition of more range types (#82) * test range from proof * add unbounded right * test for range to proof * add left unbounded * add absence tuple checks to test * test range to inclusive * test for range after proof * test for range after proof * test for range after to inclusive * test for range full proof * update verify proof func to account for unbounded ranges * add verification check for right non inclusive ranges * refactor bounds to return options * update verify function to work with range_after_* queries * fix range_after_proof test * fix range_after_to_proof test * fix range_after_to_proof_inclusive test * fix range_full_proof * define KVDigest node variant * implement encode for kvdigest * implement decode for kvdigest * add ability to convert node to kvdigest * add test for node variant conversions * add node hash computation for kvdigest node variant * feat: new node type Node::KVDigest (#89) * define KVDigest node variant * implement encode for kvdigest * implement decode for kvdigest * add ability to convert node to kvdigest * add test for node variant conversions * add node hash computation for kvdigest node variant * extract main verify proof body into closure * update verify_query to work with kvdigest node * fixed absence proof tests * remove marker comments * push KVDigest for non inclusive bound keys * add limit argument to verify_query function * add failing test for proof with limit verification * implement proof with limit verification * fix range_from proof for limits * add limit test for range_to_proof * add limit test for range_to_proof_inclusive * fix bug: don't reserve limit slot for non inclusive nodes * add limit test for range_after_proof * add limit test for range_after_to_proof * add limit test for range_full_proof * add poc failing test for offset * ignore limit as long as offset exists and is not zero * updated create_proof to make use of offset * pass offset as input to verify_query function * modify verify_query function to work with offset * fix typos * make proof_with_offset test verbose * add offset test for range_from_proof * add offset test for range_to_proof * add offset test for range_to_proof_inclusive * add offset test for range_after and range_after_to * add offset test for range_after_to_inclusive and range_full * add failing offset test for range proof * fix bug: prevent reservation of offset slot for non query keys * add offset test for range_proof_inclusive * add encoding for child_inverted and parent_inverted op codes * add encoding for push inverted op code * modify execute function to take into account the new op codes * add right_to_left logic to proof creation * modify create_full_proof api to take left_to_right bool * add poc verification that right_to_left generates correct elements * add failing test for right_to_left proof verification * modify verify_query function to take the left_to_right bool as input * modify verify_query to work with right_to_left * cleanup * add right_to_left test for range_proof, range_proof_inclusive and range_from_proof * add right_to_left proof test for remaning range types * add absence tests * minor cleanup * modify verify_query function to return limit and offset * refactor test to use new verification interface * add test to check validity of limit and offset returned after verification * define interface for proof construction and execution * wip * generate proof up to root - 1 * generating proof up to root * combine individual proofs into single structure * reading proof data from proof buffer during execution * extract proof verification algo into execute_proof function * fmt * make proof verification field public * verify proof up to root - 1 * fix bug where value hash gets cleared on decode * v0.5 implementation of path query proofs without subqueries * implement read_proof function * refactor execute proof to use proof reader * cleanup * make use of left_to_right from query * make use of limit and offset values provided by query * add test for path_query_proofs_without_subquery * get rid of non result errors * journey begins * refactor prove function to return limit and offset values * fix test to use new proof result interface * add basic test to verify the result of proof construction limit and offset propagation * cleanup * get all elements in the subtree * define interface for recursion + algo * implement prove_subqueries to construct proof without subqueries * add temporary default subquery paths function * recursively prove tree elements based on subqueries * add basic visibility test for subquery proofs * propagate limit and offset values in prove_subqueries function * generating sized merk proofs for leaf nodes of path queries * fix path_query_proof_without_subquery test * refactor proof types as enums * implement convertion from proof_type enum to bytes and vice versa * add read_proof function to proof_reader that makes no assumption of proof type * fix test_path_query_proof_without_subquery verification test * it works!! * send proof of all children for non leaf subquery nodes * update test for default subquery to get result set across subtrees * add range type subquery test * define new schema for more convuluted tree * add variable leaf size length * add test for with 3 subqueries * extract deep tree builder into a separate function * add test for subquery key * wip * add support for subquery_key without subquery * generate proof for subquery key existence * verification of intermediate subquery key proof * optionally run subquery key logic * switch get subquery to use conditional subquery getter * prevent subquery proof logic from running if no subquery * add test for conditional subqueries * remove old proof test * add test for conditional subquery + default subquery * add failing test for sized query * sized path query works * add failing test for path query proof with direction * fix directional path query test * add interesting test with mixed query directions * remove unnecessary comments * extract subquery subroutines into separate functions * basic clean up * more cleanup * don't generate or verify proof for empty path * add more comments to execute proof routine * extract merk proof to bytes generation into separate routine * extract merk proof execution into seperate routine * remove duplication from proof reader * clean up + more todos * encode and decode value_hash for kv node * fix test_follow_reference create reference element before creating reference * on reference insert, get the referenced element and pass the value hash * refactor reference insertion to use value hash of referenced value * references compute to the same hash as their referenced element * add test for path query with reference elements * allow merk to call grovedb methods using traits * Revert "allow merk to call grovedb methods using traits" This reverts commit 74731b4. * add new prove function that doesn't encode values * created intermidiary zone for future proof value substitution * proof properly handles referencesgit add . * fixed reference test * return error from generate_and_store_merk_proof * handle errors in proof reader * add test to show references must point to a valid item before insertion * fix test_too_many_indirections * clean insert function * add documenatation + cleanup * fmt * more clean up * add fix comment * cleanup * remove macro use in proof (never generating a proof on transactional data) * remove unwrap from get kv pairs function * allow large proof size with larger length encoding * reduce vector allocation + make debug asserts explicit * read directly into known size slice * cleanup * refactor: proof logic (#107) * range proof test * fix get kv pairs * add directional iter for queries * remove all_key queries * stateless get_kv wip * add failing test for unique subquery * add subquery key logic back into proof generation function * subquery logic for verification wip * refactor: extract limit and offset to struct * refactor: extracted result set to struct * update result set for subquery key logic * fix failing subquery key test * add more passing tests * all query tests work with proof * refactor proof query test to compare all result set elements * implement kv pairs iterator (prevents storing all kv pairs in memory before consumption) * cleanup * add documenation for kviterator * remove inefficient use of query position * resolve conflicts Co-authored-by: Quantum Explorer <quantum@dash.org> * simplify `check_subtree_exists`, bugfix (#110) * feat: element flags (#109) * add helper functions to create different element variants * update element interface in batch file * update element interface in delete module * update element inteface in get module * update element interface in insert module * update element interface in generate proof module * update element interface in verify proof module * update element interface in subtree module * update element interface in visualize module * update element interface in lib * update element interface in bindings * updated element interface for tests * disable serialization test, all other tests pass * add new functions to create elements with flags * add passing tests for item with flag and reference with flag * add failing test for tree type, flag is not preserved * refactor propagate changes to preserve flags * removed duplication from propagate changes * re-enabled serialization test * add support for multi byte flag data + tests * add test to show that flags are part of proofs * fmt * add serialization failing tests * fix serialization tests * remove unnecessary comments * fix: make element flag type public (#111) * make element flag type public * expose ElementFlag from grovedb lib * feat: return keys on query (#112) * feat: easily extract flags from an element (#113) * rename element flag to element flags * add convinience method for getting flags + update test * add support for empty tree with flag (#114) * feat: small improvements needed for rs-drive fees. (#115) * feat: var int serialization * fix: removed unused imports * feat: added in serialized byte size of an element * feat: added node_byte_size * made things public * query fingerprint * fixed node_byte_size * added calculate_node_byte_size class function * added required_item_space class function * fmt * refactor: small improvements (#116) * modify execute_proof to that path_query as ref (#117) * public batch module (#118) * feat: has element optimization (#119) * temp work * add `has_raw` function Co-authored-by: Evgeny Fomin <fominok@hotmail.com> * fix: seek_for_iter (#120) * fix seek_for_iter: work for single key * fmt * made some fields public (#122) * Revert "made some fields public (#122)" (#123) This reverts commit 0635758. * feat: costs for Merk operations (#121) * wip * wip * wip * wip * wip * wip * wip * wip * wip * add deletion cost info * ignore costs in grovedb * feat: Costs for GroveDB operations (#124) * wip * wip * wip * wip * wip * wip * wip * wip * wip * add deletion cost info * ignore costs in grovedb * wip * wip * wip * wip * wip * wip * fix tests * batches api * feat: batch delete up tree while empty and small batch improvements (#125) * trial * trial * fix has_raw root leaf issue * made some fields public * small fix * delete up tree while empty operations * fix * fix * fix * fix * cost context public members Co-authored-by: Evgeny Fomin <fominok@hotmail.com> * fmt (#128) * fix: handle path pointing to empty tree case (#127) * handle path pointing to empty tree case * clean up comments * feat: batches and start of worst case fees (#131) * trial * add test for sorted_pre_validated apply propagaion * slight optimisation * trial * trial * start on worst case fees * fix batches * temp work * work * extra comment * add more pretty debug implementations * fixes * fmt and fixes * small pub * small fix for deletion * remove dbg * small change * Batch worst case fees typelevel fixes (#132) * wip * wip * remove storage dependency where it's not needed * fixes Co-authored-by: Evgeny Fomin <fominok@hotmail.com> * updated dependencies * feat: `query_many` and `prove_query_many` to accept multiple queries (#130) * move path query related code to a separate file * add simple failing test for path query merge functionality * implemented simple function to combine queries + fix for failing test * clean up * implement function to get common path and next index * implements first version of build query * building path queries end to end * failing test passes * fix non-determinism in build query * add failing test for longer paths * handles merging of arbitrary sized same length paths * cleanup * modify path query merge api to take in variable number of path queries * modify test to merge more than 3 path queries * add support for duplicate exact paths mixed with different paths + possibly different level paths * add test for variable length path query merge * refactored prove and query function name and signatures * adds prove_query_many and verify_query_many * modify merge function signature to return cost (inacurate cost tho) * get common path works with length of the shortest path * cleanup * remove unnecessary comments * small fixes Co-authored-by: Quantum Explorer <quantum@dash.org> * some worst case operations (#134) * style: clippy fixes (#135) * clippy fixes * update neon verison and fixes Co-authored-by: Evgeny Fomin <fominok@hotmail.com> Co-authored-by: Wisdom Ogwu <40731160+iammadab@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merk currently supports proofs for:
This pr adds: