GroveDB v6.1.0

GroveDB 6.1.0 follows 6.0.1. It adds element moves to batches, upper-bound proof limits and aux reads by key prefix, and fixes batch, estimate and proof correctness on GROVE_V4 along with a set of panics and unbounded work reachable from untrusted input. It is the GroveDB release for Dash Platform 5.0 (protocol version 14).

GROVE_V4 changes in place. GROVE_V4 is still the latest version, and several fixes below change what it does (new version slots set to 1 in GROVE_V4). A node replaying operations executed under GROVE_V4 with 6.0.x can reach different results for the operations those fixes touch. GROVE_V1 to GROVE_V3 behave exactly as in 6.0.1.

Changes since 6.0.1

Element moves in batches (#1015)

GroveOp::Move and QualifiedGroveDbOp::move_op(path, key, new_key) rename a key within its parent. The element keeps its bytes and node value hash; for a tree, every record under it (nested Merk trees, non-Merk tree data, indexed trees' secondary Merks) is copied byte for byte to the new path's prefixes, so nothing below is re-hashed or rebalanced. Moves can swap or chain keys in one batch and are billed as storage that stays where it is. Moves of backward-reference participants, of references that would no longer resolve from the new key, inside indexed trees, in partial batches and in cost estimates are refused. Accepted from GROVE_V4 (apply_batch.move_element). The same release makes a reference resolved through a pending skip-mode InsertIfNotExists reach the element already stored there (apply_batch.reference_through_conditional_insert, GROVE_V4).

Same-batch references and replacements commit to the bytes stored (#1021)

On GROVE_V4, a reference written or refreshed in the same batch as an update of its target item commits to exactly the bytes the target stores, predicted through Merk's own just-in-time value update, once per target. Before, a sum item (SumItem, ItemWithSumItem) was assumed to keep its stored flags, so a flags update that merged them (an owned item updated in a later epoch, an unowned one gaining an owner) left the reference committed to bytes never stored, and proofs through it failed. The batch now also refuses a target whose write stores other bytes than its prediction. A replacement whose flags update leaves the flags unchanged is charged for the bytes it stores, where a new value with flags of another length used to fail the commit with storage_cost cost mismatch (apply_batch.same_batch_reference_target_prediction, merk_versions.tree.just_in_time_value_update).

Cost estimates match the apply (#1018, #1019, #1020)

From GROVE_V4, average-case batch estimates size sum, big sum, count and count-sum trees as themselves, and both average- and worst-case estimates size a SumItem or ItemWithSumItem the way it is stored. Worst-case estimates are an upper bound again for sum items (operations.average_case.average_case_propagated_aggregate_kind, operations.average_case.average_case_sum_item_cost_size, operations.worst_case.worst_case_sum_item_cost_size).

Proofs

• Upper-bound limits (#1013). ProofLimitMode::UpperBound and QueryProofVerify::execute_proof_with_limit_mode verify a proof with limit as a ceiling, so a client no longer has to know the server's limit to verify an honest proof. GroveDb::verify_aggregate_{count,sum,count_and_sum}_query_per_key_up_to_limit page carrier aggregates the same way. ProofVerificationResult::exhausted reports whether nothing further matches.
• Limit-cut V1 proofs verify (#1012). A V1 proof whose limit runs out before its query now hides a range bound it passes back through, which verify_query used to reject. Same root hash, verifier unchanged, so clients verify these as soon as the serving node upgrades.
• Empty provable trees on an aggregate path (#1010, #1011). An aggregate-on-range proof whose path runs through an empty Provable* tree above its terminal is generated instead of failing.
• Axis reads over an absent path (#965). Single-path axis reads answer with the empty result on read, prove and verify, instead of a storage error.

Indexed trees (#1003)

From GROVE_V4, a ProvableCountProvableSumIndexedTree primary accepts a bare SumItem child, aggregated as a count of one and its sum, so groups of counters can be ranked without wrapping each counter (insert.validate_indexed_child_for_variant).

Hardening against untrusted input

• MMR proof verification rejects an mmr_size no MMR can have (#974).
• Chunk ops are validated before their bytes are trusted (#975).
• Multi-epoch storage flags decode only from the canonical encoding (#977).
• Count-offset proofs refuse backward-reference rows (#988).
• fetch_chunk refuses a peer-chosen tree type of another aggregate family instead of panicking (#989).
• Bulk-append and commitment-tree proof verification is bounded before the proof is bound to a trusted root (#990).
• Bidirectional referrers commit to the bytes the flags callback stores (#991).
• An overlong path segment (#1006), an empty path on any reference hop (#1016) and a nested element wrapper (#1017) are refused instead of panicking.

Storage and performance

• StorageContext::raw_iter_aux and GroveDb::get_aux_by_key_prefix list aux entries by key prefix (#968).
• Delete builders borrow the batch's pending operations (impl PendingOperations) instead of copying them for every delete (#995). From GROVE_V4 they no longer count a skipped DeleteTree as emptying its parent.

Docs and CI

StorageFlags API docs (#999), an audit note on #779 (#1005), and PR Hygiene and review-engine updates.

Migration from 6.0.1

• Code building version structs by hand must set the new slots: GroveDBApplyBatchVersions (move_element, reference_through_conditional_insert, same_batch_reference_target_prediction), GroveDBOperationsInsertVersions (validate_indexed_child_for_variant), MerkTreeVersions (just_in_time_value_update), and the average- and worst-case estimate slots above.
• Exhaustive matches need new arms: GroveOp::Move; merk's Op::PutMoved and Op::DeleteMoved; StorageFlagsError::NonCanonicalStorageFlags.
• ProofVerificationResult has a new exhausted field.
• delete_operation_for_delete_internal and delete_operations_for_delete_up_tree_while_empty take impl PendingOperations. A slice, &Vec or vec![] compiles as before; a &mut Vec, &&Vec or &Arc<Vec<_>> is passed as ops.iter().
• TreeNode::put_value_with_fixed_cost, the two reference-hash put functions and their Walker counterparts take a &GroveVersion; TreeNode::provided_value_hash_put_final_value is now TreeNode::predict_put_final_value.
• Flags callbacks passed to apply_batch_with_element_flags_update must be deterministic: the batch predicts with them and refuses a write that stores anything else.

Versions

All GroveDB workspace crates are 6.1.0; grovedb-bincode and grovedb-bincode-derive stay at 2.1.0. GROVE_V4 is the latest GroveVersion, with the new slots above.

Contributors

Thanks to @QuantumExplorer and @shumkov.

All merged pull requests since v6.0.1

• ci: re-pin PR Hygiene by @shumkov in #959
• ci: re-pin PR Hygiene by @shumkov in #964
• fix: answer a single-path axis read over an absent path with the empty result by @QuantumExplorer in #965
• ci: re-pin PR Hygiene by @shumkov in #966
• ci: re-pin PR Hygiene to cut its GitHub request volume by @shumkov in #967
• feat(storage): list aux entries by key prefix by @QuantumExplorer in #968
• ci: re-pin PR Hygiene so the sweep reaches every pull request by @shumkov in #969
• ci: re-pin PR Hygiene so its runs queue instead of cancelling by @shumkov in #970
• ci: re-pin PR Hygiene to read each pull request once by @shumkov in #971
• ci: wake PR Hygiene only for a comment that can change the answer by @shumkov in #972
• ci: re-pin PR Hygiene to clone this repository only on its sweep by @shumkov in #973
• fix(mmr): reject non-canonical mmr_size in proof verification by @QuantumExplorer in #974
• fix(merk): validate ChunkOp bytes before trusting them by @QuantumExplorer in #975
• ci: re-pin PR Hygiene to require a green build before asking a human by @shumkov in #978
• ci: re-pin PR Hygiene so it can verify access granted by the organisation by @shumkov in #979
• ci: PR Hygiene is the merge gate; CODEOWNERS carries no rules by @shumkov in #980
• ci: re-pin PR Hygiene for state labels and /skip-bots by @shumkov in #981
• ci: re-pin PR Hygiene for the checklist in the description by @shumkov in #982
• ci: re-pin PR Hygiene so a bot line says everything about that bot by @shumkov in #983
• ci: re-pin PR Hygiene to ask a flaky read once more by @shumkov in #984
• ci: re-pin PR Hygiene for one name for the author limit by @shumkov in #985
• ci: re-pin PR Hygiene by @shumkov in #986
• ci: re-pin PR Hygiene by @shumkov in #987
• fix(replication): refuse a peer-chosen tree type of another family when serving chunks by @QuantumExplorer in #989
• fix(merk,grovedb): refuse backward-reference rows in count-offset proofs by @QuantumExplorer in #988
• fix(proof): bound bulk-append extraction before the root is bound by @QuantumExplorer in #990
• fix(batch): commit bidirectional referrers to the bytes the flags callback stores by @QuantumExplorer in #991
• ci: re-pin PR Hygiene by @shumkov in #992
• ci: re-pin PR Hygiene by @shumkov in #993
• ci: re-pin PR Hygiene by @shumkov in #994
• perf!: borrow the pending batch when building delete operations by @QuantumExplorer in #995
• ci: re-pin PR Hygiene by @shumkov in #996
• ci: re-pin PR Hygiene and wake it on a hand-over by @shumkov in #997
• ci: re-pin PR Hygiene by @shumkov in #998
• docs(storage-flags): document the real StorageFlags API by @QuantumExplorer in #999
• fix(storage-flags): accept only canonical multi-epoch flag bytes by @QuantumExplorer in #977
• ci: re-pin PR Hygiene engine to 3b98772 by @shumkov in #1000
• feat(indexed-tree)!: accept a bare SumItem under a ProvableCountProvableSumIndexedTree (GROVE_V4) by @QuantumExplorer in #1003
• chore(ci): run PR Hygiene's engine from master by @shumkov in #1004
• docs(audit): record #779 as a batch-caller gap Platform never reaches by @QuantumExplorer in #1005
• fix: refuse an overlong path segment instead of panicking by @QuantumExplorer in #1006
• fix(proof)!: prove a path through an empty provable tree (GROVE_V4) by @QuantumExplorer in #1010
• fix(proof): prove a path through an empty provable tree at every grove version by @QuantumExplorer in #1011
• fix(proof): hide a range bound a V1 proof passes after its limit by @QuantumExplorer in #1012
• fix(batch): refuse an empty path on every reference hop instead of panicking by @QuantumExplorer in #1016
• fix(batch): refuse a nested element wrapper instead of panicking by @QuantumExplorer in #1017
• feat(batch)!: move an element to a new key, keeping its stored subtree by @QuantumExplorer in #1015
• fix(batch)!: size aggregating trees as themselves in average-case batch estimates (GROVE_V4) by @QuantumExplorer in #1018
• fix(estimated-costs)!: size sum items as the apply does in average-case estimates (GROVE_V4) by @QuantumExplorer in #1019
• fix(estimated-costs)!: size sum items as the apply does in worst-case estimates (GROVE_V4) by @QuantumExplorer in #1020
• fix(batch)!: commit same-batch sum-item references and replacements to the bytes stored (GROVE_V4) by @QuantumExplorer in #1021
• chore(release): bump GroveDB to 6.1.0 by @QuantumExplorer in #1022

Full Changelog: v6.0.1...v6.1.0