GroveDB v5.0.0

GroveDB v5.0.0 Release Notes

Full Changelog: v4.1.0...v5.0.0

175 Pull Requests · 8 Contributors · 3 New Contributors

---

Breaking Changes

• Update bincode to v2.0.1 by @lklimek in #405
• Split InsertOnly into InsertOnlyKnownToNotExist and InsertIfNotExists by @QuantumExplorer in #595

New Crates

• Add grovedb-merkle-mountain-range crate by @QuantumExplorer in #408
• Add grovedb-dense-fixed-sized-merkle-tree crate by @QuantumExplorer in #409
• Extract grovedb-query crate from merk by @QuantumExplorer in #410
• Add grovedb-bulk-append-tree crate by @QuantumExplorer in #411
• Add grovedb-commitment-tree crate by @QuantumExplorer in #413

Element Types & Query System

New Element Variants

• Add Element::NonCounted wrapper to opt-out of count propagation by @QuantumExplorer in #654
• Add Element::NotSummed wrapper to opt-out of sum propagation by @QuantumExplorer in #659
• Add Element::NotCountedOrSummed wrapper for combined sum+count opt-out by @QuantumExplorer in #666
• Add Element::ReferenceWithSumItem variant + RefreshReferenceWithSumItem batch op by @QuantumExplorer in #667
• Add Element::ProvableSumTree + AggregateSumOnRange query by @QuantumExplorer in #661
• Add Element::ProvableCountProvableSumTree + dual-axis crossover proofs by @QuantumExplorer in #670

Aggregate & Provable Queries

• Aggregate sum path query by @QuantumExplorer in #364
• Add AggregateCountOnRange query for provable count trees by @QuantumExplorer in #656
• Expose aggregate_count proof verification under verify feature by @QuantumExplorer in #658
• Add no-proof query_aggregate_count entry point by @QuantumExplorer in #662
• Allow AggregateCountOnRange as carrier subquery by @QuantumExplorer in #663
• Allow SizedQuery::limit on carrier AggregateCountOnRange by @QuantumExplorer in #664
• No-prove query_aggregate_count_and_sum accumulator by @QuantumExplorer in #676
• Provable offset on ProvableCountTree / ProvableCountSumTree single-range queries by @QuantumExplorer in #669
• Reject NonCounted/NotCountedOrSummed in provable count parents by @QuantumExplorer in #672

Cost Estimation Helpers

• Add required_item/reference_with_sum_item_space helpers by @QuantumExplorer in #673
• Extend EstimatedLayerSizes/EstimatedSumTrees for sum-bearing leaves and Provable* trees by @QuantumExplorer in #674
• Get proved path query benchmarks by @pauldelucia in #372

Proof System

• Include SumItem and ItemWithSumItem in proof generation by @QuantumExplorer in #414
• Enforce succinctness and completeness for non-Merk V1 proof verification by @QuantumExplorer in #415
• Implement proof succinctness and completeness verification by @QuantumExplorer in #519
• Add key_exists_as_boundary_in_proof by @QuantumExplorer in #649
• Add boundaries_in_proof to return all boundary keys by @QuantumExplorer in #651
• Version-gate KVValueHashFeatureTypeWithChildHash to proof V1 only by @QuantumExplorer in #635
• Remove ProveOptions from V1 proofs by @QuantumExplorer in #647
• V1 trunk proofs + count==0 hash verification bypass by @QuantumExplorer in #646
• V0 proof backwards compatibility for KVValueHash item check by @QuantumExplorer in #643
• Verify combine_hash for empty trees in proof result set by @QuantumExplorer in #631
• Verify combine_hash for non-empty trees without subquery in proofs by @QuantumExplorer in #633

Commitment Tree & Orchard ZK

• Add orchard zk support by @QuantumExplorer in #407
• Add rho (nullifier) to commitment tree payload by @QuantumExplorer in #444
• Batched append_many_raw ‚Äî replace _without_frontier API by @QuantumExplorer in #751
• Migrate orchard to 0.14.0 (circuit soundness fix) by @QuantumExplorer in #758
• Add cv_net to the stored note item for OVK recovery by @QuantumExplorer in #761
• Re-export OrchardNoteEncryption + ValueCommitTrapdoor by @shumkov in #762
• Re-export try_output_recovery_with_ovk + OutgoingCipherKey by @shumkov in #763
• Re-export ProofSizeEnforcement + ActionFromPartsError by @shumkov in #760

Storage & Infrastructure

• Feature-gate storage deps so verify builds without RocksDB by @QuantumExplorer in #479
• Use transactional storage context for all specialized trees by @QuantumExplorer in #638
• Subtree dump/restore primitives by @shumkov in #752
• Per-op DeleteTree emptiness check via IsSubtreeNonEmpty enum by @QuantumExplorer in #634

Security Hardening

Critical

• Prevent KV-to-KVValueHash proof value forgery attack (C1) by @QuantumExplorer in #553
• Return errors instead of panicking in calculate_chunk_depths (C2) by @QuantumExplorer in #554
• Validate dense tree height before bit shift (C3) by @QuantumExplorer in #555

High

• Harden proof verification (SEC-005, SEC-006) by @QuantumExplorer in #481
• Use tree-type-aware hash in Merk::verify (H1) by @QuantumExplorer in #556
• Extend Tree::aggregate_data to handle all chunk node types (H2) by @QuantumExplorer in #557
• Verify root hash before committing state sync session (H3) by @QuantumExplorer in #558
• Save self subquery before overwriting in merge (H4) by @QuantumExplorer in #561
• Cap Query.items deserialization to prevent memory exhaustion DoS (H4) by @QuantumExplorer in #596
• Replace panicking unwraps in chunk restoration with error propagation (H5, M4) by @QuantumExplorer in #597
• Add depth limit to proof generation and verification to prevent stack overflow DoS (H3) by @QuantumExplorer in #598
• Batch DeleteTree now enforces emptiness check and cleans up subtree storage (H1, H2) by @QuantumExplorer in #599

Medium

• Prevent u32 overflow in epoch storage flags (M2) by @QuantumExplorer in #531
• Replace expect with error return in range query iterator (M3) by @QuantumExplorer in #533
• Return error instead of panicking on None tree in rewrite_heights (M11) by @QuantumExplorer in #542
• Initialize last_level from previous ops in continue_from_ops (M13) by @QuantumExplorer in #543
• Validate empty subtree chunks during state sync (M2) by @QuantumExplorer in #559
• Correct RangeSetItem Ord anti-symmetry violation (M3) by @QuantumExplorer in #560
• Preserve query flags in merge operations (M5) by @QuantumExplorer in #562
• Enforce validate_insertion_does_not_override_tree in batch path (M6) by @QuantumExplorer in #563
• Include segments_count bytes in worst_case_body_size (M8) by @QuantumExplorer in #565
• Return accumulated costs from commit_batch (M9) by @QuantumExplorer in #566
• Replace unreachable!() with None in RangeSet::to_query_item (M10) by @QuantumExplorer in #567
• Replace unreachable!() with overwrite in Query::insert_item (M11) by @QuantumExplorer in #568
• Replace panic!() with error returns in TreeNode/Walker attach/detach_expect (M2) by @QuantumExplorer in #602
• Return error instead of silently discarding writes when storage batch is None (M3) by @QuantumExplorer in #604
• Add visited-set cycle detection to aggregate sum query reference following (M5) by @QuantumExplorer in #601

Robustness & Panic Elimination

• Return Result from element constructors instead of panicking by @QuantumExplorer in #532
• Replace silent u8 truncation in build_prefix_body by @QuantumExplorer in #534
• Propagate error info in replication state sync by @QuantumExplorer in #535
• Replace panics with error returns in TreeNode::load by @QuantumExplorer in #537
• Consistency check now catches InsertOnly and DeleteTree by @QuantumExplorer in #539
• Prevent arithmetic overflow and truncation (L1,L3,L7,L8,L13,L18,L19) by @QuantumExplorer in #540
• Use saturating_sub for limit/offset decrements (L2) by @QuantumExplorer in #541
• Return error on deserialization failure during proof generation by @QuantumExplorer in #550
• Replace unreachable!() with error returns in Walker::detach by @QuantumExplorer in #548
• Return error instead of panicking on missing flags in JIT update by @QuantumExplorer in #552
• Use explicit .value for discarded costs in Restorer by @QuantumExplorer in #546
• Propagate error and clarify discarded costs in ChunkProducer by @QuantumExplorer in #547
• Propagate IO errors in Visualize for KeyInfoPath by @QuantumExplorer in #581
• Use try_from for path length comparison in delete_up_tree by @QuantumExplorer in #580
• Add visited-set cycle detection to batch reference resolution by @QuantumExplorer in #574
• Use parent tree type for DeleteTree batch operations by @QuantumExplorer in #573
• Distinguish sync-never-started from sync-completed by @QuantumExplorer in #584
• Replace .expect() with error propagation in visualize by @QuantumExplorer in #582
• Add InsertOnly and DeleteTree to apply_operations_without_batching by @QuantumExplorer in #575
• Add depth guard to terminal_keys recursion by @QuantumExplorer in #572
• Add bounds checks to pack_nested_bytes and encode_global_chunk_id by @QuantumExplorer in #579
• Validate version byte in Query::borrow_decode by @QuantumExplorer in #571
• Add u32 bounds checks to chunk blob serialization by @QuantumExplorer in #585
• Return Result from Link::into_reference instead of panicking by @QuantumExplorer in #569
• Propagate write errors in ChunkOp::encode_into by @QuantumExplorer in #589
• Use saturating_add in StorageCost and StorageRemovedBytes by @QuantumExplorer in #587
• Prevent u16 truncation in query limit tracking by @QuantumExplorer in #588
• Enforce set_new_transaction precondition in release builds by @QuantumExplorer in #591
• Preserve original error details in map_err closures by @QuantumExplorer in #590
• Reject proofs with unconsumed trailing bytes by @QuantumExplorer in #593
• Cap batch reference recursion to MAX_REFERENCE_HOPS by @QuantumExplorer in #606
• Enforce InsertIfNotExists existence check for Tree and Reference elements by @QuantumExplorer in #623
• Validate add-on operations consistency in apply_partial_batch by @QuantumExplorer in #622
• Enforce depth limit during Query/SubqueryBranch deserialization by @QuantumExplorer in #613
• Enforce depth limit during proof deserialization by @QuantumExplorer in #612
• Use saturating_add to prevent integer overflow in RangeAfter proof generation by @QuantumExplorer in #618
• Return error instead of panicking on missing subquery in PathQuery::merge by @QuantumExplorer in #499
• Return error instead of panicking on missing old storage flags by @QuantumExplorer in #498
• Handle Link::Uncommitted in verify_link instead of todo!() by @QuantumExplorer in #496
• Propagate errors instead of silently discarding in Restorer::finalize by @QuantumExplorer in #500
• Propagate error instead of silently discarding in Restorer::process_chunk by @QuantumExplorer in #501
• Return empty maps instead of panicking on empty tree in Merk::verify() by @QuantumExplorer in #497
• Return errors instead of panicking on Link::Modified in encoding and traversal by @QuantumExplorer in #502
• Preserve original error information in map_err calls by @QuantumExplorer in #504
• Harden safety invariants in MultiStateSyncSession by @QuantumExplorer in #507
• Use u32 for element count in pack/unpack_nested_bytes to prevent truncation by @QuantumExplorer in #505
• Validate key length on insert to prevent encoding corruption by @QuantumExplorer in #506
• Clamp BigSum values to i64 range in as_sum_i64() by @QuantumExplorer in #508
• Use u16 for root key length in global chunk ID encoding by @QuantumExplorer in #510
• Prevent u16 limit underflow in proof verification by @QuantumExplorer in #522
• Validate key length in batch operations to prevent Merk encoding corruption by @QuantumExplorer in #521
• Enforce internal-only GroveOp variants with #[non_exhaustive] by @thepastaclaw in #523, #526, #527
• Reject trailing bytes during element deserialization by @QuantumExplorer in #724
• Reject trailing bytes in append-only decoders by @QuantumExplorer in #725
• Reject append conflicts with keyed ops by @QuantumExplorer in #731

Bug Fixes

• Handle empty tree in trunk_query instead of erroring by @QuantumExplorer in #412
• Propagate deserialization errors in proof display instead of hiding them by @QuantumExplorer in #431
• Handle KV, KVValueHash, KVCount nodes in chunk restoration by @QuantumExplorer in #450
• Correct empty CommitmentTree and MmrTree hash/proof handling by @QuantumExplorer in #468
• Return error instead of panicking in prove_query_many for empty input by @QuantumExplorer in #491
• Replace panic with debug_assert in StorageFlags serialization by @QuantumExplorer in #503
• Correct version check function name in prove_query by @QuantumExplorer in #512
• Remove leftover eprintln in proof verification by @QuantumExplorer in #549
• Remove eprintln in commit_session by @QuantumExplorer in #578
• Eliminate unnecessary cleanup I/O costs for empty tree deletions by @QuantumExplorer in #639
• Version-gate commit_batch accumulated costs by @QuantumExplorer in #637
• Version-gate add_element_on_transaction (v0=Op::Put, v1=layered) by @QuantumExplorer in #759
• Include known_versions in GroveVersionError messages by @operagxoksana in #652

Performance

• Use stack buffers for varint encoding in hash functions by @QuantumExplorer in #495
• Avoid heap allocation in delete path comparisons by @QuantumExplorer in #494
• Reduce cloning in batch structure construction by @QuantumExplorer in #493
• Optimize insert-under-delete consistency check by @QuantumExplorer in #492
• Replace O(n¬≤) duplicate detection with HashMap-based O(n) by @QuantumExplorer in #516
• Use range deletion for wipe instead of per-key iteration by @QuantumExplorer in #517
• Gate grovedbg build deps behind grovedbg feature by @ZocoLini in #765

Testing

Coverage Campaigns

• Comprehensive test suite for grovedb-version by @thepastaclaw in #432
• Comprehensive test suite for costs by @thepastaclaw in #434
• Comprehensive test suite for visualize by @thepastaclaw in #435
• Comprehensive test suite for bulk-append-tree by @thepastaclaw in #436
• Comprehensive test suite for epoch-flags by @thepastaclaw in #437
• Comprehensive test suite for dense-merkle-tree by @thepastaclaw in #438
• Comprehensive test suite for element by @thepastaclaw in #440
• Comprehensive test suite for storage by @thepastaclaw in #441
• Comprehensive test suite for merk by @thepastaclaw in #442
• Test suite for grovedb-query coverage improvement by @QuantumExplorer in #443
• Missing ChunkError variant coverage by @thepastaclaw in #446

Targeted Coverage

• Add grovedb coverage test suites by @QuantumExplorer in #449
• High-value bulk-append-tree coverage tests by @QuantumExplorer in #453
• Improve dense-tree coverage and exclude test utilities by @QuantumExplorer in #454
• Cover error, visualize, reference_path, and commitment_tree by @QuantumExplorer in #455
• Improve grovedb-query merge coverage from 45% to ~85% by @QuantumExplorer in #456
• Improve merk crate coverage (branch proofs, tree types, iterator) by @QuantumExplorer in #457
• Improve MMR crate code coverage by @QuantumExplorer in #458
• Cover merk average case costs; exclude test_utils from codecov by @QuantumExplorer in #459
• Improve proof directory coverage by @QuantumExplorer in #460
• Remove 59 zero-impact tests from PR #455 by @QuantumExplorer in #461
• Remove 4 redundant zero-impact tests from PR #440 by @QuantumExplorer in #462
• Cover all QueryItem range variants for non-Merk tree proofs by @QuantumExplorer in #463
• Cover delete cost estimation functions by @QuantumExplorer in #464
• Add 33 batch/mod.rs coverage tests, remove dead code by @QuantumExplorer in #467
• Cover deserialization error paths and client append validation by @QuantumExplorer in #475
• Improve element Display, proof_node_type, and display_path coverage by @QuantumExplorer in #476
• Cover Display impls, TreeFeatureType, proofs, and query helpers by @QuantumExplorer in #477
• Cover MMR helper, node, and proof edge cases by @QuantumExplorer in #478
• Cover batch estimated costs and execution paths by @QuantumExplorer in #490
• Cover empty tree early-return path in Merk::verify() by @thepastaclaw in #524
• Proof forgery exploit tests for KVValueHash and KVRefValueHash by @QuantumExplorer in #630
• Re-enable merk AVL fuzz tests by @QuantumExplorer in #583

Documentation

GroveDB Book

• Add GroveDB book as mdBook by @QuantumExplorer in #416
• Add GroveDB book translations in 10 languages by @QuantumExplorer in #419
• Language selector and mermaid fix for translated book by @QuantumExplorer in #420
• Add 6 new book translations and flag emojis by @QuantumExplorer in #421
• Streamline README with prominent book link by @QuantumExplorer in #422
• Fix translation diacritics and terminology in es/fr/pl/id by @thepastaclaw in #448
• Add quantum cryptography chapter by @QuantumExplorer in #447
• Add Aggregate Sum Queries chapter by @QuantumExplorer in #487
• Add boundary key detection to proof system chapter by @QuantumExplorer in #650
• Add per-tree-type proof node reference by @QuantumExplorer in #644
• Sync book translations with English source by @QuantumExplorer in #642

Code Documentation & Safety Notes

• Mark internal-only GroveOp variants to prevent misuse by @QuantumExplorer in #515
• Clarify query_sums vs query_aggregate_sums relationship by @QuantumExplorer in #518
• Explain why runtime key length check is unnecessary in proof encoding by @QuantumExplorer in #525
• Document why encode_into expect is unreachable by @QuantumExplorer in #551
• Document why to_ne_bytes is intentional in build_prefix_body by @QuantumExplorer in #544
• Document why all-0xFF prefix overflow in seek_to_last is not reachable by @QuantumExplorer in #538
• Document Owner::own_result poisoning and improve panic message by @QuantumExplorer in #536
• Document balance_factor overflow impossibility by @QuantumExplorer in #564
• Document put-wins semantics in StorageBatch::delete by @QuantumExplorer in #576
• Clarify clear() only affects default column family by @QuantumExplorer in #577
• Document dangling reference behavior on element deletion by @QuantumExplorer in #605
• Document intentional panic in Link::Modified by @QuantumExplorer in #609
• Security notes for ProveOptions, consistency check, and atomicity by @QuantumExplorer in #608
• Document intentional fixed-size encoding_cost by @QuantumExplorer in #611
• Document add_parent_tree_on_subquery limit bypass by @QuantumExplorer in #610
• Document emptiness check limitation for same-batch inserts by @QuantumExplorer in #615
• Document chunk restore height placeholder safety by @QuantumExplorer in #619
• Document single-writer transaction requirement by @QuantumExplorer in #617
• Document non-atomic delete cost computation (TOCTOU) by @QuantumExplorer in #616
• Document why Owner::own_result poisoning is safe by @QuantumExplorer in #626
• Document find_subtrees stale state limitation during batch delete by @QuantumExplorer in #620
• Document cross-type aggregate safety by @QuantumExplorer in #614
• Mark intentional audit false positives so they aren't re-flagged by @QuantumExplorer in #754
• Explain why completed-chunk-length check isn't needed by @QuantumExplorer in #740

CI & Build System

• Move slow Sinsemilla tests to separate workflow by @QuantumExplorer in #417
• Modernize coverage with cargo-llvm-cov and per-crate badges by @QuantumExplorer in #418
• Run coverage on push so codecov badges work by @QuantumExplorer in #426
• Point codecov badges to develop branch by @QuantumExplorer in #428
• Remove redundant errors job, move verify check into linting by @QuantumExplorer in #429
• Skip coverage on non-code PRs by @QuantumExplorer in #430
• Add PR coverage diff checker by @QuantumExplorer in #451
• Restore coverage baseline detection in PR diffs by @QuantumExplorer in #465
• Remove redundant pre-commit workflow by @QuantumExplorer in #466
• Replace custom coverage infra with native Codecov by @QuantumExplorer in #469
• Speed up CI with sccache, test sharding, and precise filters by @QuantumExplorer in #470
• Speed up security audit with pre-built cargo-audit by @QuantumExplorer in #471
• Add Codecov component definitions for per-crate badges by @QuantumExplorer in #472
• Restore cross-crate coverage with sharded workspace tests by @QuantumExplorer in #473
• Add sccache to linting job by @QuantumExplorer in #474
• Replace deprecated actions-rs/toolchain with dtolnay/rust-toolchain by @QuantumExplorer in #483
• Fix rust-cache to cache llvm-cov target directory by @QuantumExplorer in #624
• Self-hosted Mac runner with Ubuntu fallback; restrict to internal PRs by @QuantumExplorer in #641, #645, #668
• Skip tests for doc-comment-only PRs by @QuantumExplorer in #629
• Raise patch coverage requirement from 80% to 90% by @QuantumExplorer in #660

Maintenance

• Drop nightly-only rustfmt options by @QuantumExplorer in #423
• Add pre-commit hooks and modernize CI by @QuantumExplorer in #424
• Resolve all clippy warnings for CI compatibility by @QuantumExplorer in #425
• Remove unmaintained node-grove crate by @QuantumExplorer in #427
• Add .claude to .gitignore by @QuantumExplorer in #452
• Fix author email typo and migrate to edition 2024 by @QuantumExplorer in #482
• Upgrade rand to 0.10 with workspace dependency by @QuantumExplorer in #485, #486
• Enforce deny(missing_docs) across all crates by @QuantumExplorer in #488
• Exclude debug/visualize files from coverage by @QuantumExplorer in #489
• Ignore accepted bincode advisory by @QuantumExplorer in #723
• Bump workspace crate versions to 5.0.0 by @QuantumExplorer in #764

Version Gating

• Add version gating to proof generation methods by @QuantumExplorer in #513
• Start protocol_version numbering at 1 instead of 0 by @QuantumExplorer in #528
• Reproduce merge_multiple default_subquery_branch bug (M6) by @QuantumExplorer in #648

New Contributors

• @lklimek made their first contribution in #405
• @operagxoksana made their first contribution in #652
• @ZocoLini made their first contribution in #765

Full Changelog: v4.1.0...v5.0.0