fix: model V4 gated stored-element reads in batch estimated costs - #789
Merged
Conversation
The GROVE_V4 gates delete_tree_cleanup_type_source and overwrite_indexed_cleanup_inspection each charge one extra stored-element read per DeleteTree / overwrite-capable op in the applied batch path, but the average- and worst-case batch estimators did not model that read, so estimated costs undershot applied costs by exactly one read per affected op at grove version >= 4 (surfaced by Dash Platform identity-op fee tests at protocol v14). Charge the read in both estimators, keyed off the same two version slots so V1..V3 estimates are unchanged: - overwrite gate: one average/worst-case Merk node read of the layer's estimated element, skipped when tree-override validation is on, for bare-reference elements, and for layers known to be empty — the same conditions under which the applied path skips or gets the read for free. - delete-tree gate: the parent context prefix hash plus one seek loading the layered element cost, matching Element::get_from_storage exactly; the average-case estimated delta therefore equals the applied delta field for field, which the new parity test pins. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Warning Review limit reached
Next review available in: 9 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #789 +/- ##
===========================================
+ Coverage 92.11% 92.13% +0.01%
===========================================
Files 257 257
Lines 77936 78297 +361
===========================================
+ Hits 71794 72137 +343
- Misses 6142 6160 +18
🚀 New features to boost your workflow:
|
QuantumExplorer
commented
Aug 3, 2026
QuantumExplorer
left a comment
QuantumExplorer
left a comment
Member
Author
There was a problem hiding this comment.
This will work
5 tasks
QuantumExplorer
added a commit
that referenced
this pull request
Aug 3, 2026
…elements (#790) * fix: make the V4 batch gates zero-cost by reusing already-loaded old elements The GROVE_V4 gates overwrite_indexed_cleanup_inspection and delete_tree_cleanup_type_source each bought their correctness with a dedicated stored-element read per op: every overwrite-capable op and every DeleteTree in a batch paid +1 seek and the loaded element bytes at V4, even when no indexed tree was anywhere near the batch (surfaced by Dash Platform identity-op fee tests at protocol v14 as +2000 seeks + 20 bytes per op unit). Derive the old element from data the apply already loads instead: - merk: apply_unchecked_with_old_value_observer threads an infallible (key, old_value, disposition) observer through the batch walk. The walker fetches an existing node before rewriting or deleting it, so surfacing its stored value is free — no storage read, no tracked cost. - overwrite gate: execute_ops_on_path registers overwrite-capable keys and classifies the displaced element in the observer. Bare Reference overwrites are now included (there is no longer a cost argument for leaving a reference over an indexed tree unswept), which closes #776 and un-ignores its regression test. Non-empty indexed replacements are still refused — normally by the earlier ungated empty-at-batch-insertion guard, with the classifier's NotSupported kept as defense in depth. - delete-tree gate: cleanup-namespace classification moves after apply_body, driven by the ACTUAL stored types captured by the observer (declared vs stored mismatches involving an indexed tree are rejected there). The pre-apply Error/Skip emptiness checks read the stored element once up front — the same single read V1..V3 pay directly or inside the child-merk open — and hand it to a new open-with-parent-element helper so nothing is read twice. The V1..V3 pre-scan path is byte-for-byte the released behaviour, now shared by both apply_batch entry points via scan_delete_tree_ops. - estimated costs: revert the batch estimator modeling of the gated reads (#789) in both the average- and worst-case models — with the applied reads gone the estimators were overshooting by exactly one read per gated op. A new cost-parity test pins a batch of plain overwrites plus a plain DeleteTree to an identical CostResult under GROVE_V3 and GROVE_V4, and the V4 refresh-reference cost test now asserts the same constants as its _v3_keeps_live_costs companion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: cover the pre-scan branches and observer paths the gate rework added Raises patch coverage over the 90% bar (codecov/patch was at 87.76%): - grovedb: four DeleteTree pre-scan tests — missing-key emptiness read, same-batch child-delete carve-out (with a plain Delete elsewhere and a Skip-policy sibling), the V1..V3 declared-type-at-face-value shapes on GROVE_V3, and a two-mismatch batch exercising the observer's short-circuit after the first gate rejection. - merk: consolidate the six duplicated inert-callback bundles in the ops test module into one apply_to_plain helper (whose no-op observer now executes via the update tests), and add a build-with-mid-delete test covering build's delete arm. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs+test: address review — capture-struct return doc, tighter overwrite asserts - Document apply_batch_structure's BatchApplyCaptures return (both fields) instead of the stale two-tuple description. - Pin the non-empty indexed batch-insertion refusal to the exact InvalidBatchOperation message instead of a three-variant alternation. - Require the indexed-to-Item safe-subset overwrite to succeed instead of tolerating either outcome. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue being fixed or feature implemented
GROVE_V4gatesapply_batch.delete_tree_cleanup_type_sourceandapply_batch.overwrite_indexed_cleanup_inspection, each charging one extra stored-element read perDeleteTree/ overwrite-capable op in the applied batch path. The batch estimated-cost paths (grovedb/src/batch/estimated_costs/{average,worst}_case_costs.rs) did not model this read, so estimated costs undershot applied costs by exactly that read on affected ops at grove version >= 4. Surfaced by Dash Platform's identity-op fee tests at protocol v14 (grove v4): applied processing fees came out exactly one read above the v3-era values while estimated-mode paths still produced v3-era numbers.What was done?
Both estimators now charge the gated read, keyed off the same two version slots so V1..V3 estimates are byte-for-byte unchanged:
InsertOrReplace/Replace/Patch): one average-/worst-case Merk node read of the layer's estimated element. Skipped when tree-override validation is on, for bare-reference elements, and for layers known to be empty — the same conditions under which the applied path skips the read or gets it for free (a get on an empty Merk costs nothing), which keeps the existing fresh-insert estimated==applied tests exact.storage_loaded_bytesshapeElement::get_from_storagereports for tree elements. This read is deterministic (raw storage get, no Merk caching), so the estimated V4−V3 delta equals the applied V4−V3 delta field for field.New tests (per gate, average and worst case):
test_average_case_delete_tree_type_source_read_estimated_vs_applied_parity— runs the same batch applied and estimated on V3 and V4 and asserts the two deltas match exactly (seek, loaded bytes, storage cost, context-prefix hash calls).test_average_case_overwrite_inspection_read_gated_v4_vs_v3/test_worst_case_overwrite_inspection_read_gated_v4_vs_v3— pin the estimated delta to exactly one modeled node read, and pin the suppression conditions (validation on, bare reference) to a zero delta.test_worst_case_delete_tree_type_source_read_gated_v4_vs_v3— pins the worst-case delta to context hash + one worst-case layered load.Existing exact-cost expectations that run at
latest()(= V4) were updated by the modeled read, with comments documenting the delta; theestimated == appliedseek assertion intest_batch_root_one_tree_insert_op_under_element_average_case_costsbecameapplied + 1 == estimatedbecause the applied path gets that particular read for free from the cached root node.Also made
grovedb/src/estimated_costs/worst_case_costspub(crate)so the batch estimator can reuseWORST_CASE_FLAGS_LEN.How Has This Been Tested?
cargo test -p grovedb— full crate suite green (2544 tests), including the 6 new/updated estimator tests.cargo check -p grovedb --no-default-features --features minimaland--features verify— feature-gated builds unaffected.cargo clippy -p grovedb --all-targets— no new lints.Breaking Changes
None on released versions: both charges are gated on the V4 feature slots, so V1..V3 estimated outputs are unchanged. V4 estimated outputs grow by exactly the read the V4 applied path already charges.
Checklist:
🤖 Generated with Claude Code