Integrate with atree inlining and data deduplication #2882
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit updates Cadence to use newer version of onflow/atree with register inlining and data deduplication features. - Updated StorableDecoder callbacks - Updated StringAtreeValue to implement atree.ComparableStorable - Updated SomeStorable to implement atree.ContainerStorable - Updated these structs to implement TypeInfo interface: * compositeTypeInfo * EmptyTypeInfo * VariableSizedStaticType * ConstantSizedStaticType * DictionaryStaticType - Updated decodeStorable() to decode atree inlined array, map, and compact map - Updated to use ReadOnly iterators when possible - Updated to use NonReadOnly iterators
fxamacker
added
Feature
Performance
Storage Breaking Change
fxamacker
requested review from
turbolent,
SupunS and
dsainati1
as code owners
Cadence Benchstat comparisonThis branch with compared with the base branch onflow:feature/atree-register-inlining commit fd29ad8 Collapsed results for better readability
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
fxamacker
force-pushed
the
fxamacker/integrate-with-atree
branch
from
c8cc585
to
ca1a9ea
Compare
turbolent
reviewed
Oct 19, 2023
| @@ -18043,7 +18059,7 @@ func (v *DictionaryValue) ForEachKey( | |||
| } | |||
|
|
|||
| iterate := func() { | |||
| err := v.dictionary.IterateKeys( | |||
| err := v.dictionary.IterateReadOnlyKeys( | |||
There was a problem hiding this comment.
Are we sure that the procedure passed to DictionaryValue.ForEachKey does not mutate? I'd be conservative and switch to a read-only iterator after we ensured it. It might also be good to indicate that in the name then
There was a problem hiding this comment.
Are we sure that the procedure passed to DictionaryValue.ForEachKey does not mutate?
Hmm, do you mean the procedure might mutate map value by iterated key?
Using a non-readonly vs readonly iterator probably wouldn't make a difference here because mutation through iterator is done by callback in returned map value. Since procedure only processes map key, mutating map value is not possible by callback. Actually, atree only has readonly iterators for map keys for the same reason.
Thoughts?
There was a problem hiding this comment.
[...] because mutation through iterator is done by callback in returned map value
Not quite sure what that means. Do you maybe have an example?
mutating map value is not possible by callback
The callback is free to do anything, including removing keys, inserting new keys, mutating values in the dictionary, etc.
I guess your point is still that mutable vs read-only iterator doesn't matter?
| @@ -28,6 +28,7 @@ type StringAtreeValue string | |||
|
|
|||
| var _ atree.Value = StringAtreeValue("") | |||
| var _ atree.Storable = StringAtreeValue("") | |||
| var _ atree.ComparableStorable = StringAtreeValue("") | |||
There was a problem hiding this comment.
Is this also needed for Uint64AtreeValue?
There was a problem hiding this comment.
Unless I'm mistaken, it probably isn't needed because:
atree.ComparableStorableis only needed foratree.OrderedMapkeys in CadenceCompositeValueStringAtreeValueseems to be the only key type used byCompositeValue
There was a problem hiding this comment.
StringAtreeValue is also used for "non-interpreter.Value" storage, i.e. for the atree ordered maps used for account storage, and Uint64AtreeValue is only used for those.
For example, saving a value to /storage/foo actually is implemented as storing a value inside of the account storage's ordered map for the storage domain, under key StringAtreeValue("foo")
There was a problem hiding this comment.
This was discussed in today's meeting and it isn't for Uint64AtreeValue needed as mentioned above.
Co-authored-by: Bastian Müller <bastian@turbolent.com>
fxamacker
force-pushed
the
fxamacker/integrate-with-atree
branch
from
af06ef7
to
bc7b622
Compare
Co-authored-by: Bastian Müller <bastian@turbolent.com>
SupunS
reviewed
Oct 19, 2023
Co-authored-by: Supun Setunga <supun.setunga@gmail.com>
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## feature/atree-register-inlining #2882 +/- ##
===================================================================
+ Coverage 79.51% 79.52% +0.01%
===================================================================
Files 336 336
Lines 79191 79560 +369
===================================================================
+ Hits 62969 63273 +304
- Misses 13916 13974 +58
- Partials 2306 2313 +7
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
ramtinms
reviewed
Oct 20, 2023
…ows using read-only iterator
…eturn shallow copy
6 tasks
|
@turbolent Thanks for reviewing commit 1ecf35c on Friday! Today is Canadian holiday and Friday was US holiday, maybe we can sync about this before more holidays come up. 😄
Unless I missed something, I think we can skip incorporating all of the separate suggestions related to the last feedback. However, my reasons vary for some suggestions and I would like to get your thoughts about some new ideas/questions. 🙏 I will summarize here instead of replying to each suggestion individually because they are related. There are two main scenarios in your comments:
First Scenario
cadence/runtime/interpreter/value.go Lines 2769 to 2783 in 1ecf35c Second ScenarioThis is more involved because:
cadence/runtime/stdlib/account.go Lines 1072 to 1111 in 1ecf35c Alternative Suggestion for Second ScenarioWould it make sense to do these instead?
So Thoughts? |
|
TODO for this PR:
TODO for another PR, ideally before:
|
If remove parameter is true in Transfer(), iterated values are removed so non-readonly iterators need to be used.
- Renamed atRoot param to hasNoParentContainer for Transfer() and DeepRemove() - Added comments for each instances of hasNoParentContainer for Transfer() - For SomeValue.Transfer() and PublishedValue.Transfer(), use received hasNoParentContainer argument as inner value's Transfer() parameter
For SomeValue.DeepRemove(), PathCapabilityValue.Transfer(), and IDCapabilityValue.DeepRemove(), use received hasNoParentContainer argument as inner value's DeepRemove() parameter. Also, add comments for hasNoParentContainer in DeepRemove().
|
Hey @turbolent, thanks for a great meeting this morning to go over this PR again and also cover other issues in
Thanks for posting these TODOs from our meeting! 👍
Done in commit: Rename atRoot param for Transfer() and DeepRemove()
Done in commit: Use Atree non-readonly iterators in Transfer()
Done in commit: Use hasNoParentContainer for inner value DeepRemove() I hope you don't mind, 🙏 I was already there and wanted to have this PR wrapped up.
Thanks for tackling those outside this PR! 👍 |
ramtinms
reviewed
Jan 3, 2024
Comment on lines
+688
to
+690
| if nestedLevels == 1 { | ||
| return s.encode(e) | ||
| } |
There was a problem hiding this comment.
can this also be handled as part of encodeMultipleNestedLevels (instead of returning error there) ?
There was a problem hiding this comment.
Sorry for delay in reply, I missed this comment. 🙏
Maybe not in this case because encoding format is differently for nestedLevel == 1 (using encode()) vs nestedLevel > 1 (using encodeMultipleNestedLevels()):
- if nested level == 1, only innermost unwrapped storable is encoded
- if nested level > 1, then 2-element array is encoded: first element is nested levels, second element is innermost unwrapped storable.
6 tasks
|
The latest commits look good 👍 |
turbolent
reviewed
Jan 23, 2024
| valueComparator := newValueComparator(interpreter, locationRange) | ||
| hashInputProvider := newHashInputProvider(interpreter, locationRange) | ||
|
|
||
| iterator, err := v.dictionary.Iterator(valueComparator, hashInputProvider) |
There was a problem hiding this comment.
We have to use a mutable iterator here, because the values that are returned by the iterator might get mutated, so need to have the parent notification callback set up, correct?
|
@fxamacker Do you happen to remember what I meant with
I vaguely remember we discussed how there's a potential problem with that approach in the inlining version |
@turbolent Yeah, I remember that too. For context:
Part of the discussion to simplify
|
turbolent
approved these changes
Jan 24, 2024
turbolent
merged commit deffd50
into
feature/atree-register-inlining
8 of 11 checks passed
This was referenced Jan 25, 2024
turbolent
pushed a commit
that referenced
this pull request
Jan 26, 2024
Cadence smoke test is reporting false alarm about "slab was not reachable". To summarize, storage.CheckHealth() should be called after array.DeepRemove(), rather than in the middle of array.DeepRemove(). CheckHealth() is called in the middle of array.DeepRemove() when: - array.DeepRemove() calls childArray1 and childArray2 DeepRemove() - DeepRemove() calls maybeValidateAtreeValue() - maybeValidateAtreeValue() calls CheckHealth() The details are more complex than this oversimplified summary. For more context, see comments/discussion on GitHub at #2882 (comment)
turbolent
pushed a commit
that referenced
this pull request
Jan 26, 2024
Cadence smoke test reported another false alarm about "slab was not reachable". To summarize, storage.CheckHealth() should be called after DictionaryValue.Transfer() with remove flag, rather than in the middle of DictionaryValue.Transfer() with remove flag. The details are more complex than this oversimplified summary. For more context, see GitHub comments at #2882 (comment)
This was referenced Feb 15, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #2809
Updates onflow/atree#292
This commit updates Cadence to use newer version of onflow/atree with register inlining and data deduplication features introduced by onflow/atree#342 and onflow/atree#345.
This PR requires onflow/atree#352.
Changes
Updated StorableDecoder callbacks
Updated StringAtreeValue to implement atree.ComparableStorable
Updated SomeStorable to implement atree.ContainerStorable
Updated these structs to implement TypeInfo interface:
Updated decodeStorable() to decode atree inlined array, map, and compact map
Updated to use ReadOnly iterators when possible
Updated to use NonReadOnly iterators
TODO
Cadence team 🙏 will need to make additional changes:
*InterpreterandLocationRangeto some functions in order to use non-readonlyatree.OrderedMapiteratorsatree.Arrayandatree.OrderedMapand needs to implementatree.ContainerStorableinterfacemasterbranchFiles changedin the Github PR explorer