-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reduce RAM and persistent storage by deduplicating inlined dict type info #369
Reduce RAM and persistent storage by deduplicating inlined dict type info #369
Conversation
This change deduplicates Cadence dictionary type and composite type info, resulting in reduced memory and also persistent storage. More specifically, this encodes inlined atree slab extra data section as two-element array: - array of deduplicated type info - array of deduplicated extra data with type info index
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## feature/array-map-inlining #369 +/- ##
==============================================================
+ Coverage 62.45% 70.38% +7.92%
==============================================================
Files 15 15
Lines 10919 12653 +1734
==============================================================
+ Hits 6820 8906 +2086
+ Misses 3119 2749 -370
- Partials 980 998 +18 ☔ View full report in Codecov by Sentry. |
I found some improvements to deduplication while migration + validation was running last night. |
Work on this PR is paused this morning to meet about and begin work on urgent request to help with Cadence 1.0 migration.
This PR has about 1/2 day of effort left to improve its deduplication for further reduction of memory and persistent storage. |
Yesterday, this passed atree inlining & deduplication migration with validation enabled (for all payloads for all accounts) using mainnet Jan 19, 2024 checkpoint as input. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me
This commit bumps atree version to benefit from deduplication of atree inlined Cadence dictionary type info. For more info, see onflow/atree#369
This commit bumps atree version to benefit from deduplication of atree inlined Cadence dictionary type info. For more info, see onflow/atree#369
Closes #358 Updates epic #292
This PR deduplicates Cadence dictionary type resulting in reduced memory and persistent storage.
More specifically, this encodes inlined atree slab extra data section as two-element array:
TODO (est. 1.5 - 2 days effort total):
[x] - initial deduplication of dict type info
[x] - build, run, and pass migration with validation enabled for all accounts and all payloads from mainnet snapshot
[x] - improve deduplication (1/2 day effort to wrap up + rerun test using mainnet snapshot)
UPDATE (Feb 29, 2024): 🛑 Work on this PR is paused to work on urgent request to create tooling for Cadence 1.0 migration.
NOTE: This PR is expected to produce small improvements to atree inlining & deduplication results 📊.
main
branchFiles changed
in the Github PR explorer