Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix snapshots manager #5764

Merged
merged 3 commits into from Dec 13, 2023
Merged

Fix snapshots manager #5764

merged 3 commits into from Dec 13, 2023

Conversation

iulianpascalau
Copy link
Contributor

@iulianpascalau iulianpascalau commented Dec 11, 2023

Reasoning behind the pull request

  • during a sync from genesis on testnet, the metachain node ends up having the PeerAccountsTrie DB at ~13GB even if the config.toml specified that SnapshotsEnabled = false and PeerStatePruningEnabled = true
    The snapshotting process waited for epoch change & snapshot markup even if the trie storage manager did not perform snapshots. The result was that the memoryEvictionWaitingList started dropping stored keys so the correct state pruning was not performed.

Proposed changes

  • added an earlier exit point on the snapshotsManager handler

Testing procedure

  • standard system test
  • start from genesis on testnet on all shards and measure the space occupied by the DBs.

Pre-requisites

Based on the Contributing Guidelines the PR author and the reviewers must check the following requirements are met:

  • was the PR targeted to the correct branch?
  • if this is a larger feature that probably needs more than one PR, is there a feat branch created?
  • if this is a feat branch merging, do all satellite projects have a proper tag inside go.mod?

@ssd04 ssd04 self-requested a review December 11, 2023 16:32
Comment on lines +161 to +162
if !trieStorageManager.IsSnapshotSupported() {
log.Debug("skipping snapshot as the snapshot is not supported by the current trieStorageManager",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm thinking if ShouldTakeSnapshot can be used instead of adding IsSnapshotSuppoted method. for trie storage manager without snapshot ShouldTakeSnapshot already returns false.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I have added the new function is that the original trieStorageManager that supports snapshotting has logic inside the ShouldTakeSnapshot. And that logic we want to execute after the node steps into the new epoch. So it's kind of a chicken-and-egg problem without the new function.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It won't work as it is right now. There are a few things that need to be set before ShouldTakeSnapshot is called if the snapshots are enabled. Will refactor in a future PR to merge these 2 funcs.

Comment on lines +161 to +162
if !trieStorageManager.IsSnapshotSupported() {
log.Debug("skipping snapshot as the snapshot is not supported by the current trieStorageManager",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It won't work as it is right now. There are a few things that need to be set before ShouldTakeSnapshot is called if the snapshots are enabled. Will refactor in a future PR to merge these 2 funcs.

Copy link

codecov bot commented Dec 12, 2023

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (1c2116b) 80.14% compared to head (741ced4) 80.14%.

Files Patch % Lines
...r/evictionWaitingList/memoryEvictionWaitingList.go 33.33% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           rc/v1.6.0    #5764   +/-   ##
==========================================
  Coverage      80.14%   80.14%           
==========================================
  Files            708      708           
  Lines          94017    94027   +10     
==========================================
+ Hits           75347    75359   +12     
+ Misses         13326    13325    -1     
+ Partials        5344     5343    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@valentin-lup valentin-lup left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested on Public Testnet Stack

Copy link
Collaborator

@gabi-vuls gabi-vuls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Normal allin test: master-e134e5314d -> fix-snapshots-manager-81f69cfdee

--- Specific errors ---

block hash does not match 12439
wrong nonce in block 4188
miniblocks does not match 0
num miniblocks does not match 0
miniblock hash does not match 0
block bodies does not match 0
receipts hash missmatch 0

/------/

--- Statistics ---

Nr. of all ERRORS: 22
Nr. of all WARNS: 9673
Nr. of new ERRORS: 21
Nr. of new WARNS: 132
Nr. of PANICS: 0

/------/

@iulianpascalau iulianpascalau merged commit 3935103 into rc/v1.6.0 Dec 13, 2023
8 checks passed
@iulianpascalau iulianpascalau deleted the fix-snapshots-manager branch December 13, 2023 11:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants