Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tortoise: tortoise state loading will be growing in time #3006

Closed
dshulyak opened this issue Dec 16, 2021 · 4 comments
Closed

tortoise: tortoise state loading will be growing in time #3006

dshulyak opened this issue Dec 16, 2021 · 4 comments
Assignees

Comments

@dshulyak
Copy link
Contributor

dshulyak commented Dec 16, 2021

After persistent state was removed tortoise loads state from meshdb and atxdb when the node is started. In the initial implementation i am also re-verifying everything, as it allowed to reuse code for rerun almost as is.

However after #3000 is merged rerun in full mode will be rather slow. So instead of verifying when node is booted we can just load state.

the end goal is to make restart fast, we can achieve that:

  • by eliminating bottlenecks in tortoise
  • by snapshotting tortoise when node is stopped gracefully
  • by loading state without verifications
  • by offloading state loading into background, and blocking components that rely on tortoise
@dshulyak
Copy link
Contributor Author

Note: we don't need state in full.weight for verified layers, but we need to know which ballots are good, because the definition is recursive.

@dshulyak dshulyak self-assigned this Dec 28, 2021
@dshulyak
Copy link
Contributor Author

dshulyak commented Jan 4, 2022

Also need to compute real margins for layers between verified and processed, as we will use them for voting. So it is not completely possible to avoid vote counting. Will postpone this change.

@dshulyak dshulyak removed their assignment Jan 4, 2022
@dshulyak
Copy link
Contributor Author

dshulyak commented Oct 6, 2022

We can do this after mutable ballots will be implemented. They allow us to verify goodness of the ballot without recursively decoding all previous votes, instead we can just compare if ballot encoded opinion is identical to local opinion.

@dshulyak dshulyak assigned dshulyak and unassigned dshulyak Aug 29, 2023
@dshulyak dshulyak changed the title tortoise: load state without verification tortoise: tortoise state loading will be growing in time Sep 3, 2023
@dshulyak dshulyak self-assigned this Sep 8, 2023
@countvonzero
Copy link
Contributor

minor issue i encountered is that the loading is not interrupt-able with sigterm as it doesn't check context cancel.

bors bot pushed a commit that referenced this issue Sep 18, 2023
related: #3006

- tally votes once per window, or after all ballots were loaded
- load ballots without joining identities table. that information already available in tortoise
- set tortoise log level to warn on mainnet. it is not pleasant that info is printed every time node is restarted
bors bot pushed a commit that referenced this issue Sep 20, 2023
related: #3006

it also reduced 300MB of used memory on my machine.

ballot signs opinion hash, and provides sidecar to decode that opinion hash. if node decoded one sidecar it can store it in memory
and reuse for all following ballots with the same opinion hash.
bors bot pushed a commit that referenced this issue Sep 20, 2023
related: #3006
includes: #5009

this improved recovery by 20s on my computer, from 1m to 40s. will upload benchmarks from source before merge
bors bot pushed a commit that referenced this issue Sep 20, 2023
related: #3006
includes: #5009

this improved recovery by 20s on my computer, from 1m to 40s. will upload benchmarks from source before merge
bors bot pushed a commit that referenced this issue Sep 22, 2023
related: #3006

check if context was cancelled , and exit with context error if it was cancelled
@bors bors bot closed this as completed in 26b8b55 Oct 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
2 participants