Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The number of accounts/ files in TdS and SLP snapshots is too high #8168

Closed
mvines opened this issue Feb 7, 2020 · 7 comments
Closed

The number of accounts/ files in TdS and SLP snapshots is too high #8168

mvines opened this issue Feb 7, 2020 · 7 comments
Assignees
Milestone

Comments

@mvines
Copy link
Member

@mvines mvines commented Feb 7, 2020

Snapshots are taking 1000 slots or worse to unpack and process on TdS/SLP, due to the large number of accounts/ files in them. At this time, TdS has over 4500 files in accounts/, SLP has almost 3000.

This slows down the ability of validators to quickly boot and restart.

@mvines mvines added this to the Tofino v0.23.3 milestone Feb 7, 2020
@ryoqun

This comment has been minimized.

Copy link
Member

@ryoqun ryoqun commented Feb 10, 2020

This is partly affected by this #8148, which causes some unneeded storage entries not to be deleted forever.

@mvines

This comment has been minimized.

Copy link
Member Author

@mvines mvines commented Feb 15, 2020

This is starting to be a real problem. I'm getting snapshots on TdS and SLP that are from more than 10 minutes ago, which causes new validators to catch up very slowly

@ryoqun

This comment has been minimized.

Copy link
Member

@ryoqun ryoqun commented Feb 19, 2020

Status update: I've created a fix locally which reduces the current TdS snapshot into 1/5. PRs are coming shortly after.

@mvines

This comment has been minimized.

Copy link
Member Author

@mvines mvines commented Feb 27, 2020

oops, this is not fixed!

@mvines mvines reopened this Feb 27, 2020
TdS Potholes automation moved this from Closed to Needs triage Feb 27, 2020
@mvines mvines moved this from Needs triage to High priority in TdS Potholes Feb 27, 2020
@mvines mvines modified the milestones: Tofino v0.23.8, Tofino v0.23.9 Feb 29, 2020
@ryoqun

This comment has been minimized.

Copy link
Member

@ryoqun ryoqun commented Mar 13, 2020

Finally, my almost 1-month worth labor is paying off:

TdS:

image

SLP:

image

@mvines mvines modified the milestones: v1.0.7, v1.0.8 Mar 16, 2020
@mvines

This comment has been minimized.

Copy link
Member Author

@mvines mvines commented Mar 16, 2020

@ryoqun - I think we can call this issue fixed now right?

@ryoqun

This comment has been minimized.

Copy link
Member

@ryoqun ryoqun commented Mar 16, 2020

@mvines yes from operational standpoint!

There is still minor security issue, which I noticed and could be mitigated by more aggressive cleaning implementation. But there are bunch of similar low-priority security issues when deserializing a snapshot, which is tracked by #7167. So, closing this now.

@ryoqun ryoqun closed this Mar 16, 2020
TdS Potholes automation moved this from High priority to Closed Mar 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
TdS Potholes
  
Closed
Linked pull requests

Successfully merging a pull request may close this issue.

2 participants
You can’t perform that action at this time.