Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove PendingBootstrapStore and apply bootstrap changesets in a continuous transaction #7285

Open
sync-by-unito bot opened this issue Jan 23, 2024 · 0 comments

Comments

@sync-by-unito
Copy link

sync-by-unito bot commented Jan 23, 2024

As part of FLX sync we need to guarantee that once we begin applying changesets from a fully received bootstrap, we apply all changesets in that bootstrap and that no user writes happen during the bootstrap application. Right now we do this by storing all the received bootstraps in a separate side-table and then applying them in a series of write transactions where we do not release the write lock between commits.

This has the downside of having to store all the data twice, once in the bootstrap store and then again when we apply the changesets. If the database can give us an interface to break up these big writes into chunks (so that we don't use unbounded amounts of memory) and also roll back the database to the beginning of the bootstrap if an error occurs, we can get rid of the pending bootstrap store and have a simpler, faster, more correct way of processing bootstraps. We'll also get significantly better visibility into how far into the bootstrap we are because the download progress will also be the application progress.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant