-
-
Notifications
You must be signed in to change notification settings - Fork 507
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: rework record sync for improved reliability #1478
Merged
Merged
Commits on Jan 3, 2024
-
feat: rework record sync for improved reliability
So, to tell a story 1. We introduced the record sync, intended to be the new algorithm to sync history. 2. On top of this, I added the KV store. This was intended as a simple test of the record sync, and to see if people wanted that sort of functionality 3. History remained syncing via the old means, as while it had issues it worked more-or-less OK. And we are aware of its flaws 4. If KV syncing worked ok, history would be moved across KV syncing ran ok for 6mo or so, so I started to move across history. For several weeks, I ran a local fork of Atuin + the server that synced via records instead. The record store maintained ordering via a linked list, which was a mistake. It performed well in testing, but was really difficult to debug and reason about. So when a few small sync issues occured, they took an extremely long time to debug. This PR is huge, which I regret. It involves replacing the "parent" relationship that records once had (pointing to the previous record) with a simple index (generally referred to as idx). This also means we had to change the recordindex, which referenced "tails". Tails were the last item in the chain. Now that we use an "array" vs linked list, that logic was also replaced. And is much simpler :D Same for the queries that act on this data. ---- This isn't final - we still need to add 1. Proper server/client error handling, which has been lacking for a while 2. The actual history implementation on top This exists in a branch, just without deletions. Won't be much to add that, I just don't want to make this any larger than it already is The _only_ caveat here is that we basically lose data synced via the old record store. This is the KV data from before. It hasn't been deleted or anything, just no longer hooked up. So it's totally possible to write a migration script. I just need to do that.
Configuration menu - View commit details
-
Copy full SHA for 5dbaeab - Browse repository at this point
Copy the full SHA 5dbaeabView commit details -
Configuration menu - View commit details
-
Copy full SHA for 22495cf - Browse repository at this point
Copy the full SHA 22495cfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 57c217c - Browse repository at this point
Copy the full SHA 57c217cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 47444ba - Browse repository at this point
Copy the full SHA 47444baView commit details
Commits on Jan 4, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 785e787 - Browse repository at this point
Copy the full SHA 785e787View commit details -
Configuration menu - View commit details
-
Copy full SHA for 707b436 - Browse repository at this point
Copy the full SHA 707b436View commit details -
Configuration menu - View commit details
-
Copy full SHA for e72010d - Browse repository at this point
Copy the full SHA e72010dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 94886d3 - Browse repository at this point
Copy the full SHA 94886d3View commit details -
Configuration menu - View commit details
-
Copy full SHA for d1a74f8 - Browse repository at this point
Copy the full SHA d1a74f8View commit details -
Configuration menu - View commit details
-
Copy full SHA for 23abb98 - Browse repository at this point
Copy the full SHA 23abb98View commit details -
Configuration menu - View commit details
-
Copy full SHA for c9e46bd - Browse repository at this point
Copy the full SHA c9e46bdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 28a804c - Browse repository at this point
Copy the full SHA 28a804cView commit details
Commits on Jan 5, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 0e8bd74 - Browse repository at this point
Copy the full SHA 0e8bd74View commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.