Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: rework record sync for improved reliability #1478

Merged
merged 13 commits into from
Jan 5, 2024
Merged

Commits on Jan 3, 2024

  1. feat: rework record sync for improved reliability

    So, to tell a story
    
    1. We introduced the record sync, intended to be the new algorithm to
       sync history.
    2. On top of this, I added the KV store. This was intended as a simple
       test of the record sync, and to see if people wanted that sort of
       functionality
    3. History remained syncing via the old means, as while it had issues it
       worked more-or-less OK. And we are aware of its flaws
    4. If KV syncing worked ok, history would be moved across
    
    KV syncing ran ok for 6mo or so, so I started to move across history.
    For several weeks, I ran a local fork of Atuin + the server that synced
    via records instead.
    
    The record store maintained ordering via a linked list, which was a
    mistake. It performed well in testing, but was really difficult to debug
    and reason about. So when a few small sync issues occured, they took an
    extremely long time to debug.
    
    This PR is huge, which I regret. It involves replacing the "parent"
    relationship that records once had (pointing to the previous record)
    with a simple index (generally referred to as idx). This also means we
    had to change the recordindex, which referenced "tails". Tails were the
    last item in the chain.
    
    Now that we use an "array" vs linked list, that logic was also replaced.
    And is much simpler :D
    
    Same for the queries that act on this data.
    
    ----
    
    This isn't final - we still need to add
    
    1. Proper server/client error handling, which has been lacking for a
       while
    2. The actual history implementation on top
        This exists in a branch, just without deletions. Won't be much to
        add that, I just don't want to make this any larger than it already
        is
    
    The _only_ caveat here is that we basically lose data synced via the old
    record store. This is the KV data from before.
    
    It hasn't been deleted or anything, just no longer hooked up. So it's
    totally possible to write a migration script. I just need to do that.
    ellie committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    5dbaeab View commit details
    Browse the repository at this point in the history
  2. update .gitignore

    ellie committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    22495cf View commit details
    Browse the repository at this point in the history
  3. use correct endpoint

    ellie committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    57c217c View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    47444ba View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2024

  1. Configuration menu
    Copy the full SHA
    785e787 View commit details
    Browse the repository at this point in the history
  2. lint, remove unneeded host_id

    ellie committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    707b436 View commit details
    Browse the repository at this point in the history
  3. remove prints

    ellie committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    e72010d View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    94886d3 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    d1a74f8 View commit details
    Browse the repository at this point in the history
  6. add record sync to auto sync

    ellie committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    23abb98 View commit details
    Browse the repository at this point in the history
  7. satisfy the almighty clippy

    ellie committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    c9e46bd View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    28a804c View commit details
    Browse the repository at this point in the history

Commits on Jan 5, 2024

  1. feedback

    ellie committed Jan 5, 2024
    Configuration menu
    Copy the full SHA
    0e8bd74 View commit details
    Browse the repository at this point in the history